Real world Scala hAkking NLJUG JFall 2011
Download as PPTX, PDF1 like1,213 views
The document provides an overview of using Scala and Akka for building distributed sensor networks. Some key points: - The speaker uses Scala and Akka at their company for building distributed systems to manage traffic and sensor networks. - Akka actors are used to build distributed and fault-tolerant systems. Camel is used for integration between actors. Remote actors allow building systems that span multiple machines. - Examples of systems built include a border control sensor network and distributed traffic management systems. Scala and Akka provide benefits like less code, fault tolerance, and ability to interoperate with existing Java systems. - Topics covered include using Akka actors, remote actors, Camel integration,
Real world Scala hAkking NLJUG JFall 2011
- 1. Real World Scala hAkking Raymond Roestenburg
- 2. About me • Java since 1997 • Scala since 2010 • Mostly backend systems • Lead architect Traffic Management at CSC • Akka committer code: http://github.com/RayRoestenburg blog: http://roestenburg.agilesquad.com twtr: @RayRoestenburg
- 3. Company I work for Traffic Management • Section Control • Weigh in Motion • Traffic Information • Traffic Enforcement
- 4. Agenda • @MIGO-BORAS sensordomain • Sensor networks • Typesafe stack • Akka Actors • Apache Camel Integration • Remote Actors • Monitoring • Testing • Deployment
- 5. @MIGO-BORAS • Koninklijke Marechaussee • MTV (Mobiel Toezicht Vreemdelingen) • Prevent illegal border crossing • Sensor domain • Fixed locations • Mobile vehicles • Cameras, Radars • Vehicle properties • January 2012
- 6. Sensor networks • Usable for many applications in our field • Distributed along highways and provincial roads • Traffic Enforcement • Traffic Information • Other applications • Roadside systems • Range of sensor types – Camera’s, Infrared, Laser, Piezoelectric, Loops, Mobile units, … • Remotely Configure & Control
- 7. Why Scala & Akka • Less code, less bugs • Less code, same VM power • Experienced team (Java, C#, C) • 3 experienced Java devs • 1 experienced C# dev with no Java experience • Need to interoperate with existing Java and C (JNA) • Simplify development (C->Java->Scala) • Ubuntu
- 8. Scala 2.9.x • Learning curve (Start with simple features) • Traits, Tupels, Stackable Traits • First Class Functions • Scala Collections • map, flatMap, partition, foreach, filter, groupBy.. • Option type • map.. orElse, foreach,.. • Case classes • REPL, DSLs, Extension methods
- 9. Typesafe Stack • Current Stack • Open Source • SBT 0.7.7 • Scala 2.9.0-1 • Akka 1.1.3 • Camel & Camel Components 2.7.0 (FTP,Mina,Jetty,…) • Jboss Netty 3.2.4 • When we started early 2010 • Akka 0.7, Scala 2.8.0Beta1
- 10. Akka Usage Core Actors, Remote, TestKit, Serialization, STM, Agents, Dataflow, Async HTTP (Mist), FSM, Transactors Modules MicroKernel, Camel, AMQP, Scalaz, Spring, OSGi, DataFlow, Persistence,
- 11. Akka Actors ActorRef Mailbox Actor Message to send Dispatcher Threads
- 12. Akka Actors ActorRef Mailbox Actor Dispatcher Threads
- 13. Akka Actors ActorRef Mailbox Actor Dispatcher Threads
- 14. Akka Actors ActorRef Mailbox Actor Dispatcher Threads
- 15. Akka Actors ActorRef Mailbox Actor Dispatcher Threads
- 16. Akka Actors ActorRef Mailbox Actor Dispatcher Threads
- 17. One-way messaging Shared Dispatcher
- 18. Two-way
- 20. Camel Actors • Camel Component integration • Consumers and Producers
- 21. Camel Consumers
- 22. Camel Producers
- 23. Camel Producers • Use CamelContext
- 25. Dispatchers Camel Producers Local Processing Camel Consumers Heartbeat, Remote Actors Cleanup, Schedule, Long Running, Redelivery, Remote Error Handling
- 26. Messages • Case classes • Simple data containers • toProtobuf and fromProtobuf on case class and companion object for remoting • Every msg has a • correlation Id to the event • Timestamp • Logging, Tracing, Performance
- 27. Event Process Chains Load Balancer EndPoint (Consumer/Producer) Forwarding Actor
- 28. Remote Actors • Client • Actor.remote.actorFor(name, host,port) • Server • Actor.remote.register(name, actorRef) • remoteActorFor: (String, String, Int) => ActorRef = remote.actorFor
- 29. Remote Actors Internals • RemoteActorRef • RemoteServer • RemoteClient • Netty Server • Netty Channels • Lifecycle Event • Serialization (Scala, listeners Java, (S)JSON, ProtoBuf • RemoteProtocol RemoteActorRef RemoteProtocol ActorRef Mailbox Actor RemoteClient RemoteServer
- 30. Remote Actors (Akka 1.1.3) • JBoss Netty (Channels) • Not as transparent as you would like • Remote Lifecycle Listeners • Closing sockets, reconnect • Own Guaranteed Delivery implementation – Specific requirements – Bases on Idempotent Receiver and Repeating messages after reconnect – Non-Trivial – Heartbeats – Exp Backoff
- 31. Remote configuration • #compression-scheme = "zlib“ #leave out • zlib-compression-level = 0 • client: reconnection-time-window = 2000000 • client: read-timeout (we use 60) • backlog = 4096 # Netty backlog for connections, should suffice, increase if failures happen
- 32. Monitoring • Application level monitoring • Listener Actors contain Custom JMX MBeans • Direct passthrough of msg or translation
- 34. Testing one-way Actor Output
- 35. Microkernel deployment • sbt dist (akka-sbt-plugin) • Creates dist directory • init.d scripts
- 36. Questions?
Editor's Notes
- #13: ActorRef is an immutable reference which exclusively gives access to the Actor.Actors run on Dispatchers. A Dispatcher runs a mailbox to process messages.Amount of messages per mailbox run is configurable. (actor.throughput, default 5)
- #18: One way messaging between two ActorsActors can process messages in parallel
- #19: One way messaging between two ActorsActors can process messages in parallel
- #20: !!! Is preferred over !!Futuresalso run onDispatchers.FutureTimeoutException happens onfuture.get. IfonlyonComplete and failure are registered, yougetnoFutureTimeoutException