Do's and don'ts when deploying akka in production
- 1. Deploying Akka in Production Do’s and Don’ts when Mike Nash
- 2. Introduction We’ve deployed dozens of Akka-based services and apps to production in the past 5 years We’ve scaled up to tens of millions of events per hour, day after day Our team has learned where the potholes are
- 3. Architecting for Performance Going reactive before the first line of code is written
- 4. Code Quality and Architecture Quality = lower cost of future changes Correct architecture = longevity, e.g the system lives on even if parts are replaced in flight Many small pieces usually work better than a few big ones Find out about Ports and Adapters/ Onion Architecture: Keep domain and infrastructure distinct
- 5. Delivery Guarantees Understand what guarantees Akka provides - and what it doesn’t The best design is often the one that doesn’t need many guarantees Most “guarantees” are imaginary in any case
- 6. Single Points of Contention Database?
- 7. Blocking Operations Look for blocking operations
- 8. Immutability Encapsulate state in Actors (or Agents) Messages must be immutable
- 9. Keep it Simple Just because you can doesn’t mean you should Simple, independent pieces are best Keep your code out of other people’s databases Distinguish between Commands and Events
- 10. Think about back-pressure Think about back pressure: Look for any situations where mailboxes might be allowed to grow without limits Slow producer is not a big problem Slow consumer can be a big problem Akka streams Think about dispatchers and routers: The defaults are not always what you want.
- 11. Serialization Think about for remote actors Default requires some class path magic Choices include text, json, protobuf, many others, all with tradeoffs Benchmark your serializer/ deserializer!
- 12. Actor? Future? Agent? Actors encapsulate State or Behaviour, they have a mailbox and a name Futures contain and retrieve the result of a concurrent operation Agents encapsulate a value (at a single location) - immediate read, async writes
- 13. Testing Common performance blockers under heavy load
- 14. Testing Beyond the Obvious Unit and Functional are the Foundation Akka TestKit: Test actors as if they were synchronous “Negative” tests are essential
- 16. JVM Tuning
- 17. Memory You need more than you think you need
- 18. Know your Garbage Collectors Must run under load to see characteristics Defaults are probably not what you want Takes more memory to free up more memory… -verbosegc is your friend
- 19. Production Deployments Tips for a rock-solid deployment pipeline
- 20. Deployment Pipeline Not just about getting to production How will you deploy new versions without downtime? How will you revert if you have to? It’s 2 am - Do you know where your dependencies are?
- 21. Deployment Packaging How will you get your app to production? Onejar? In our experience, external containers are not helpful How will you tell if its the right version and revision? Rolling Restarts?
- 22. Dependencies Your application or service sits atop a tall stack of dependencies Is the stack in the same order on production? Is it the same stack?
- 23. Configuration Akka’s configuration works great - don’t fix it! Discovery beats hard-wiring Procrastinate
- 26. Statsd, Graphite Custom timing and counts
- 27. Kamon Specifically for Reactive Applications
- 28. What happens when….? Boundary Conditions
- 29. Logging Related to monitoring, but very different Surprisingly expensive - consider async, macros (like ScalaLogging) to omit when level disabled Line numbers are *very* expensive, check your formats Log rolling strategy is critical Consider Graylog or similar to view logs across multiple nodes, filter, route and alert on log contents Consider ssh server or remote message to change log level on the fly for a running system Logback’s Mapped Diagnostic Context can be handy
- 30. Security Beyond the usual security concerns any application has How will you prevent “fake” messages from being handled? There is such a thing as too much of a good thing
- 31. Summary