Deployment Nirvana
0 likes459 views
This document discusses the challenges faced by the author in deploying and scaling a microservices architecture. Some of the initial problems encountered were services flapping and resulting downtime during deployments. Various solutions tried included service registries, routing layers, and versioning at the service and inter-service level. However, complexity grew and developer experience suffered. The author advocates for simpler deployments that prioritize engineer productivity through better tooling while still enabling rolling deployments and isolation of services.
Deployment Nirvana
- 1. Deployment NirvanaDeployment NirvanaDeployment Nirvana Adrian Pike @adrianpike
- 2. The Web is changing. Monolithic Apps Services
- 3. The Web is changing. Scaling hardware is “easy”! 😊
- 4. The Web is changing. Scaling people is really really hard.
- 5. The Web is changing. Complex app, simple deployments
- 6. The Web is changing. Simple apps, complex deployments
- 9. Hi, I'm Adrian. 🙌 • Stride • Moz • Welltok That guy.
- 10. In the beginning, we wanted to build a software. In the beginning, we wanted to build a software.
- 11. Moz Local • Primarily about transforming API data. • Parallel IO —> node.js! • Large pile of independent services.
- 12. Why SoA • Team scaling vs. performance scaling. • Minimize cognitive load. • Composability!
- 13. Why not SoA • Complex Interdependencies. • Reliability. • Debugging. • Toolchains. • Integration testing. • Deployment.
- 14. A decision was made.
- 15. The descent into madness.🌀
- 16. The descent into madness.🌀 • Circus. • Collateral flapping. • Deploys result in downtime. • Service-level versioning.
- 17. Problem 0: Circus Today’s hard lesson - vet your tools.
- 18. Problem 1: Collateral Flap • We sucked at node. • Domains weren’t a thing. • Crashes happened a lot.
- 19. Problem 1: Collateral Flap • Multiple instances of a given backend. • Service registry and an HTTP proxy. We “solved” it!
- 20. Problem 2: Deployment Downtime • Well now that we’ve got a port registry…
- 21. Problem 2: Deployment Downtime • Fire up a new backend that registers itself. • Cleanly reap old backends, they finish their tea, crumpets, and requests and go away. We “solved” it!
- 22. DMV + Seaport + Viaduct • Pile of home-rolled, loosely coupled services.
- 23. DMV + Seaport + Viaduct is the greatest thing. • Shields crashes! • Zero-downtime deploys! • Lots of control!
- 24. DMV + Seaport + Viaduct is the worst thing ever. • Client tooling is immature. • Development is basically torture. • Inter-service versioning is unsolved.
- 25. Service versioning • “Solved” problem. • URL Versioning • Query Params • Timestamps • Inter-service dependencies are real, and require our attention. • Is there a better way?
- 26. Service versioning redux • git commit -m “replaced cats with dogs, life’s better now” • b8d825ddd9eb63336f1da4ec46b1a47eecab7427 • That looks suspiciously like a version.
- 27. Routing layer versioning • We’ve got a HTTP proxy already. • Let’s extend it to route to git SHA’s as versions. • We route domains to default SHAs. production.foo.com 5e3… Service! staging.foo.com 61a… A different service!
- 28. Routing Layer Versioning • Advantages: • As an engineer, I don’t have to build versioning. • We can test multiple releases simultaneously. • Disadvantages: • Deployed services have to be SHA reflective. • We have to run multiple instances of a given service. • Inter-service versioning.
- 29. Inter-service versioning • Service A is dependent on Service B. • Think a business logic service (A) and a data layer service (B). • They might need to change in lockstep. A B
- 30. Inter-service versioning • We build a “Release version” to encapsulate all our services at a given revision. A B V1 A B’ V2
- 31. Inter-service Communication • When A communicates with B, it’s got to specify a version. • Three ways; • I have to think about it. (Use API versioning or some such) • Lock at deploy time. • Pass a header.
- 32. Client Tooling • Developer environment • Just use Foreman. • Assets and such. • Just 12factorize.
- 33. Lessons learned • Inter-service versioning is unsolved. • Rolling deployments are required. • Shielding engineers from complexity. • Caveat: Without magic.
- 34. Let’s deploy better. 🌅 • Focus on engineer happiness. • Better tooling. • Minimize moving parts. • 4 daemons + 2 persistence layers == wat • Make it pluggable.