Microservices Antipatterns
7 likes3,161 views
The document presents insights on navigating the challenges of microservices architecture at the QCon conference. It emphasizes starting with monolithic applications, addressing common pitfalls like managing schemas and service dependencies, and solutions such as semantic versioning and circuit breakers. Additionally, it advocates for automating processes to enhance operation efficiency and prevent deployment issues.
Microservices Antipatterns
- 1. Quick poll…
- 2. InfoQ.com: News & Community Site • 750,000 unique visitors/month • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • News 15-20 / week • Articles 3-4 / week • Presentations (videos) 12-15 / week • Interviews 2-3 / week • Books 1 / month Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations /cloud-anti-patterns
- 3. Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide Presented at QCon London www.qconlondon.com
- 4. Welcome to sunny London!
- 5. Tammer Saleh Geek: Unix, Ruby, Golang, etc Cloud Foundry @ Pivotal http://tammersaleh.com | tsaleh@pivotal.io
- 6. Microservice Anti-patterns How not to go down in flames.
- 7. Why microservices? What is a microservice, and why do I care?
- 8. Monolithic Entire application in a single codebase, deployed and scaled as a single unit.
- 9. Monolithic Hard to scale the application. Impossible to scale the team.
- 10. It’s not about code… It’s about teams.
- 11. But it can go wrong. Here are the most common problems we see in the wild, and how to fix them.
- 12. Overzealous Services The most common mistake is to start with microservices
- 14. Solution: Start monolithic and extract Microservices are complex and add a constant tax to development. Build a boring application and extract services as needed.
- 17. Congratulations You’re now a microservice architect.
- 18. Schemas everywhere /dashboard /… /… /tweet v1 v1 V1 Schema Deploy v2
- 19. Schemas everywhere /dashboard /… /… /tweet v1 v2 V1 Schema Deploy v2 Migrate v2
- 20. /dashboard /… /… /tweet v1 v2 V2 Schema Deploy v2 Schemas everywhere Migrate v2
- 21. /dashboard /… /… /tweet v2 v2 V2 Schema Schemas everywhere
- 22. Solution: Gatekeeper /dashboard /… /… /tweet v1 v2 Tweets /bulk_add Tweet Service GET /tweets GET /tweets/ID Owns database and migrations
- 23. Lock-step deployment Service B Service A Service C — v1 v1 v1
- 24. Lock-step deployment Service B Service A Service C — v2 v1 v1 Deploy v2
- 25. Lock-step deployment Service B Service A Service C — v2 v1 v1 Deploy v2
- 26. Lock-step deployment Service B Service A Service C — v2 v2 v2 Deploy v2
- 27. Solution: Semantic Versioning Service B Service A Service C — v1 v1 v1 vMajor.Minor.Patch MYBAD.SHINY.OOPS
- 28. Solution: Semantic Versioning Service B Service A Service C — v1.2 v1 v1 Deploy v1.2 (extra stuff) :) :)
- 29. Solution: Semantic Versioning Service B Service A Service C — v1.2 v1.2 v1.2 Deploy v1.2
- 30. Solution: Semantic Versioning Service B Service A Service C — v1.2 v1.2 v1.2 Deploy v2 (breaking change) Service C — v2
- 31. Solution: Semantic Versioning Service B Service A Service C — v1.2 v1.2 v2 Service C — v2 Deploy v2
- 32. Solution: Semantic Versioning Service B Service A Service C — v1.2 v2 v2 Service C — v2 Deploy v2
- 33. OMG ALL THE STEPS!!! See Rule #1
- 34. Spiky load between services Must maintain enough servers to handle peak load.
- 35. /tweet /dashboard /… /… Spiky load between services
- 36. /tweet /dashboard /… /… Spiky load between services
- 37. Queues in between services provide buffers that smooth traffic. Solution: Amortize via queues Peak load is now much lower.
- 39. Solution: Amortize via queues /tweet /dashboard /… /… Worker Complexity: Now clients must deal with asynchronous responses.
- 40. Hardcoded IPs and Ports Simple to get started, but immediately leads to deployment issues.
- 41. Solution 1: Discovery Service etcdconsul
- 42. Solution 1: Discovery Service Service A Service B (192.168.0.2:1234) Where is B? 192.168.0.2:1234
- 43. Solution 1: Discovery Service Service A Service B (192.168.0.2:1234) Hey there!
- 44. Solution 1: Discovery Service Complexity: Your code must understand the service lookup system.
- 45. Solution 2: Centralised router Service AService B Router Service B Service A Service AService B A A BB A A BBA A BB
- 46. Solution 2: Centralised router Simplicity: “It’s just DNS.”
- 47. Router vs Discovery Service Both require service registration. Discovery Service is simpler to build and scale, since it doesn’t need to route all data. Discovery service does fewer network hops. Router is transparent. Router can be exposed externally. Router can cache transparently. Router can round-robin. Both require HA and scalability.
- 48. Dogpiles Service BService A You there? No. How about now? I’m busy. But I wanna chat! Go away Whachya doin? Please leave me alone. Wanna be friends? I hate you.
- 49. Dogpiles Service B Service A Service A Service A Service A x 100
- 50. Solution: Circuit Breaker Service B Service A Service A Where’s B? B’s sick. Wait 10 min. Hello! Go to Hell! Hey, B’s sick :(
- 51. Debugging hell Turns out, distributed systems are hard.
- 52. Debugging Hell Service 3 Service 1 Service 2 time:1428374783 source:service3 msg:”Error with payment!” Where?? Why?? Who??
- 53. Solution: Correlation IDs Service Service Service CID: 1234 CID: 1234 CID:1234 time:1427127483 source:service1 id:1234 msg:”Received request…” time:1427348748 source:service2 id:1234 msg:”Processing payment" time:1428374783 source:service3 id:1234 msg:”Error with payment!”
- 54. Solution: Correlation IDs 1. Tag all incoming requests with unique ID 2. Service saves ID for all incoming requests 3. Include that ID in all log lines, etc. 4. Tag new requests with that ID ServiceLogs ID ID ID ID Complexity: Must be done manually.
- 55. Missing Mock Servers Each consuming team has to create their own mocks and stubs.
- 56. Missing Mock Servers Service A Team A
- 57. Missing Mock Servers Service A Service B HTTP Team A Team B Product Surface Area
- 58. Missing Mock Servers Service A Service B HTTP Mock of Service A HTTP Team A Team B For each service… For each team…
- 59. Service A Service B HTTP Mock of Service A HTTP Team A Team B Better…Solution: Service Team Provides the Mock Team B still needs to know how to run Mock Service A
- 60. Solution: Service Team Owns the Client Service A Service B HTTP H TTP Team A Team B Client A MOCK=“true” Best… Mock of Service A Product Surface Area Both modes are tested in CI Team A can change the protocol as they see fit
- 61. Flying Blind
- 62. Solution: Graphs, alerts, pages.
- 63. Solution: Graphs, alerts, pages. Alerting Rules EnginePlatform Events OMGWAT???AppsAppsAppsAppsAppsAppsAppsAppsAppsAppsApps
- 64. Snowflakes
- 65. Snowflakes Ruby XSS exploit! BASH remote exploit!
- 68. Solution: Golden Image Golden OS Image Common Runtimes and Frameworks + Base Platform =
- 72. Solution: Predictable Pipelines Need to trust your tests, your platform, and your automation.
- 74. Operational Explosion! Operations block Development
- 75. Solution: AUTOMATE ALL THE THINGS!!!!! Form a team to build tools that enable developers to manage the system in an entirely automated way.
- 76. OMG ALLTHETHINGS???? App deployment, infrastructure provisioning, OS installation, configuration management, database provisioning, disaster recovery, application monitoring, HA, blue-green deployments, self-healing, scaling, runtime installation, log rotation, backups, security updates, database upgrades, application logs, system logs, database logs, continuous integration, continuous deployment, service discovery, monitor queue usage, security monitoring, hotspot detection, error monitoring, issue notification and escalation, virtual machine migration, shard rebalancing, circuit breaker monitoring, resiliency testing, database snapshots, flux capacitors, ion overdrive maintenance, change the oil, dog feeding, cat shooting, pig eating…
- 77. Time and Money Solution: AUTOMATE ALL THE THINGS!!!!!
- 79. In summary… Start boring and extract to services. Understand the hidden schemas. Amortize traffic with queues. Decouple through discovery tools. Contain failures with circuit breakers. Enable other teams through mockable clients. Kill your snowflakes. Automate your deployments. Build in operations tools from the beginning. Make use of a platform like Cloud Foundry.
- 80. Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations/cloud- anti-patterns