Microservices lessons from trenches
Download as PPT, PDF4 likes2,627 views
This document discusses lessons learned from implementing microservices in production environments. It outlines both the benefits of microservices, such as scalability and fault tolerance, as well as challenges, including loose coupling, greenfield development, and culture. Specific lessons include containing failures, using caching and logging for fault tolerance, building contracts to enable loose coupling, starting with a monolith then extracting services, and ensuring an automated testing and deployment culture. The key takeaway is that microservices are difficult to implement and the challenges must outweigh the benefits for an organization.
Microservices lessons from trenches
- 1. Microservices: lessons from the trenches Team Lead at ThoughtWorks Mehdi Khalili twitter: @MehdiKhalili http://www.mehdi-khalili.com/presentations/microservices-lessons- dddmel/ http://www.mehdi-khalili.com/microservices-lessons/
- 4. Benefits of microservices • Scalability • Loose Coupling • Fault Tolerance • Scalable Development • Technical Diversification • Continuous Delivery
- 7. Challenges • Fault Tolerance • Loose Coupling • Greenfield • Culture
- 10. 30 services with 99.9% up-time 43m downtime per service a month 1296m downtime a month one day of downtime a month
- 12. contain the failure Netflix: assumed broken components
- 13. Reading • caching • data duplication! • graceful degradation
- 14. Actions and events • guaranteed delivery • eventual consistency
- 15. be proactive about failure
- 16. The story of recycling app pool!
- 17. • monitoring endpoints • synthetic monitoring be proactive about failure
- 18. troubleshooting
- 19. The story of missing param!
- 20. • extensive logging • correlation id • log aggregation troubleshooting Correlation Id Service From Timestamp Message 996a3975-4996 Registration::Creat e nil 02 Aug 2014 19:55:15 AEST +10:00 bbbe64ab-c84c Cart::Get nil 02 Aug 2014 19:55:15 AEST +10:00 996a3975-4996 Accounting::Creat e Registration::Create 02 Aug 2014 19:55:16 AEST +10:00 996a3975-4996 Avatar::Create Registration::Create 02 Aug 2014 19:55:16 AEST +10:00 NullReferenc e
- 21. • extensive logging • correlation id • log aggregation troubleshooting Correlation Id Service From Timestamp Message 996a3975-4996 Registration::Creat e nil 02 Aug 2014 19:55:15 AEST +10:00 bbbe64ab-c84c Cart::Get nil 02 Aug 2014 19:55:15 AEST +10:00 996a3975-4996 Accounting::Creat e Registration::Create 02 Aug 2014 19:55:16 AEST +10:00 996a3975-4996 Avatar::Create Registration::Create 02 Aug 2014 19:55:16 AEST +10:00 NullReferenc e
- 22. • extensive logging • correlation id • log aggregation troubleshooting Correlation Id Service From Timestamp Message 996a3975-4996 Registration::Creat e nil 02 Aug 2014 19:55:15 AEST +10:00 bbbe64ab-c84c Cart::Get nil 02 Aug 2014 19:55:15 AEST +10:00 996a3975-4996 Accounting::Creat e Registration::Create 02 Aug 2014 19:55:16 AEST +10:00 996a3975-4996 Avatar::Create Registration::Create 02 Aug 2014 19:55:16 AEST +10:00 NullReferenc e
- 23. Loose Coupling
- 25. you need the big picture
- 27. you need the big picture beautiful diagrams!
- 28. you need the big picture Correlation Id Service From Timestamp Message 0a669422-5e44 Registration::Creat e nil 02 Aug 2014 19:55:15 AEST +10:00 bbbe64ab-c84c Accounting::Create User::AddAccount 01 Aug 2014 05:10:01 AEST +10:00 996a3975-4996 Accounting::Create Registration::Create 02 Aug 2014 19:55:16 AEST +10:00 e30e4e37-b1e1 Avatar::Create Registration::Create 02 Aug 2014 19:55:16 AEST +10:00 NullReferenc e
- 29. catch contract violation at build time not run-time
- 30. consumer provider consumer consumer driven contracts catch contract violation at build time
- 31. catch contract violation at build time
- 32. consumer provider consumer service libraries service lib service lib catch contract violation at build time
- 35. you need well defined service boundaries greenfield leads to unknown boundaries
- 38. always start with monolith extract the bits that would benefit from microservices
- 39. every greenfield project is a startup“a human institution designed to create a new product or service under conditions of extreme
- 40. Minimum Viable Product “that version of a new product which allows a team to collect the maximum amount of validated learning about
- 41. microservices is hard to implement and deploy and monitor
- 44. The Culture
- 46. nothing can save you from cultural failure
- 47. “our release process has 43 manual and approval steps” “fixing the bug took me 5 minutes, releasing it to prod took 3 weeks!” heavy processes
- 48. “test automation might work for others but our system is different.” “we have 30 developers and 85 testers!” no automation
- 49. “we need to stop obsessing about quality and focus on getting this out soon.” no respect for quality
- 50. “we need a DevOps team!” “shortcut” to happiness
- 51. test automation deployment automation infrastructure automation DevOps culture The Culture
- 52. Netflix: ” you code it, you'll deploy it and you'll be on pager-duty.”
- 53. How do you sell microservices? DO NOT
- 54. takeaways
- 55. Fault tolerance • Contain the failure • Caching, data duplication and graceful degradation • Guaranteed delivery and eventual consistency • Be proactive about failure • Monitoring • Use extensive logging
- 56. Loose coupling • You need the big picture • Self documenting application • Build-time contract verification • Consumer Driven Contracts • Service Libraries
- 57. Avoid greenfield microservices • Start with monolith • Learn your domain model • Microservices is NOT MVP
- 58. takeaway • Culture, culture, culture! • Test automation • Deployment automation • Infrastructure automation • DevOps culture
- 60. benefits vs overheads Scalability Operations & monitoring Loose coupling Contract violation Fault tolerance Fault isolation Scalable development Communication overhead Lower cognitive load Losing the big picture Tech diversification Too many things to learn Continuous delivery DevOps culture
- 61. do it to the extent its benefits outweigh overheads and challenges
- 62. Next time you want to do microservices • Do you actually need it? • Do you need it now? • Are the challenges worth the benefits? • Does it fit your CULTURE?
- 63. Resources
Editor's Notes
- #6: We see something awesome and we get really excited about it and think I want to be like that. I want to do that
- #13: Assume everything your system talks to is broken and write it so it can cope!
- #16: Don't wait for it to fail
- #18: to expose the error count, response times, up time, transaction per second simulate an action or path that a customer or end-user would take. functionality, availability, and response time measures
- #19: Don't wait for it to fail
- #20: We were getting exception on an endpoint because it was missing a param
- #21: Create on the edge service and pass it on to all services
- #27: We were missing the big picture
- #32: Promotes bad api design
- #33: fallbacks, caching Publish the lib on build time and verify the consumers on CI
- #37: Too chatty Had to deploy a few services each time Eventual consistency was a pain
- #38: Very chatty Transaction boundaries Req/Res
- #42: Fault isolation Eventual consistency Asynchrony Who’s in charge of auth? What happens when a new service is created?
- #44: Create the cheapest MVP
- #45: Doesn’t matter how technically good you are. You can’t win the cultural battle
- #46: This actually shows one of the bigger risks with microservices: when it hurts, it really hurts!!
- #52: Had to revert my unit tests!! Long manual release processes with sign-offs and approval gates Configuration drift Unfair on your ops team to monitor many services Is your dev team ready to be on-call?