Continuous Delivery in Practice (extended)
Download as PPTX, PDF3 likes1,273 views
The document discusses continuous delivery (CD) principles and practices within Kenshoo's real-time bidding (RTB) project, emphasizing its importance in delivering incremental changes to users. It outlines a practical approach to CD, including ten field-tested tips such as full deployment automation, rigorous code reviews, and maintaining stable builds. The document also highlights the technical and organizational setup of the RTB project, which involved a small team and a robust infrastructure to achieve high availability and responsiveness.
Continuous Delivery in Practice (extended)
- 1. Continuous Delivery In Practice Lessons from Kenshoo’s RTB project
- 2. Who, What, Where Tzach Zohar: ● System Architect ● tzach.zohar@kenshoo.com Kenshoo: ● Founded 2006 ● Online Marketing Technology ● >500 employees ● 12 World Wide locations
- 3. Agenda ● Continuous Delivery: What? Why? ● RTB Project ● How: 10 Field Tested Tips ● The Process ● Appendices
- 4. Continuous Delivery: Definition(s) “Continuous Delivery (CD) is a design practice …blah blah blah… Techniques such as automated testing, continuous integration …blah blah blah... resulting in the ability to rapidly, reliably and repeatedly push out enhancements ...blah blah blah.” - Wikipedia
- 6. Continuous Delivery: Definition(s) “Continuous delivery is a set of principles and practices to reduce the cost, time, and risk of delivering incremental changes to users.” - Jez Humble
- 7. Continuous Delivery: Definition(s) “Continuous Delivery is a software development discipline where you build software in such a way that the software can be released to production at any time” - Martin Fowler
- 8. Continuous Delivery: Why bother? “Our highest priority is to satisfy the customer through early and continuous delivery of valuable software” First principle of the Agile Manifesto
- 9. Continuous Delivery: Why bother? Better suited product Responsiveness Less waste Higher quality Simplicity Recommended Further Reading on ThoughtWorks
- 10. Background: RTB Project ● ~1.5 years ● ~3 developers, 1 PM, 0.5 Ops (no QA) ● ~Dozens of paying clients ● ~50 servers (AWS) ● ~1.5M requests per minute ● ~7ms average response time ● ~99.9% availability
- 11. Background: RTB Project Frontend Cluster Highly available, high throughput ~20 node cluster Backend Single node, internal APIs FBX Facebook RTB API Reporting Cluster Elastic Map Reduce (EMR) on-demand 16-node cluster Cassandra Cluster Highly available, high throughput ~24 node cluster S3 Raw traffic logs
- 12. Background: RTB Project ~5-10 deployments / week
- 13. How?
- 14. 1. The Obvious ● Single branch (details later) ● Full, Fast, Reliable coverage ● Full deployment automation ● Fast feedback ● ABCD - Always Be Continuously Deploying
- 15. ● Unit: complete functional coverage ● Integration: with external systems - thin! ● Behavioral: we use Cucumber ● Staging: verify actual server upgrade 2. Four-Layer Test Suite
- 17. Staging: verify compatibility of new build with other components’ production builds 2. Four-Layer Test Suite
- 18. 3. Keep Builds Stable Do not overlook a test that “sometimes fails”, trusting build status is crucial
- 19. 3. Keep Builds Stable ● Random data tests ● Asynchronous tests ● Integration tests Be suspicious of:
- 20. 4. Master Is Always Shippable On every commit? Not Quite We follow the “GitHub Flow”: Local Master Local Feature Branch Master Feature Branch 1. pull 3. push 2. checkout 4. Merge
- 21. 4. Master Is Always Shippable “Merge” == Build and Deploy credit: tal.salmona@kenshoo.com
- 22. 5. Rigorous Code Reviews ● Because “merge” means “deploy”! ● Insist on proper coverage ● Insist on code cleanliness ● Insist on consistent design ● Insist!
- 24. 6. Real-Time Feedback Detect issues immediately and visually
- 25. 7. Keep Upgrade in Mind (1) Use the “Parallel Change” pattern when changing cross-node APIs / Data 1. Write: old Read: both 2. Write: new Read: both 3. Write: new Read: new deploy deploy
- 26. 8. Keep Upgrade in Mind (2) Verify backward compatibility in tests
- 27. 9. A/B Testing Apply new features to a limited user-group Measure business results per-group (Not by branching)
- 28. 9. A/B Testing Splitting into groups correctly is important
- 29. 9. A/B Testing It’s easy to mess up (neglecting biases, wrong grouping, wrong comparison methods) This excellent talk by LivePerson’s Shlomo Lahav helped us a lot
- 30. 10. Own It Constantly check builds Constantly collect feedback Constantly check monitors Answer the phone at 3am
- 31. 10. Own It
- 32. That’s It.
- 33. The Process ● Greenfield? That’s easy: ○ Start with deployment and build ○ Deploy a Hello World application ○ Every new feature is test-covered
- 34. The Process (RTB) 1. Increase Unit+Integration Coverage Create naive deployment Automation Create monitoring Manual Staging tests 2. Automated staging Downtime eradicated Manual (but often) deployment trigger 3. Autopilot - deploy upon commit ~ 9 Months ~ 3 Months
- 35. Appendix A. Partial Tool List Testing: JUnit, Cucumber, Nose Build / CI: Jenkins, Gradle, JaCoCo Code Review: GitHub Provisioning: Puppet Deployment: Fabric, boto Monitoring: Metrics, Graphite
- 36. Appendix B. Are You Ready? Unit Coverage > 90%? Good Staging Tests? Informative Monitors? Builds Are Kept Green? No API Breaking Changes? Rigorous Code Reviews? Support Has Your Phone Number? Do You Own it? Not Ready No Yes credit: tal.salmona@kenshoo.com