Continuous delivery in practice (public)
Download as PPTX, PDF0 likes972 views
Continuous delivery practices were implemented for an RTB project at Kenshoo to allow for faster development and delivery. This included having a single code branch, comprehensive testing at the unit, integration, behavioral, and staging levels, keeping builds stable, ensuring the master branch is always deployable, conducting rigorous code reviews, using real-time feedback, managing upgrades carefully, and A/B testing. It is important to constantly monitor builds and collect feedback to own the process.
Continuous delivery in practice (public)
- 1. Continuous Delivery In Practice Lessons from Kenshoo’s RTB project
- 2. Who, What, Where Tzach Zohar: ● System Architect ● tzach.zohar@kenshoo.com Kenshoo: ● Founded 2006 ● Online Marketing Technology ● ~450 employees ● 12 World Wide locations
- 3. Continuous Delivery: Why bother? Faster development, delivery and feedback makes our clients and us happier. The “us” alone is worth it.
- 4. Continuous Delivery: Why bother? Better suited product Responsiveness Less waste Higher quality
- 5. Background: RTB Project ● ~1.5 years ● ~4 developers ● ~Dozens of paying clients ● ~50 servers (AWS) ● ~1.5M requests per minute ● ~7ms average response time ● ~99.9% availability ● ~5-10 deployments / week
- 6. How?
- 7. 1. The Obvious ● Single branch ● Full, Fast, Reliable coverage ● Full deployment automation ● Fast feedback ● ABCD - Always Be Continuously Deploying
- 8. ● Unit: complete functional coverage ● Integration: with external systems - thin! ● Behavioral: we use Cucumber ● Staging: verify actual server upgrade 2. Four-Layer Test Suite
- 10. Staging: verify compatibility of new build with other components’ production builds 2. Four-Layer Test Suite
- 11. 3. Keep Builds Stable Do not overlook a test that “sometimes fails”, trusting build status is crucial
- 12. 3. Keep Builds Stable Be Suspicious of: ● Random data tests ● Asynchronous tests ● Integration tests
- 13. 4. Master Is Always Shippable On every commit? Not Quite: Use GitHub Pull Requests “Merge” == Build and Deploy credit: tal.salmona@kenshoo.com
- 14. 5. Rigorous Code Reviews ● Remember “merge” means “deploy”! ● Insist on proper coverage ● Insist on code cleanliness ● Insist on consistent design ● Insist!
- 16. 6. Real-Time Feedback Detect issues immediately and visually
- 17. 7. Keep Upgrade in Mind (1) Use the “Parallel Change” pattern when changing cross-node APIs / Data 1. Write: old Read: both 2. Write: new Read: both 3. Write: new Read: new deploy deploy
- 18. 8. Keep Upgrade in Mind (2) Verify backward compatibility in tests
- 19. 9. A/B Testing Apply new features to a limited user-group Measure business results per-group (Not by branching)
- 20. 10. Own It Constantly check builds Constantly collect feedback Constantly check monitors Answer the phone at 3am
- 21. 10. Own It
- 22. That’s It.
- 23. Appendix A. Partial Tool List Testing: JUnit, Cucumber, Nose Build / CI: Jenkins, Gradle, JaCoCo Code Review: GitHub Provisioning: Puppet Deployment: Fabric, boto Monitoring: Metrics, Graphite
- 24. Appendix B. Are You Ready? Unit Coverage > 90%? Good Staging Tests? Informative Monitors? Builds Are Kept Green? No API Breaking Changes? Rigorous Code Reviews? Support Has Your Phone Number? Do You Own it? Not Ready No Yes credit: tal.salmona@kenshoo.com