Continuous Integration @ Haptik
- 1. Prateek Gupte Director of Engineering Continuous Integration @ Haptik Our journey to hourly releases
- 2. Everyone’s Personal Assistant to get things done. • Highest rated App in India • 40,000 conversations per day • 1 million messages processed per day • A dozen+ API integrations 100,000+ user reviews
- 3. Machine Learning Scientists Full Stack Developers Apps (iOS + Android + Backend) QA Haptik Team Structure ProductTeam1 ProductTeam2 ProductTeamn …
- 4. Git Flow
- 5. Problems faced • Bi Monthly releases • Batched releases • Something always breaks • Monolithic apps are tough; Microservices can be tougher • Out of sync environments
- 6. • Zero downtime • High frequency, low risk deployments • Automate changes in infrastructure • Minimise integration problems • Manage dependencies • packages • OS level dependencies • Empower engineers to release when they want
- 7. DevelopmentDevelopment Staging Production Development Jenkins Configuration Files for infrastructure Infra Configuration Haptik Admin + Mogambo Haptik Admin + Mogambo develop master S3
- 9. Tests are crucial • You can’t have CI without any tests • Quantifying the value of tests • Legacy Code • Writing tests backwards • Writing tests v/s writing code
- 10. It’s all about the data • Making data available to everyone • Chat Data in Mongo • Configuration Data in MySQL • Environment Configuration in S3 • ML Models • Moving data backwards and forwards across environments • Conflicts in data
- 11. • Blue Green Deployments for Spot Servers • Pay only if you save • Some pretty amazing cost savings
- 12. Where we went wrong • The chaos of the staging environment • CI is a culture not a set of tools • Our experiments with a preprod environment • “Do we have confidence to release this version of software to Production?”
- 13. • Where are the tests? • Unit Tests • API Tests • Integration Testing • Test Coverage • Maintain consistency across environments • Free the Data! • Reduce QA time - Catch bugs early 2
- 14. DevelopmentDevelopment Staging Production Development Jenkins Test Environment Configuration Files for infrastructure Infra Configuration Application Configuration, Models, Chat Flows and copies Haptik Admin + Mogambo Haptik Admin + Mogambo feature PRbuild develop master S3
- 15. Mobile CI • Staging builds can point to any environment • Daily Develop Builds sent to the entire team • Integration tests run every night on develop build • Unit tests that run on every PR to block merges
- 16. Things that worked 👍 • Teams have their own releases cycles • Teams ship faster and with better quality • Manual overhead and co-ordination removed • Apps are hardened every day
- 17. It’s not about the tools It’s about the culture
- 18. Road ahead • Reach > 80% Coverage across our system • Enforce coverage on all Pull Requests • API Tests • Continuous deployments
- 19. Haptik Plug We’re Hiring! • Android Engineer • Full Stack Engineer • Senior Machine Learning Scientist • DevOps Engineer • QA Engineer