The Road to Continuous Deployment
0 likes213 views
The document describes the process of migrating a legacy monolithic system called San Diego to a microservices architecture with continuous deployment. It discusses how the system was originally complex and difficult to work with. The approach taken was to use a strangler pattern, building API-first services for each domain object that could be deployed independently. This allowed deploying new features continuously using Docker containers, load balancing, and automation. The results were improved performance, velocity, and job satisfaction for the team. Lessons learned include gaining team acceptance for change and ensuring stability of the new systems.
The Road to Continuous Deployment
- 2. THE ROAD TO CONTINUOUS DEPLOYMENT - A CASE STUDY MICHIEL ROOK @michieltcs
- 4. THE SYSTEM - SAN DIEGO ▸ ... or the Big Ball Of Mud ▸ Large legacy monolith ▸ Generates significant income ▸ Slow & complex ▸ Technical debt @michieltcs
- 5. SAN DIEGO FRONTEND MYSQL DB SAN DIEGO BACKEND LOAD BALANCERS / VARNISH ITBANEN INTERMEDIAIR NATIONALEVACATUREBANK SAN DIEGO FRONTEND SAN DIEGO FRONTEND SAN DIEGO FRONTEND SAN DIEGO BACKEND SAN DIEGO BACKEND SAN DIEGO BACKEND MEMCACHE FTP EXT. SERVICES SOLR @michieltcs
- 6. THE SYSTEM - SAN DIEGO ▸ Infrequent, manual releases ▸ Fragile tests ▸ Frequent outages & issues ▸ Frustrated team ▸ Low confidence modifying existing code @michieltcs
- 7. GOALS ▸ Reduce issues ▸ Reduce cycle time ▸ Increase productivity ▸ Increase motivation
- 9. APPROACH ▸ Strangler pattern ▸ API first ▸ Services per domain object (job, jobseeker, ...) ▸ Migrate individual pages @michieltcs
- 11. APPROACH ▸ Services behind load balancers ▸ Access legacy db’s ▸ Continuous deployment ▸ Docker containers ▸ Frontends are services @michieltcs
- 13. DEV BUILD / TEST CONTINUOUS INTEGRATION @michieltcs
- 14. DEV BUILD / TEST ACCEPTANCE PRODUCTION CONTINUOUS DELIVERY @michieltcs
- 15. DEV BUILD / TEST STAGING PRODUCTION CONTINUOUS DEPLOYMENT @michieltcs
- 16. WHY CONTINUOUS DEPLOYMENT ▸ Small steps ▸ Early feedback ▸ Reduce cycle time ▸ Reduce risk ▸ Experiments! @michieltcs
- 18. ONLY COMMIT TO MASTER
- 19. EVERY COMMIT GOES TO PRODUCTION
- 20. PAIR PROGRAMMING
- 21. QUALITY GATES
- 22. FEATURE TOGGLES, A/B TESTS @michieltcs
- 23. DASHBOARDS
- 24. DEVOPS
- 27. CONTINUOUS TESTING UNIT TESTS INTEGRATION TESTS ACCEPTANCE TESTS UI TESTS SMOKE TESTS Cost Speed Exploratory testing Monitoring @michieltcs
- 28. PIPELINE AS CODE node { stage 'Run tests' sh "phpunit" sh "behat" stage 'Build docker image' sh "phing build" sh "docker build -t jobservice:${env.BUILD_NUMBER} ." sh "docker push jobservice:${env.BUILD_NUMBER}" stage 'Deploy acceptance' sh "ansible-playbook -e BUILD=${env.BUILD_NUMBER} -i acc deploy.yml" stage 'Deploy production' sh "ansible-playbook -e BUILD=${env.BUILD_NUMBER} -i prod deploy.yml" } @michieltcs
- 29. DEPLOYING PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER docker pull @michieltcs
- 30. DEPLOYING PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER docker run @michieltcs
- 31. DEPLOYING PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER wait_for: port=8080 delay=5 timeout=15 @michieltcs
- 32. DEPLOYING PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER uri: url: http://localhost:8080/_health status_code: 200 timeout: 30 @michieltcs
- 33. DEPLOYING PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER template: src=haproxy.cfg.j2 dest=/etc/haproxy/haproxy.cfg service: name=haproxy state=reloaded @michieltcs
- 34. DEPLOYING PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER template: src=haproxy.cfg.j2 dest=/etc/haproxy/haproxy.cfg service: name=haproxy state=reloaded @michieltcs
- 35. DEPLOYING PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER docker stop docker rm @michieltcs
- 37. FEEDBACK!
- 38. RESULTS
- 39. RESULTS ▸ Total build time per service < 10 minutes ▸ Significantly improved page load times ▸ Improved audience stats (time on page, pages per session, session duration, traffic, seo ranking, etc) ▸ Increased confidence and velocity ▸ Experimented with new tech/stacks (angular, jvm, event sourcing) ▸ More fun @michieltcs
- 40. LESSONS LEARNED ▸ Team acceptance ▸ Change is hard ▸ Mentality / discipline ▸ Docker stability/orchestration ▸ Issues with traffic between Amazon <-> on-premise datacenter @michieltcs
- 41. LESSONS LEARNED ▸ Experience with new tech ▸ Stability of build pipelines ▸ Business alignment ▸ Limit feature toggles ▸ Keep focus on replacing legacy application @michieltcs
- 44. Turn-key Continuous Deployment Zero downtime deployments Modern, autoscaling infrastructure with built-in monitoring Pipeline in five minutes
- 45. THANK YOU! @michieltcs / michiel@make.io Turn-key Continuous Deployment