Modern CI/CD in the microservices world with Kubernetes
4 likes888 views
The document discusses modern CI/CD practices in a microservices architecture using Kubernetes, emphasizing the need for optimized deployment pipelines and compatibility management. Key steps include dockerizing microservices, defining a versioning policy, implementing system-level and deployment pipelines, and leveraging Kubernetes for efficient resource management. The document also highlights the importance of testing and quality gates to ensure successful deployments and reduce chaos in development environments.
Modern CI/CD in the microservices world with Kubernetes
- 1. Modern CI/CD in the microservices world with Kubernetes Mikalai Alimenkou @xpinjection https://t.me/xpinjection_channel https://xpinjection.com
- 4. Disclaimer This talk is based on personal experience
- 5. Hello world samples on conferences
- 6. Real world is a little bit more complex
- 7. CI/CD of the son of your mother’s friend After moving to its own cloud, Amazon engineers deploy code every 11.7 seconds, on average reducing both the number and duration of outages at the same time.
- 8. Why this is not your case? Product is developed in iterative way (Scrum, 2 weeks) Engineering culture is far from perfect Level of developers is not so high No big investments in test automation at all levels Engineering practices are not used on regular basis Legacy services or dependencies Business is not ready to continuous delivery … many more reasons
- 9. Chaos deployment in DEV/QA QA/DEV environment always contain latest versions of all microservices Deployment on QA/DEV is done continuously by different dev teams when something is ‘READY’ Compatibility of microservices is not verified There is no way to rollback on specific system version QA life looks like a hell (random issues, non- reproducible results, 500 HTTP codes everywhere, chaos testing)
- 10. Does it look familiar to you?
- 11. Old monolith mindset doesn’t work
- 12. Microservices has their own lifecycles
- 13. Difference between CI and CD
- 14. Step #1: Dockerize your microservices Immutable infrastructure principle Artifacts promotion with tags Unified blocks for CD stage, no dependency on technical stack Simplified deployment and environment management Common configuration approach Efficient resources utilization
- 16. Step #2: Specify versioning policy Semantic versioning as a basis Major version for API breaking changes or major service release Minor version corresponds to current dev iteration Patch version is reserved for hot fixes Commit hash as suffix to snapshot versioning and tracing Build date as an additional optional suffix
- 17. CI pipeline for each microservice QUALITY GATES: - Unit tests - Integration tests - Static code analysis - API tests - Contract tests - Security checks - … FULL VERSION: 2.3.0-350fca-20200218
- 18. What is shippable version of the system?
- 19. Step #3: Introduce compatibility sets Store full list of compatible microservices as dedicated artifact VCS provides historical timeline, rollback, full tracing Used as primary artifact for Continuous Delivery Promoted between different environments in CD pipeline (dev, qa, stage, prod) Add middleware versions for full compatibility set
- 20. Compatibility set in properties file Clear logical versions management independent from concrete deployment technologies Flexible usage for different purposes Separation between deployment artifacts versioning Customization step before deployment
- 21. Compatibility set in Helm/helmfile Single versioned storage for environment configuration Ability to specify compatibility on configuration level The same toolset is used for all environments Additional level of abstraction around K8S manifests Helm is needed as an additional component
- 22. Step #4: Implement system level pipeline Triggered when new candidate is built Uses latest compatible set as a baseline Overrides versions for ALL pending candidates Creates new environment for system deployment Deploys all microservices according to their versions Waits for all health checks to become OK Runs system level quality gates Stores new compatibility set in case of SUCCESS Retag and promote artifacts for the next stage
- 23. System pipeline for compatibility checks QUALITY GATES: - Health checks - Smoke API tests - Smoke UI tests - Smoke system tests - Smoke security tests - … ADDITIONAL TAGS: qa-ready, set-283gpt
- 24. Example from the real life USER SERVICE 1.0.3-abc EVENT SERVICE 1.0.0-cde SCHEDULE SERVICE 0.9.1-fgh COMPATIBLE SET 1.0.3-prs NEW CANDIDATE 1.0.0-cde 1.0.3-prs 0.9.1-fgh FAILED SYSTEM BUILD 1.0.1-tuv FIX APPLIED 1.0.3-prs 0.9.1-fgh SUCCESS SYSTEM BUILD 1.0.1-tuv 1.0.3-prs 1.0.1-tuv 0.9.1-fgh COMPATIBLE SET
- 25. System pipeline: important notes If pipeline fails, then candidate is not approved and it's artifact is invisible for other environments, but pending for the next pipeline run Every build takes ALL available pending candidates Different concurrency strategies may be applied: • single build with lock on compatibility set • concurrent builds with “last win” approach • concurrent builds with optimistic locking in VSC
- 26. We are almost GitOps ready now!
- 27. New tools become available for CD
- 28. Step #5: Implement deployment pipeline Triggered manually, automatically by successful system pipeline build or by specific schedule Predefined set of environments (dev, qa, stage, prod) New environment may be created with specified or generated name Only appropriate compatible sets may be deployed Environment TTL could be set manually or smart default value is applied
- 29. Full CI/CD cycle
- 30. FEATURE BRANCH PIPELINE: - Environment name equals to feature branch name - Reused to speed up development - DEV compatibility set is used as a baseline - Artifacts on microservices pipeline are tagged with branch name - For each microservice latest available version from the branch with the same name is used as candidate PULL REQUEST PIPELINE: - Triggered on each PR - Special flag to activate system level quality gates - DEV compatibility set is used as a baseline - Only light system level quality gates are run - No compatibility sets updated
- 32. It is time for Kubernetes to show up!
- 33. Step #6: Speed up environments in K8S Create dedicated namespace for each environment Configure minimal limits and scale factors everywhere Share common external heavy resources (DB, Elasticsearch, Redis, Kafka, etc.) Invest in performant K8S nodes Use prepopulated images with data to speed up Configure common observability tools Reuse environments where possible with smart TTLs Destroy environments asynchronously
- 34. Use full power of K8S namespaces COMMON Glossary Service Redis Cache FB-243 User Service Reuse from DEV STAGE User Service Event Service Schedule Service Reuse from DEV QA User Service Event Service Schedule Service DEV User Service Event Service Schedule Service RDBMS Kafka Cluster Elasticsearch
- 35. How to share common resources? It is really important to follow cloud-native 12 factors! Redis via key prefixes or databases RDBMS via databases or schemas Kafka via common topics prefix Elasticsearch via indexes
- 36. Now developers could use Skaffold continuously QC is shifted as left as possible Several teams could develop in common environment
- 37. Step #7: Start using K8S native CD tools Dockerize tests with required test environment Execute tests as K8S jobs Manage test infrastructure in K8S Choose suitable K8S native CD tool
- 38. Summary and take aways Dockerize your microservices Specify versioning policy Introduce compatibility sets Implement system level pipeline Implement deployment pipeline Speed up environments in K8S Start using K8S native CD tools Enjoy your life in amazing CI/CD world!