Amazon EKS multi-cluster gitops-bridge
1 like506 views
The document discusses Amazon EKS multi-cluster topologies using a GitOps approach. It describes key challenges with multi-cluster management and proposes using GitOps with the GitOps Bridge pattern to define infrastructure and Kubernetes resources declaratively. This allows enforcing standards across clusters and automating deployments. It provides examples of cluster topologies like standalone, centralized hub-spoke, and discusses managing teams, workloads, addons and configurations through this approach.
Amazon EKS multi-cluster gitops-bridge
- 1. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. A M A Z O N E K S M U L T I - C L U S T E R T O P O L O G I E S Amazon EKS Multi-Cluster The Gitops Bridge Pattern Carlos Santana Sr. Kubernetes Solutions Architect AWS CNCF Ambassador Isaac Mosquera Principal Containers Specialist AWS Christina Andonov Sr. Specialist Solutions Architect AWS
- 2. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. 1. What are customers building 2. Key challenges 3. Proposed solution 4. How we solved the problem 5. How to use Agenda
- 3. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. What are customers building AI/ML Autonomous Vehicles Robotics Modeling, Training, and Inference Legacy Apps .NET Apps Homegrown Apps Monolith Analytics Streaming MapReduce Batch Web Static Dynamic Backend DB IoT Mobile
- 4. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Enforce security standards and best practices across clusters to automate deployments Define boundaries between multiple teams Provision multiple workloads at scale Cluster management Team management Workload management Install add-ons and their dependencies Add-on management Configuration management Automate configuration and upgrade lifecycle from a single source of truth Key challenges
- 5. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Proposed solution G I T O P S Build Test Scan Operate/Fix Deploy/Verify Observe/Alert Immutability Firewall Git becomes the single source of truth for the system’s desired state, enabling reproducible automated deployments, cluster management, and monitoring.
- 6. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. What is GitOps P R I N C I P L E S A system managed by GitOps must have its desired state expressed declaratively Desired state is stored in a way that enforces immutability, versioning and retains a complete version history Software agents continuously observe actual system state and attempt to apply the desired state Software agents automatically pull the desired state declarations from the source Enforces Consistency Reduces Business Risk Enhances Auditability Boosts Security
- 7. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Cluster Management
- 8. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Multi-cluster challenges Enforcing best practices across clusters Consistent cluster lifecycle management Supporting multiple teams with guardrails in place Easily onboarding new applications
- 9. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Multi-Cluster Kubernetes 9 Cloud (prod) Cloud (pre-prod) 1.22 1.23 1.24 1.25 1.26
- 10. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. EKS Blueprints
- 11. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Cluster management EKS Blueprints AWS Cloud Virtual private cloud Internet gateway Public subnet Private subnet Amazon EKS NAT gateway AWS account Instances Instances Instances Role Permissions
- 12. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Infrastructure as Code with Terraform and CDK Based on AWS best practices and recommendations Integrated with popular K8s tools and services Fully extensible and customizable Amazon EKS Blueprints An open-source framework that allows you to configure and deploy complete Amazon EKS clusters across accounts and Regions
- 13. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Cluster Deployment
- 14. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Challenges with IaC and GitOps • Terraform § Great for setting up underlying infrastructure § Not design to work with Kubernetes § Imperative, can lead to drift configuration • GitOps § Designed for Kubernetes resources § Declarative and convergence, minimizing drift § Limited scope, focused on app deployment rather than infrastructure
- 15. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. GitOps Bridge: IaC -> GitOps IaC Internet gateway Virtual private cloud (VPC) NAT gateway Private subnet Public subnet AWS KMS key Role Amazon EKS Security group Kubernetes Gatekeeper Karpenter Cert-manager Instances GitOps cluster.yaml
- 16. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. GitOps Bridge: IaC -> GitOps IaC Internet gateway Virtual private cloud (VPC) NAT gateway Private subnet Public subnet AWS KMS key Role Amazon EKS Security group Instances GitOps cluster.yaml Kubernetes Cert-manager Gatekeeper Karpenter cluster config Addons Apps
- 17. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. GitOps Bridge: IaC -> GitOps IaC Internet gateway Virtual private cloud (VPC) NAT gateway Private subnet Public subnet AWS KMS key Role Amazon EKS Security group GitOps cluster.yaml Kubernetes Cert-manager Gatekeeper Karpenter cluster config Addons Apps Instances
- 18. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. GitOps Bridge: IaC -> GitOps GitOps cluster.yaml Kubernetes Cert-manager Gatekeeper Karpenter cluster config Addons Apps Instances ack terraform kops ansible pulumi cdk capa crossplane
- 19. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Cluster Topologies
- 20. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Standalone/distributed Namespace Tenant AWS account Amazon EKS Tenant AWS account Amazon EKS Namespace On premises Kubernetes Namespace On premises Kubernetes Namespace Full ArgoCD UI/CLI API Server Redis Server Repo Controllers Addons + Apps
- 21. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Centralized/Hub-Spoke (Push) Central Amazon EKS cluster Central AWS account Namespace Tenant AWS account On premises Amazon EKS Tenant AWS account Kubernetes On premises Kubernetes Amazon EKS Addons + Apps Full ArgoCD UI/CLI API Server Redis Server Repo Controllers
- 22. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Centralized/Hub-Spoke (Shared) Central Amazon EKS cluster Central AWS account Namespace Tenant AWS account On premises Amazon EKS Tenant AWS account Kubernetes On premises Kubernetes Amazon EKS App-2 repo App-1 repo App-4 repo Addons App-3 repo Full ArgoCD UI/CLI API Server Redis Server Repo Controllers
- 23. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Centralized/Hub-Spoke (Agent) Central Amazon EKS cluster Central AWS account Namespace Tenant AWS account On premises Amazon EKS Tenant AWS account Kubernetes On premises Kubernetes Amazon EKS App-2 repo App-1 repo App-4 repo App-3 repo Core ArgoCD UI/CLI API Server Redis Server Repo Controllers https://argo-cd.readthedocs.io/en/stable/operator-manual/core https://akuity.io/blog/reducing-argocd-operational-burden https://github.com/open-cluster-management-io/argocd-pull-integration Full ArgoCD UI/CLI API Server Redis Server Repo Controllers Addons
- 24. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Teams
- 25. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Separation of concerns I would like to standardize the deployment process for application teams while enforcing organizational standards Platform team Development teams I would like to have full control of my application and its dependencies deployment lifecycle
- 26. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Identity & Access Management Policy Management Namespace as a Service Multi-Team Considerations
- 27. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon EKS (Prod) Teams management Frontend Team Backend Team Platform Team Frontend Repository Platform Repository Frontend ArgoCD Projects Platform ArgoCD Projects Backend Repository Backend ArgoCD Projects Amazon EKS (Staging) workloads addons
- 28. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Add-Ons
- 29. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Add-Ons Security Cilium Gatekeeper Kyverno Observability Prometheus Fluent Bit OTEL Reliability Karpenter Autoscaler Keda Delivery ArgoCD Flux Crossplane Others
- 30. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Add-ons management G I T O P S B R I D G E C O N F I G U R A T I O N Control Plane Addons App Of ApplicationSet Addon-1 charts/ environments/ Addon-2 App Of ApplicationSets Addon-1 ApplicationSet Addon-2 ApplicationSet clusters/ Addon-1 Application Addon-2 Application Platform repo Platform team Amazon EKS AWS account OTEL
- 31. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Configuration Management
- 32. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon EKS (Central) AWS account (Central) AWS account (Tenant A) Amazon EKS (Staging) Amazon EKS (Prod) ui v1.0 dynamodb v1.0 ui v1.0 dynamodb v1.0 frontend backend frontend backend apps ui base prod dynamodb/ Apps repo Apps teams staging clusters/ base prod staging Configuration management E N V I R O N M E N T S ( S T A G I N G V S . P R O D )
- 33. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Configuration management D E P L O Y A N D S C A L E Amazon EKS (Central) AWS account (Central) AWS account (Tenant A) Amazon EKS (Staging) ui v2.0 dynamodb v1.0 frontend backend apps ui base prod dynamodb/ Apps repo Apps teams staging clusters/ base prod staging Amazon EKS (Prod) ui v1.0 dynamodb v1.0 frontend backend
- 34. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Configuration management A W S S E R V I C E S W I T H G I T O P S Amazon EKS (Central) AWS account (Central) AWS account (Tenant A) Amazon EKS (Staging) Amazon EKS (Prod) ui v2.0 ui v2.0 frontend frontend Amazon DynamoDB (Staging) apps ui base prod clusters/ Apps repo staging Apps teams Amazon DynamoDB (Production) Control Plane addons aws ack clusters/ Platforms repo environment/ Platform team
- 35. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. GitOps Bridge: ApplicationSet (Addon versions) version in dev is 1.6.0 version in staging is 1.5.5 version in production is 1.5.4 Cluster opt-in for the addon Chart name and repo in a single place Merge generator Prevent Outages
- 36. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. GitOps Bridge: ApplicationSet (overrides) Metadata based on IaC Namespace based on IaC Override values files Value files in git
- 37. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Resources https://github.com/gitops-bridge-dev https://aws-ia.github.io/terraform-aws-eks-blueprints/patterns/gitops-getting-started-argocd/ https://aws-ia.github.io/terraform-aws-eks-blueprints/patterns/gitops-multi-cluster-hub-spoke-argocd/
- 38. Thank you! © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.