Scaling production grade EKS Multi-Cluster environments using GitOps
Download as PPTX, PDF1 like200 views
The document discusses strategies for scaling production-grade Kubernetes multi-cluster environments using GitOps practices, emphasizing the orchestration and management of workloads, security, and configuration. It outlines the importance of defining boundaries between teams, automating deployments, and harnessing tools within the Kubernetes ecosystem to enhance operational efficiency. Additionally, it highlights GitOps principles for maintaining an immutable and version-controlled desired state of the system, ultimately facilitating reliable and reproducible deployments.
Scaling production grade EKS Multi-Cluster environments using GitOps
- 1. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Scaling production grade Kubernetes Multi-Cluster environments using GitOps Rodrigo Bersa EKS Specialist Solutions Architect AWS Carlos Santana EKS Specialist Solutions Architect AWS KCD Washington, DC
- 2. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. I have chosen to use Kubernetes – now what?
- 3. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. What customers are building Apps and services Mobile IoT Static websites Complex web apps .NET apps Legacy homegrown Linux apps Monoliths Autonomous vehicles (object tracking, sensor fusion) Robotics (vision, grasping, motion control) Modeling, training, and inference Real-time MapReduce Batch
- 4. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Enforce security standards and best practices across clusters to automate deployments Define boundaries between multiple teams Provision multiple workloads at scale Cluster management Team management Workload management Install add-ons and their dependencies Add-on management Configuration management Automate configuration and upgrade lifecycle from a single source of truth Challenges and Goals
- 5. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. CNCF Cloud Map “There is no shortage of amazing tooling in the K8s ecosystem, but there is no guide for how to put all the tools together”
- 6. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Kubernetes Journey Choose an orchestrator Decisions
- 7. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Kubernetes Journey Choose an orchestrator Data Plane Compute Decisions
- 8. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Kubernetes Journey Choose an orchestrator Data Plane Compute Cluster Addons Decisions
- 9. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Kubernetes Journey Choose an orchestrator Data Plane Compute Cluster Addons Decisions Day 2 Operations
- 10. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Cluster add-ons Security Cilium Gatekeeper Kyverno Observability Prometheus Fluent Bit OTEL Reliability Karpenter Autoscaler Keda Delivery ArgoCD Flux Crossplane Other
- 11. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Principle of GitOps A system managed by GitOps must have its desired state expressed declaratively Desired state is stored in a way that enforces immutability, versioning and retains a complete version history Software agents continuously observe actual system state and attempt to apply the desired state Software agents automatically pull the desired state declarations from the source Enforces Consistency Reduces Business Risk Enhances Auditability Boosts Security
- 12. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. GitOps Continuously Reconcile Build Test Scan Operate/Fix Deploy/Verify Observe/Alert Immutability Firewall Git becomes the single source of truth for the system’s desired state, enabling reproducible automated deployments, cluster management, and monitoring.
- 13. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Secure environments
- 14. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Private environments Corporate datacenter Kubernetes Cluster Build Infrastructure deployment Corporate Network
- 15. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Public repositories Corporate datacenter Kubernetes Cluster Build Deploy Corporate Network Infrastructure deployment
- 16. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Private repositories Corporate datacenter Kubernetes Cluster Repository Scan Scan Store Deploy Corporate Network Build Pull Grype Clair Infrastructure deployment
- 17. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Trusted repositories Corporate datacenter Kubernetes Cluster Repository Scan Scan Store Deploy Corporate Network Build Grype Clair Infrastructure deployment Pull Image replication
- 18. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 19. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Cloud environment AWS Cloud Amazon Inspector Amazon EKS Amazon ECR VPC Private subnet VPN connection Scan Deploy AWS Direct Connect Secured Network Endpoints AWS PrivateLink Store Build Infrastructure deployment Image replication AWS CodeCommit
- 20. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Team management
- 21. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Identity & Access Management Policy Management Namespace as a Service Multi-team considerations
- 22. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Separation of concerns Platform engineers Platform builders – build and integrate tools that provision, manage and secure the cloud computing infrastructure Software engineers Application builders – free to focus on building applications that deliver business value to customers
- 23. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Teams management (k8s) Dev Team A Dev Team B Platform Team Audit Team Kubernetes Cluster Control Plane Data Plane Instances Instances Developer RBAC Temp RBAC Admin RBAC
- 24. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Teams management (cloud) Developer Role Admin Role Dev Team A Dev Team B Platform Team Audit Team Temporary Role VPC Availability Zone 1 Availability Zone 2 Managed Node Group Karpenter “Groupless” Amazon EKS AWS Cloud
- 25. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Dev/Test Cluster Teams management (ArgoCD) Dev Team A Dev Team B Platform Team Audit Team Apps ArgoCD Projects Apps Repository Audit ArgoCD Project Policy Repository Platform ArgoCD Project Platform Repository Production Cluster workloads policies addons
- 26. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Automated cluster deployment
- 27. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Automated deployment (k8s) Build Git push IaC repository Trigger IaC Pipeline Control Plane Data Plane Instances Instances Dev/Test Cluster Control Plane Data Plane Instances Instances Production Cluster Dev Team A Dev Team B Platform Team Audit Team Corporate datacenter Teams Repository Scanning Pull Corporate Network Push
- 28. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Automated deployment (Cloud) Build Git push IaC repository Trigger Dev Team A Dev Team B Platform Team Audit Team Teams Scan Pull Push IaC Pipeline AWS Cloud Production account Dev/Test account Availability Zone 2 Availability Zone 2 Amazon EKS Managed Node Group Karpenter “Groupless” Amazon EKS Managed Node Group Karpenter “Groupless” Amazon EKS VPC VPC Availability Zone 2 Availability Zone 2 VPC Managed Node Group Karpenter “Groupless” Development tools Secured Network
- 29. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Infrastructure as Code with Terraform and CDK Based on AWS best practices and recommendations Integrated with popular K8s tools and services Fully extensible and customizable Amazon EKS Blueprints An open-source framework that allows you to configure and deploy complete Amazon EKS clusters across accounts and Regions
- 30. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Cluster creation with Amazon EKS Blueprints • Infrastructure as Code (IaC) • AWS CDK • HashiCorp Terraform • Addons • OSS • AWS • Partner • GitOps • ArgoCD - (New GitOps-Bridge) EKS Blueprints
- 31. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Multi-Cluster Management with GitOps (Topologies)
- 32. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Standalone/distributed GitOps Namespace Tenant AWS account Amazon EKS Tenant AWS account Amazon EKS Namespace On premises Kubernetes Namespace On premises Kubernetes Namespace Full ArgoCD UI/CLI API Server Redis Server Repo Controllers
- 33. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Centralized/Hub-Spoke (Push) Central Amazon EKS cluster Central AWS account Namespace Tenant AWS account On premises Amazon EKS Tenant AWS account Kubernetes On premises Kubernetes Amazon EKS Full ArgoCD UI/CLI API Server Redis Server Repo Controllers
- 34. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Centralized/Hub-Spoke (Shared) Central Amazon EKS cluster Central AWS account Namespace Tenant AWS account On premises Amazon EKS Tenant AWS account Kubernetes On premises Kubernetes Amazon EKS App-2 repo App-1 repo App-4 repo Platform Config App-3 repo Full ArgoCD UI/CLI API Server Redis Server Repo Controllers
- 35. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Centralized/Hub-Spoke (Agent) Central Amazon EKS cluster Central AWS account Namespace Tenant AWS account On premises Amazon EKS Tenant AWS account Kubernetes On premises Kubernetes Amazon EKS App-2 repo App-1 repo App-4 repo Platform Config App-3 repo Core ArgoCD UI/CLI API Server Redis Server Repo Controllers https://argo-cd.readthedocs.io/en/stable/operator-manual/core https://akuity.io/blog/reducing-argocd-operational-burden https://github.com/open-cluster-management-io/argocd-pull-integration Full ArgoCD UI/CLI API Server Redis Server Repo Controllers
- 36. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Reliability and Performance
- 37. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. ArgoCD Scaling Challenges 37 Image: https://colocatedeventseu2023.sched.com/event/1JoAP/scaling-argo-security-and-multi-tenancy-in-aws-eks-at-the-new-york-times-david-grizzanti-luke-philips-the-new-york-times
- 38. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Scaling ArgoCD 38 • Enable HPA (API, Repo, Redis) • Controller Replica shards (random, round-robin) • Tunning (timeouts, processors queues) • Reconcile Optimization ignoreResourceUpdates vs. ignoreDifferences https://argo-cd.readthedocs.io/en/stable/operator-manual/high_availability https://argo-cd.readthedocs.io/en/stable/operator-manual/reconcile
- 39. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Monitoring ArgoCD 39 Prometheus Use Operator or Setup service labels OpenTelemetry/ADOT Alerts AMP (Amazon Managed Service for Prometheus) Grafana ArgoCD Dashboard (tweak) Sync time Work queue AMG (Amazon Managed Grafana) Logging Find k8s resources properties to ignore
- 40. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. GitOps Bridge
- 41. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. GitOps Bridge: IaC and Addons https://github.com/gitops-bridge-dev ack terraform kops ansible pulumi cdk capa crossplane IaC Virtual private cloud (VPC) Subnet Kubernetes Amazon EKS IAM Role NAT gateway Instances Infrastructure Kubernetes CR (ArgoCD Cluster) metadata: annotations: aws_alb_role_arn: arn…. labels: enable_aws_alb: true 1 2 3 ./aws/aws-cloudwatch-metrics-appset.yaml ./aws/aws-csi-ebs-resources-appset.yaml ./aws/aws-csi-efs-driver-appset.yaml ./aws/aws-csi-fsx-driver-appset.yaml ./aws/aws-fluentbit-appset.yaml ./aws/aws-gateway-api-controller-appset.yaml ./aws/aws-load-balancer-controller-appset.yaml ./aws/aws-node-termination-handler-appset.yaml ./aws/aws-oss-cert-manager-appset.yaml ./aws/aws-oss-cluster-autoscaler-appset.yaml ./aws/aws-oss-crossplane-providers-appset.yaml ./aws/aws-oss-external-dns-appset.yaml ./aws/aws-oss-external-secrets-appset.yaml ./aws/aws-oss-karpenter-appset.yaml ./aws/aws-oss-privateca-issuer-appset.yaml ./aws/aws-oss-velero-appset.yaml ./aws/aws-secrets-store-csi-appset.yaml ./oss/argo-cd-appset.yaml ./oss/argo-events-appset.yaml ./oss/argo-rollouts-appset.yaml ./oss/argo-workflows-appset.yaml GitOps
- 42. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. GitOps Bridge: ApplicationSet (Addon versions) version in dev is 1.6.0 version in staging is 1.5.5 version in production is 1.5.4 Cluster opt-in for the addon Chart name and repo in a single place Merge generator Prevent Outages
- 43. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. GitOps Bridge: ApplicationSet (overrides) Metadata based on IaC Namespace based on IaC Override values files Value files in git
- 44. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. GitOps Bridge Configuration Platform team Platform Repo Region Kubernetes Account Control Plane Addons App Of AppSet Addon-1 charts/ environments/ Addon-2 App Of ApplicationSets Addon-1 ApplicationSet Addon-3 ApplicationSet Addon-2 ApplicationSet clusters/ Kubernetes CR (ArgoCD Cluste) metadata: annotations: aws_alb_role_arn: arn…. labels: enable_aws_alb: true Addon-1 Application Addon-3 Application Addon-2 Application
- 45. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Thank you! © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Carlos Santana carrlos@amazon.com Rodrigo Bersa bersr@amazon.com @csantanapr csantanapr bersa