ACDKOCHI19 - Turbocharge Developer productivity with platform build on K8S and AWS services.
1 like63 views
Intuit has developed a modern SaaS platform leveraging Kubernetes clusters on AWS to enhance service onboarding and release cycles, achieving significant improvements in deployment times. The platform includes multiple tools such as an addon manager, instance manager, and deep monitoring solutions to manage multi-tenancy and optimize resource usage efficiently. Ongoing developments aim to enhance stability, monitoring, and utilize emerging technologies like Istio and OpenTelemetry.
ACDKOCHI19 - Turbocharge Developer productivity with platform build on K8S and AWS services.
- 1. Intuit Modern SaaS Platform Running Kubernetes Clusters at scale on AWS Laks
- 2. ● Why we built it? ● What is the Modern SaaS platform? ● How we built it? Agenda
- 3. Intuit Confidential and Proprietary 3 Measured value Expected Improvement Recoverability X 10X faster Release cycle time (days) (PR to deploy) Y 1.4Y faster Release frequency (days) Z 3Z faster Goals
- 4. Intuit Confidential and Proprietary 4 Monolith vs Microservice
- 5. Intuit Confidential and Proprietary 5 Service Onboarding Prior to M-SaaS I want to develop new micro service ● Create Service on Service Portal ● Get an Cloud Account - ● Create all the basic resources in AWS ● If you have Web server do this... ● Monitoring Alerting Logs….. ● : ● Take care of AWS SG, EC2, Auto scaling, R53, Ingress, Egress ● Create ○ Git Repo ○ Pipeline ○ Set up CD … ○ ……. Takes few weeks to set up and run the service Steep learning curve Multiple AWS Accounts Take care of security patches periodically
- 6. Intuit Confidential and Proprietary 6 Monolith vs Microservice
- 7. Intuit Confidential and Proprietary 7 Service Onboarding Today…. with M-SaaS VELOCITY OPERABILITY SECURITY I want to develop new micro service AppD Microservice 15 minutes
- 8. Gitops Flow
- 9. Intuit Confidential and Proprietary 9 The Modern SaaS Platform @ Intuit
- 10. Intuit Confidential and Proprietary 10 ● Design and development started in Jan ‘18 ● First application was running Kafka on Kubernetes ● Running clusters in dev/test, pre-prod and prod environments since Apr ‘18. ● Over 150 Kubernetes clusters and 3000 namespaces today… Journey so far ...
- 11. Intuit Confidential and Proprietary 11 Journey so far … Services on MSaaS
- 12. Intuit Confidential and Proprietary 12 Journey so far … Namespaces on MSaaS
- 13. Intuit Confidential and Proprietary 13 Journey so far … Clusters on MSaaS
- 14. Intuit Confidential and Proprietary 14 ● Intuit Kubernetes Service ○ Using Kops today ○ Moving to EKS ● Intuit Kubernetes Service Manager (may open source) ● Custom Resources for cluster lifecycle management (aka. Keiko) Modern SaaS platform today ...
- 15. Intuit Confidential and Proprietary 15 alb-ingress kube-dns fluentd metrics prometheus autoscaler Addons User namespace 1 User namespace 2 User namespace 3 User namespace n Applications kube-apiserver kube-proxy K8s Control Plane kube-scheduler kube-controlleretcd Each Kubernetes cluster today ...
- 16. Intuit Confidential and Proprietary 16 Master Nodes alb-ingress kiam eventrouter metrics kube-dns autoscaler Addons kube-apiserver kube-proxy K8s Control Plane kube-scheduler kube-controlleretcd Each Kubernetes cluster today ...
- 17. Intuit Confidential and Proprietary 17 The Challenges
- 18. Intuit Confidential and Proprietary 18 Addons - Common functionality needed by all apps on a cluster - DNS, log forwarding, metrics, identity, etc. - Integrate with other AWS services such as ALB.
- 19. Intuit Confidential and Proprietary 19 Multi-tenancy - What does each tenant mean? - Namespace? - Kubernetes objects with the same label? - Some CRD? We decided to go with Kubernetes Namespaces
- 20. Intuit Confidential and Proprietary 20 More Multi-tenancy issues - Noisy neighbour - Customized setup - Tenant specific AMIs - Tenant specific instance types - Cost accounting
- 21. Intuit Confidential and Proprietary 21 Resilience and hardening ... - Pods stuck in terminating state ... - EC2 instance networking broken … - Bunch of 502s during upgrade...
- 22. Intuit Confidential and Proprietary 22 Deep monitoring - Not enough to simply check if components are “up” - Deep monitoring - Actually exercise the functionality - Periodically - Preferably automatic remediation
- 23. Intuit Confidential and Proprietary 23 Cost efficiency - How do we reduce costs?
- 24. Intuit Confidential and Proprietary 24 The Solutions
- 25. Intuit Confidential and Proprietary 25 Addon-Manager Addons are critical components within a Kubernetes cluster that provide basic services needed by applications like DNS, Ingress, Metrics, Logging, etc. Addon Manager provides a CRD for lifecycle management of such addons using Argo Workflows.
- 26. Intuit Confidential and Proprietary 26 Addon-Manager
- 27. Intuit Confidential and Proprietary 27 Multi-tenancy solutions - Instance Group per Namespace - Customized labels - Centralized upgrades We decided to go with ...
- 28. Intuit Confidential and Proprietary 28 Instance-manager - Declaratively provision and manage ASGs (nodes) - Number and type of nodes - Labels and taints - Subnets and security groups $ kubectl create -f /tmp/hello_world.yaml instancegroup.instancemgr.keikoproj.io/hello-world created $ kubectl get igs NAME STATE MIN MAX GROUP NAME PROVISIONER STRATEGY AGE hello-world Ready 2 3 shri-east-2-instance-manager-hello-world-NodeGroup-16Y8ZA1ZJW8JK eks-cf crd 3m nodes Ready 2 3 shri-east-2-instance-manager-nodes-NodeGroup-1K1T3YSXCCCK9 eks-cf crd 1d
- 29. Intuit Confidential and Proprietary 29 Upgrade-manager - Upgrade Manager provides RollingUpgrade, a Kubernetes native mechanism for doing rolling- updates of instances in an AutoScaling group using a CRD and a controller.
- 30. Intuit Confidential and Proprietary 30 Governor Governor improves the stability of large Kubernetes clusters by proactively terminating failed but stuck pods and misbehaving nodes.
- 31. Intuit Confidential and Proprietary 31 Active-monitor Active-Monitor is a Kubernetes custom resource controller which uses Argo Workflows for deep cluster monitoring.
- 32. Intuit Confidential and Proprietary 32 Minion-manager Minion-manager enables the intelligent use of Spot Instances in Kubernetes clusters on AWS. This is done by factoring in on-demand prices, spot-instance prices and current state of the AutoScalingGroups.
- 33. Intuit Confidential and Proprietary 33 Kube-forensics Kube-forensics allows a cluster administrator to dump the current state of a running pod and all its containers so that security professionals can perform offline forensic analysis.
- 34. Intuit Confidential and Proprietary 34 Keiko “Keiko provides a set of independent open-source tools for orchestration and management of multi-tenant, reliable, secure and efficient Kubernetes clusters at scale.” github.com/keikoproj Instance manager Kube forensics Upgrade manager Active monitor Addon manager Governor Minion manager
- 35. Intuit Confidential and Proprietary 35 Keiko github.com/keikoproj Instance manager Kube forensics Upgrade manager Active monitor Addon manager Governor Minion manager github.com/keikoproj twitter.com/keikoproj
- 36. Intuit Confidential and Proprietary 36 Keiko Orchestration Instance-manager Upgrade-manager Reliability Governor Cost Eff Minion Manager Addon Manager Security Kube-Forensics Monitoring Active-monitor
- 37. Intuit Confidential and Proprietary 37 Coming up ... - Kubernetes control plane using EKS - Multi-cluster Service Mesh using Istio - OpenTelemetry - GitOps for AWS resources - Experimentation platform - And more ...
- 38. Intuit Confidential and Proprietary 38 There’s a lot happening ... <We are hiring />