Managed Container Orchestration with Amazon ECS
- 1. Managed Container Orchestration with ECS Philipp Garbe (@pgarbe) Scout24
- 3. β Philipp Garbe β Lead Platform Engineer @Scout24 β Docker Captain β Living in Bavaria β Working in the Cloud About Me
- 4. Our purpose: Inspiring your best decisions. We connect people, cars and homes.
- 6. Improve the developer experience With kind permissions of Alexey Gravanov
- 8. How to run it in production?
- 9. So many choices... Amazon ECS Docker Swarm Azure Container Services Cloud Foundryβs Diego https://www.linux.com/news/8-open-source-CONTAINER-ORCHESTRATION-TOOLS-KNOW CoreOS Fleet Google Container Engine Kubernetes Mesosphere Marathon
- 10. Amazon ECS Docker Swarm Azure Container Services Cloud Foundryβs Diego https://www.linux.com/news/8-open-source-CONTAINER-ORCHESTRATION-TOOLS-KNOW CoreOS Fleet Google Container Engine Kubernetes Mesosphere Marathon
- 11. Deep Integration with AWS β Cluster and Services can be created with CloudFormation β Tasks have their own IAM role β Dynamic port mapping with ALB Easy to maintain β Cluster state is managed by AWS Follows our IT principals β All orchestration tools can run containers! Why ECS?
- 12. Under the hood
- 13. What? Itβs not managed? ECS has no AutoScaling? No automated Node draining? BUT: We donβt want to run / care about the cluster
- 14. Resources: Service: Type: 'Custom::InfinityService' Properties: ServiceToken: 'arn:aws:lambda:eu-west-1:123456789012:function:angus' ServiceName: my-service ImageName: 210987654321.dkr.ecr.eu-west-1.amazonaws.com/awesome-app ContainerPort: 9000 Make it simple
- 15. Infinity & Angus
- 16. Infinity
- 17. Infinity = Managed ECS Cluster ECS + β Automated node draining β Automated Scaling β Cross-Account Proxy β Watchdog Lambda β Some tweaks
- 18. Automated Node Draining The Problem ECS offers Node Draining, but ASG is not interested in it
- 19. Automated Node Draining ASG Lifecycle Hook + SNS + Lambda
- 20. Automated Scaling The Problem No proper CloudWatch metric to scale out and in
- 21. β CPU Utilization β Memory Utilization β CPU Reservation β Memory Reservation ECS Metrics
- 22. New metric: # of schedulable containers
- 23. Example Have enough resources for 1 - 2 of our max containers Scale out when # of schedulable containers < 1 Scale in when # of schedulable containers > 2
- 24. How many schedulable containers?
- 25. How many schedulable containers?
- 26. How many schedulable containers?
- 27. How many schedulable containers?
- 28. Cross-Account Proxy The Problem TaskRoleArn supports only roles of the same account
- 29. Default Roles ecsAutoScalingRole ecsContainerInstanceRole ecsServiceRole ecsTaskRole β Read CloudWatch Metrics β Modify App AutoScaling β ECR: Get Images β ECS: De/Register Container Instances β De/Register Instances with Load Balancer β Everything your task needs to do
- 32. β Re-route call to ECS-Agent β ECS-Agent gets credentials based on configured TaskRole β TaskRole needs only one permission: AssumeRole β X-Acc-Proxy assumes role (Role ARN comes from Docker Label) β X-Acc-Proxy returns credentials from assumed role Cross Account Proxy
- 34. WatchDog Lambda The Problem Timeout of creating an ECS service which donβt get healthy is 3h!
- 35. WatchDog Lambda Lambda who checks for status codes like β βEssential container in task exitedβ β βTask failed ELB health checksβ β Cancels the CloudFormation stack update
- 36. β HealthCheck in cfn-init β βVersionβ in UserData β Partitions for images and logs β Container cleanup ECS_ENGINE_TASK_CLEANUP_WAIT_DURATION Tweaks
- 37. Angus
- 38. >500 lines of CloudFormation template β Load balancer β Security Groups β Auto scaling β Roles β Alarms ECS Service and Taskdefinition
- 40. Resources: Service: Type: 'Custom::InfinityService' Properties: ServiceToken: 'arn:aws:lambda:eu-west-1:123456789012:function:angus' ServiceName: my-service ImageName: 210987654321.dkr.ecr.eu-west-1.amazonaws.com/awesome-app ContainerPort: 9000 Make it simple
- 41. β Like AWS Service Catalog but for managed services β Translator to map properties of a custom resource to CloudFormation stack parameters β Handles the whole CloudFormation workflow β Supports versioned templates β Allow to roll out template changes to existing stacks What is Angus?
- 42. Angus
- 43. Team: stack.yaml CloudFormation specific Angus specific Template specific Service: Type: 'Custom::InfinityService' Properties: ServiceToken: > 'arn:aws:lambda:eu-west-1:123456789012:function:angus' ServiceName: my-service ImageName: nginx Cpu: 512 Configuration: - MY_ENV_VAR: my-value
- 44. Team: stack.yaml Service: Type: 'Custom::InfinityService' Properties: ServiceToken: > 'arn:aws:lambda:eu-west-1:123456789012:function:angus' ServiceName: my-service ImageName: nginx Cpu: 512 Configuration: - MY_ENV_VAR: my-value Defines service type (which template should be used) Defines stack name in platform account Arn of Angus lambda
- 45. Team: stack.yaml Service: Type: 'Custom::InfinityService' Properties: ServiceToken: > 'arn:aws:lambda:eu-west-1:123456789012:function:angus' ServiceName: my-service ImageName: nginx Cpu: 512 Configuration: - MY_ENV_VAR: my-value Platform: my-service AWSTemplateFormatVersion: '2010-09-09' Parameters: ImageName: Type: String Description: Name of the Docker Image Cpu: Type: Number Default: 512 MinValue: 0 MaxValue: 4096 Description: CPU reservation for the task Configuration1: Type: String Description: An item of possible environment variables Default: ''
- 46. Template Versioning β Allows you to introduce new features (like HealthCheckPath) β Allows you to apply new features from AWS Rollout strategies β Canary releases for existing stacks β For new stacks But: Be backwards compatible!
- 47. Questions ?
- 48. Automated Node Draining: https://aws.amazon.com/blogs/compute/how-to-automate-container-instance-draining-in-amazon-ecs/ IAM Reference: https://iam.cloudonaut.io CloudFormation Hacks: http://garbe.io/blog/2017/07/17/cloudformation-hacks/ AutoScaling of ECS clusters: β http://garbe.io/blog/2017/04/12/a-better-solution-to-ecs-autoscaling/ β https://help.spotinst.com/hc/en-us/articles/115004452129-EC2-Container-Service-ECS-Autoscaler High quality CloudFormation templates http://templates.cloudonaut.io Useful resources
- 49. www.scout24.com Philipp Garbe Twitter: @pgarbe Blog: garbe.io