ECS and ECR deep dive

© 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Apr 21 2016
Amazon EC2 Container
Service Deep Dive
Shiva N, Solution Architect, AWS

Agenda
The Basics
Infrastructure Setup
Infrastructure Management
Deploying Applications
PaaS on ECS
Using the CLI

TaskDefinitions
Containers
Clusters
Container Instances
Key Components

Amazon ECS Infrastructure
Setup

Amazon ECS Infrastructure Setup
Amazon ECS Cluster Setup
Amazon ECR Setup

Amazon ECS Cluster Setup
There are many ways to provision cluster infrastructure
v AWS – CloudFormation, Simple Systems Manager,
Autoscale Groups, OpsWorks, ECS-CLI
v Others - Terraform, PaaS, Partners
Let’s talk about CloudFormation

Cluster Setup with AWS CloudFormation
CloudFormation supports ECS cluster, service and task
definition resources
Use AWS::IAM::Role to create ECS service role and
container instances role
Launch container instances using
AWS:AutoScaling::LaunchConfiguation and
AWS:AutoScaling::AutoScalingGroup

"Resources" : {
"ECSCluster": {
"Type": "AWS::ECS::Cluster"
},
"ECSAutoScalingGroup" : {
"Type" : "AWS::AutoScaling::AutoScalingGroup",
"Properties" : {
"VPCZoneIdentifier" : { "Ref" : "SubnetID" },
"LaunchConfigurationName" : { "Ref" : "ContainerInstances" },
"MinSize" : "1",
"MaxSize" : { "Ref" : "MaxSize" },
"DesiredCapacity" : { "Ref" : "DesiredCapacity" }
},
[…]
},

"ContainerInstances": {
"Type": "AWS::AutoScaling::LaunchConfiguration",
"Metadata" : {
"AWS::CloudFormation::Init" : {
"config" : {
"commands" : {
"01_add_instance_to_cluster" : {
"command" : { "Fn::Join": [ "", [ "#!/bin/bashn", "echo
ECS_CLUSTER=", { "Ref": "ECSCluster" }, " >> /etc/ecs/ecs.config" ] ] }
}
},
[…]
}
}
}

Amazon ECR Setup
You have read and write access to the repositories you
create in your default registry, i.e.
<aws_account_id>.dkr.ecr.us-east-1.amazonaws.com
Repository names can support namespaces, e.g. team-
a/web-app.
Repositories can be controlled with both IAM user access
policies and repository policies.

Amazon ECR Setup
# Authenticate Docker to your Amazon ECR registry
> aws ecr get-login
docker login -u AWS -p <password> -e none https://<aws_account_id>.dkr.ecr.us-east-
1.amazonaws.com
> docker login -u AWS -p <password> -e none https://<aws_account_id>.dkr.ecr.us-east-
1.amazonaws.com
# Create a repository called ecr-demo
> aws ecr create-repository --repository-name ecr-demo
# Push an image to your repository
> docker push <aws_account_id>.dkr.ecr.us-east-1.amazonaws.com/ecr-demo:v1

Amazon ECS Infrastructure
Management

Amazon ECS Infrastructure Management
Monitoring & Logging
Scaling ECS
Service Discovery & Configuration Management
Security

Monitoring and Logging on Amazon ECS
Monitoring with Amazon CloudWatch
Configuring logging in Task Definition
Amazon CloudTrail
Monitoring Amazon ECS with Datadog
Monitoring Amazon ECS with Sysdig Cloud

Metric data sent to CloudWatch in 1-minute periods and
recorded for a period of two weeks
Available metrics: CPUReservation, MemoryReservation,
CPUUtilization, MemoryUtilization
Available dimensions: ClusterName, ServiceName

Use the Amazon CloudWatch Monitoring Scripts to monitor
additional metrics, e.g. disk space:
# Edit crontab
> crontab -e
# Add command to report disk space utilization to CloudWatch every five minutes
*/5 * * * * <path_to>/mon-put-instance-data.pl --disk-space-util --disk-space-used --disk-
space-avail --disk-path=/ --from-cron

Configuring Logging in Task Definition
logConfiguration task definition parameter
Requires version 1.18 or greater of the Docker Remote API
Maps to docker run --log-driver option
Log drivers: json-file, syslog, journald, gelf, fluentd

Logging with Amazon CloudWatch Logs
• Logging container with syslogd
and CloudWatch Logs Agent
• Attach /var/log Volume to
Logging container (Sidecar
pattern)
• Link other containers
syslogd
CloudWatch Logs
Agent
CloudWatch
Logs
Container instance
ECS Cluster
ECS Agent
Logs
Docker
Logs

Logging Amazon ECS API with AWS CloudTrail
{
"eventVersion": "1.03",
"userIdentity": {…},
"eventTime": "2015-10-12T13:57:33Z",
"eventSource": "ecs.amazonaws.com",
"eventName": "CreateCluster",
"awsRegion": "eu-west-1",
"sourceIPAddress": "54.240.197.227",
"userAgent": "console.amazonaws.com",
"requestParameters": {
"clusterName": "ecs-cli"
},

Logging Amazon ECS API with AWS CloudTrail
"responseElements": {
"cluster": {
"clusterArn": "arn:aws:ecs:eu-west-
1:560846014933:cluster/ecs-cli",
"pendingTasksCount": 0,
"registeredContainerInstancesCount": 0,
"status": "ACTIVE",
"runningTasksCount": 0,
"clusterName": "ecs-cli",
"activeServicesCount": 0
}
},
[…]

Monitoring Amazon ECS with Datadog

Monitoring Amazon ECS with Sysdig Cloud

Scaling Amazon ECS
AutoScaling your Amazon ECS cluster
Scaling your Services with Lambda

Setup ECS Cluster with AutoScaling
Create LaunchConfiguration
• Pick instance type
depending on resource
requirements, e.g. memory
or CPU
• Use latest Amazon Linux
ECS-optimized AMI, other
distros available
Create AutoScaling group and
set to cluster initial size

AutoScaling your Amazon ECS Cluster
• Create CloudWatch
alarm on a metric, e.g.
MemoryReservation
• Configure scaling
policies to increase and
decrease the size of your
cluster

Scaling your Services with Lambda
• Cloudwatch metrics tied
to SNS
• SNS triggers Lambda
Container Scaling
function
• Lambda scales task
count on cluster
• Bonus - Extensible
‘cluster intelligence’ layer

Service Discovery &
Configuration Management

Service Discovery on Amazon ECS
Service Discovery with ECS Services & Route 53
Service Discovery with Weaveworks
Service Discovery and Configuration Management with
Consul
Service Discovery and Configuration Management with
etcd

• Route 53 private hosted zone
• Set search path on hosts with DHCP option sets
• Define ECS services with ELB
• Create CNAMEs for each ELB

Task
Task TaskTask
ECS
Service
Application
router, e.g.
nginx
Internal ELB with
CNAME, e.g.
api.example.com
Route 53
private zone,
e.g.
example.com

Service Discovery with Weaveworks
DNS interface for cross-host
container communication
Gossip protocol to share
grouped updates
Overlay network between hosts

Service Discovery and Configuration
Management with Consul
Three main components:
• Consul agent - Runs on each node, responsible for
checking the health of the services and of the node
itself.
• One or more Consul servers - Store and replicate
data, leader elected using the Raft consensus
algorithm
• Registrator agent - Automatically
register/deregisters services based on published
ports and metadata from the container environment
variables defined in the ECS task definition

Management with Consul
ECSCluster
consul-server
ECS Instance
consul-agent
registrator
ECS Instance
Back end 1
Back end 2
consul-agent
registrator
ECS Instance
Front end
ECSCluster

Management with etcd
etcd
registrator
ECS Instance
Container 1
Container 2
confd etcd
registrator
ECS Instance
Container 1
Container 2
confd etcd
registrator
ECS Instance
Container 1
Container 2
confd

Security
ECS IAM Policies and Roles
ECR IAM Policies and Roles
Image Vulnerability Scanning with Twistlock

ECS IAM Policies and Roles
The ECS agent calls the ECS APIs on your behalf, so
container instances require an IAM policy and role that
allows these calls.
The ECS service scheduler calls the EC2 and ELB APIs on
your behalf to register and deregister container instances
with your load balancers.
Use AmazonEC2ContainerServiceforEC2Role and
AmazonEC2ContainerServiceRole managed policies
(respectively)

ECR IAM Policies and Roles
ECR uses resource-based permissions to control access.
By default, only the repository owner has access to a
repository.
You can apply a policy document that allows others to
access your repository.
Use managed policies for IAM users or roles that allow
differing levels of control:
AmazonEC2ContainerRegistryFullAccess,
AmazonEC2ContainerRegistryPowerUser or
AmazonEC2ContainerRegistryReadOnly

Image Vulnerability Scanning with Twistlock

Deploying Applications
Scheduling Containers
Automating Deployments

Scheduling Containers on ECS
Batch Jobs
ECS Task scheduler
Run tasks once
Batch jobs
RunTask (random)
StartTask (placed)
Long-Running Apps
ECS Service scheduler
Health management
Scale-up and scale-down
AZ aware
Grouped Containers

Scheduling Containers: Long-running App
Optionally run your service behind a load balancer.
One load balancer per service.
ELB currently supports a fixed relationship between the
load balancer port and the container instance port.
If a task fails the ELB health check, the task is killed and
restarted (until service reaches desired capacity).

Update service’s task definition (rolling update)
Specify a deployment configuration for your service:
• minimumHealthyPercent: lower limit (as a percentage of
the service's desiredCount) of the number of running
tasks that must remain running in a service during a
deployment.
• maximumPercent: upper limit (as a percentage of the
service's desiredCount) of the number of running tasks
that can be running in a service during a deployment.

Scheduling Containers: Long-running app
Deploy using the least space: minimumHealthyPercent =
50%, maximumPercent = 100%

Deploy quickly without reducing service capacity:
minimumHealthyPercent = 100%, maximumPercent =
200%

Blue-Green Deployments
• Define two ECS services
• Each service is associated w/ ELB
• Both ELBs in Route 53 record set
with weighted routing policy, 100%
Primary, 0% Secondary
• Deploy to Blue or Green service and
switch weights
TaskTask
Route 53
record set
with
weighted
routing
policy
0%
100%

Automating Deployments
Continuous Delivery to ECS with Jenkins
Continuous Delivery to ECS with Shippable

4. Push image to
Docker registry
2. Build image
from sources
3. Run test on image
1. Code push
triggers build
5. Update Service
6. Pull image

Easy Deployment
Developers – Merge into master, done!
Jenkins Build Steps
Trigger via Webhooks, Monitoring, Lambda
Build Docker image via Build and Publish plugin
Push Docker image into Registry
Register Updated Job with ECS API

Continuous Delivery to ECS with Shippable

PaaS on ECS
AWS Elastic Beanstalk
Convox
Remind Empire

Uses Amazon ECS to coordinate deployments to
multicontainer Docker environments
Takes care of tasks including cluster creation, task
definition and execution

Elastic Beanstalk uses a Dockerrun.aws.json file that
describes how to deploy containers.
The Dockerrun.aws.json file includes three sections:
• AWSEBDockerrunVersion: Set to "2" for multicontainer
Docker environments.
• containerDefinitions: An array of container definitions.
• volumes: Creates mount points in the container instance
that a container can use.

Convox
# Initialize your app and create default manifest
> convox init
# Locally build and run your app as declared in the manifest
> convox start
# Create app
> convox apps create my_app
# Deploy app, output ELB DNS name
> convox deploy
[...]
web: http://my_app-1234567890.us-east-1.elb.amazonaws.com

Remind Empire
Control layer on top of Amazon ECS that provides a
Heroku like workflow
Any tagged Docker image can be deployed to Empire as
an app
• When you deploy a Docker image to Empire, it will
extract a Procfile from the WORKDIR
• Each process type in the Procfile maps directly to an
ECS Service

Remind Empire
Routing layer backed by internal ELBs
• An application that specifies a web process will get an
internal ELB attached to its ECS Service
• When a new internal ELB is created, an associated
CNAME record is created in Route53 under the internal
TLD, enabling service discovery via DNS

Using the CLI
Configuring the ECS CLI
Cluster Setup with the ECS CLI
Deploy Compose App with ECS CLI
Scaling with ECS CLI

Easily create Amazon ECS clusters & supporting
resources such as EC2 instances
Run Docker Compose configuration files on Amazon
ECS
Available today – http://amzn.to/1jBf45a

# Configure the CLI using environment variables
> export AWS_ACCESS_KEY_ID=<my_access_key>
> export AWS_SECRET_ACCESS_KEY=<my_secret_key>
> ecs-cli configure --region us-east-1 --access-key
$AWS_ACCESS_KEY_ID --secret-key $AWS_SECRET_ACCESS_KEY --cluster
ecs-cli-demo
# Configure the CLI using an existing AWS CLI profile
> ecs-cli configure --region us-west-2 --profile ecs-profile --
cluster ecs-cli-demo

Cluster Setup with the ECS CLI
# Creates a new ECS cluster with two container instances in an
existing VPC
> ecs-cli up --capability-iam --keypair my_ecs_keypair --size 2 -
-security-group sg-a12bc34d --vpc vpc-0e9dc8b7 --subnets subnet-
12ab34cd,subnet-56ef78ab --instance-type t2.medium
# Creates a new ECS cluster with one container instance in a new
VPC
> ecs-cli up --capability-iam --keypair my_ecs_keypair --azs us-
east-1a,us-east-1c --cidr 192.169.0.0/24 --port 22 --instance-
type t2.medium

Docker Compose lets you define and run multi-container
applications:
1. Define app environment with Dockerfile
2. Define services that make up your app in docker-
compose.yml
3. Run docker-compose up to start and run entire app

proxy:
build: ./proxy
ports:
- "80:80"
links:
- web
web:
build: ./web
command: bundle exec rails server -b 0.0.0.0
environment:
- SECRET_KEY_BASE=secretkey
expose:
- "3000"

> ecs-cli compose up
> ecs-cli compose ps
> ecs-cli compose service create
> ecs-cli compose service start

Scaling with ECS CLI
> ecs-cli scale n
> ecs-cli compose scale n
> ecs-cli compose service scale n

ECS and ECR deep dive

More Related Content

Viewers also liked (11)

Similar to ECS and ECR deep dive (17)

More from Shiva Narayanaswamy (11)

Recently uploaded (20)

ECS and ECR deep dive