Amazon EC2 Container Service Deep dive
19 likes12,762 views
２０１５年１２月７日に開催されたIVS CTO Night & Day 2015 WinterのSession B-2 : EC2 Container Service Deep diveの資料です。イベントの様子や他の資料は以下ブログをご覧ください。 http://aws.typepad.com/sajp/2015/12/ivs-cto-night-day-2015-winter-powered-by-aws.html
![17
Task Definition: Container Definition
{
"name": "simple-demo",
"image": "foo/my-demo",
"cpu": 10,
"memory": 500,
"portMappings": [
{
"containerPort": 80,
"hostPort": 80
}
],
"mountPoints": [
{
"sourceVolume": "my-vol",
"containerPath": "/var/www/my-vol"
}
],
"entryPoint": [
"/usr/sbin/apache2",
"-D",
"FOREGROUND"
],
"essential": true
},
{
"name": "busybox",
"image": "busybox",
"cpu": 10,
"memory": 500,
"volumesFrom": [
{
"sourceContainer": "simple-demo"
}
],
"entryPoint": [
"sh",
"-c"
],
"command": [
"while true; do /bin/date > /var/www/my-vol/date; sleep 1; done"
],
"essential": false
}](https://image.slidesharecdn.com/20151207ctonightecs-151215025634/85/Amazon-EC2-Container-Service-Deep-dive-17-320.jpg)
![Service Discovery
$ aws ecs describe-services --service api--web
"loadBalancers": [{
"containerName": "web”,
"containerPort": 9001,
"loadBalancerName”: "2888...a31d4c”
}]
$ curl http://api
Ok](https://image.slidesharecdn.com/20151207ctonightecs-151215025634/85/Amazon-EC2-Container-Service-Deep-dive-64-320.jpg)
Amazon EC2 Container Service Deep dive
- 1. 1 Amazon EC2 Container Service Deep dive Ryosuke Iwanaga Solutions Architect, Amazon Web Services Japan
- 2. 2 Agenda • ContainerとDevOps • Case studies – Sony: Batch jobs • Summary
- 4. 4 DevOps lifecycle Build Test ProductionDevelopment <> <> Application Code Artifact
- 5. 5 DevOps lifecycle Build Test ProductionDevelopment <> <> + AMI Provisioning Code Application Code Artifact Provisioning Code {} Config {} Config {} Config
- 6. 6 After Docker… + <>+ Build Test ProductionDevelopment <> <> + {} {} {}
- 7. 7 After Docker… + Provisioning Code <> Application Code Docker Image + Dockerfile Docker Image Build Test ProductionDevelopment
- 8. 8 Dev and Ops • Dev – インフラに関するコードを書かずに、アプリをCI/CDする – 価値あるコードを生み出し続ける • Ops – どんなアプリが動くかに依存しないインフラ管理 – スケーラビリティ、コスト最適化
- 9. 9 Infrastructure for Docker containers Build Test ProductionDevelopment Registry Service A Service B Service C
- 10. 10 Resource Utility of Heterogeneous Containers 35% 85% ~80% Amazon ECS
- 11. 11 Resource Utility of Heterogeneous Instances Amazon ECS Amazon EC2 Spot Fleet + c3.xlarge c3.xlarge c3.xlarge r3.8xlarge r3.8xlarge r3.8xlarge c3.8xlarge c3.8xlarge c3.8xlarge c3.4xlarge c3.4xlarge c3.4xlarge r3.2xlarge r3.2xlarge r3.2xlarge
- 12. 12 Cluster管理とScheduler • Cluster管理 – 計算機群のリソース、 状態を常に管理 • Scheduler – Cluster全体を見て適切 にContainerを配置 CPU: 500 Mem: 300 CPU: 10 Mem: 30 CPU: 2000 Mem: 1000 CPU: 10 Mem: 30CPU: 10 Mem: 30 Scheduler Cluster Manager
- 14. 14 Docker Task Container Instance Amazon ECS Container ECS Agent ELB Internet ELB User / Scheduler API Cluster Management Engine Task Container Docker Task Container Instance Container ECS Agent Task Container Docker Task Container Instance Container ECS Agent Task Container AZ 1 AZ 2 Key/Value Store Agent Communication Service http://aws.typepad.com/sajp/2015/07/under-the-hood-of-the-amazon-ec2-container-service.html
- 15. 15 Amazon ECS: Task Definition
- 16. 16 Amazon ECS: Task Definition • Containerの集合を定義 – 必ず同じInstanceで稼働 – 要求するリソースを指定 • CPU, memory, (Port) • ボリュームも定義可能 – Instanceのファイルシス テムを利用できる • バージョニングが可能
- 17. 17 Task Definition: Container Definition { "name": "simple-demo", "image": "foo/my-demo", "cpu": 10, "memory": 500, "portMappings": [ { "containerPort": 80, "hostPort": 80 } ], "mountPoints": [ { "sourceVolume": "my-vol", "containerPath": "/var/www/my-vol" } ], "entryPoint": [ "/usr/sbin/apache2", "-D", "FOREGROUND" ], "essential": true }, { "name": "busybox", "image": "busybox", "cpu": 10, "memory": 500, "volumesFrom": [ { "sourceContainer": "simple-demo" } ], "entryPoint": [ "sh", "-c" ], "command": [ "while true; do /bin/date > /var/www/my-vol/date; sleep 1; done" ], "essential": false }
- 18. 18 Task Definition: Overview Shared data volume PHP App Time of day App Task Definition
- 19. 19 Task Definition: Overview Container Instance Schedule Shared data volume PHP App Time of day App Task Definition Task
- 21. 21 Amazon ECS: Service • Web/APIの様に長期稼働する ワークロードに最適 • Taskを必要数保ってくれるスケ ジューラ – 自動復旧にも対応 • 新しいTask Definitionをデプロ イしつつ切替 • Elastic Load Balancingとの連 携も可能
- 23. 23 Amazon ECS: ServiceのUpdate • Serviceが使うTask DefinitionをUpdateすると、 新しいTaskをデプロイできる • 空いているリソースで新しいTaskを起動しなが ら、徐々に古いTaskを止めていく – Blue-Green deployment – 中間では、新旧のTaskが混在する
- 24. 24 Cluster Amazon ECS: ServiceのUpdate Service Task Definition:1 Task:1Task:1 Task:1
- 25. 25 Cluster Amazon ECS: ServiceのUpdate Service Task Definition:1 Task Definition:2 Task:1Task:1 Task:1
- 26. 26 Cluster Amazon ECS: ServiceのUpdate Service Task Definition:1 Task Definition:2 Task:1Task:1 Task:1Task:2 Task:2
- 27. 27 Cluster Amazon ECS: ServiceのUpdate Service Task Definition:1 Task Definition:2 Task:1Task:2 Task:2
- 28. 28 Cluster Amazon ECS: ServiceのUpdate Service Task Definition:1 Task Definition:2 Task:1Task:2 Task:2Task:2
- 29. 29 Cluster Amazon ECS: ServiceのUpdate Service Task Definition:1 Task Definition:2 Task:2 Task:2Task:2
- 30. 30 Case: Sony (Batch jobs)
- 31. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Konstantin Wilms, Enterprise Solutions Architect, AWS Ben Masek, CTO, Solutions Engineering Group, Sony PSA October 2015 CMP405 Containerizing Video Creating the Next Generation Video Transcoding Pipeline
- 36. Amazon ECS スケジュール Amazon EC2 処理 中心となるサービス群 保存 AWS Lambda 繋ぎ Amazon EFS & Amazon S3 100ms単位課金 インフラ管理不要 カスタムのバイナリ 中心となるストレージ S3の代わり コンテナとリンク 仕事の単位 複数のクラスタ エンコードに特化 スケジューラの代わり 一般的なコンテナ CPUを使い切る
- 37. アーキテクチャ全体 Source Storage Ingest Container Storage EC2 Bootstrap Transcode ECS EC2 Manager Target Stitch Manager 入力ファイルがやってくると、 イベント駆動なワークフローの 全体が開始される エンコードのためのAmazon ECSの起動用shellスクリプト Amazon S3は長期保管用、 Amazon EFSは一時的/コンテナ 用ストレージ 中央集権のマネージャー Amazon ECSのスケジューラを 利用 マネージャーは分割し統合する 補助的な処理に、Lambda 他のワークフローとも、イベン ト駆動で連携
- 38. Auto Scaling 単純な複数AZを使った スケール ワークロードの単位で決 定論的なスケジューリン グのために Min=Max=Desired いつでも簡単に捨てるこ とができる インスタンスのタイプと トータルメモリが重要 (~1.5MB エンコーダ)
- 39. Amazon EC2 インスタンスの初期化 起動時にECSクラスタ に参加させる ECSクラスタの各イン スタンスでAmazon EFSをマウントする どのAZかを知るため にメタデータにクエ リする これ以外の魔法は、 全てDockerと Lambdaで起こる
- 40. Amazon EFSでメディア処理 メディアのチャンク用 の一時的なストレージ メディアのソースと チャンクを移動する必 要がない Scales linearly 線形にスケールする 巨大なファイルを並列 で読み書きするのに理 想的 ECSクラスタにアタッチ するためのエンドポイ ントが各AZにある
- 41. Amazon ECSクラスタとスケジューラの設定 キャパシティの単位 共有ステータス、楽観的 並行制御 単純なスケジューラの設 計 – list*, describe*, run*, start* 10 CPU単位で我々は キャパシティ管理 ‘swimlane’ でエンコー ドのアーキテクチャをモ デル化
- 42. コンテナの構造 FROM ubuntu:14.04 ENV FLASK_JOB https://s3/xxx/jobs/h264.sh ENV FLASK_FRAG https://s3/xxx/media/frag-000.mp4 ENV FLASK_THREADS 4 RUN wget http://xxx.com/ffmpeg/builds/ffmpeg-static.tar.xz && tar xvfJ ffmpeg-static.tar.xz -C /usr/local/bin RUN apt-get -y install libav-tools RUN wget http://xxx.com/qpsnr.deb && dpkg -i qpsnr.deb 起動時の処理とスレッド 数を環境変数で設定 固定した環境変数により 簡単に開発とテストがで きる (デフォルトの呼び出 しはデバッグ) メモリの利用率を分析し て、ホストのリソースを 消費し尽くさないように 単一のAmazon ECS Task Definitionに紐付ける 420MBのイメージ、CLI 無しなら350MB CMD: s3 cp s3://…/bootstrap.sh . && chmod && bootstrap.sh
- 43. コンテナのデプロイパイプライン docker save 2408e7a48bac | sudo docker-squash -t transcode-toolkit:1.01s -verbose | docker load ロジックを起動処理で開始するので、 非常に単純 Kitematicが役立つ 容量削減と高速なデプロイのために、 静的リンクしたバイナリを使う 自動化も可能だし、手動でもいい Webhooks & ビルド自動化 Unionファイルシステムのレイヤが増 えるので、コンテナをSquashする publicかprivateのDocker registryを 使う
- 44. 44 まとめ
- 45. 45 Batch Processing Open-source Paas Real-time Image Transformation Solr Search Cluster PaaSGaming Engine Web Application Platform Microservices Backend Mapping Solution Amazon ECS: Some Examples…
- 47. 47 Amazon ECSまとめ • 多数の本番稼働実績、多くのパートナー – 2,000 containersが動いているお客様、エンタープライズ・大規模企業 • とてもシンプルなのに強力 – Master的なサーバ、Configuration KVS等の管理不要 – Service schedulerが、Blue-Greenデプロイや自動復旧してくれる • 次世代のサービスインフラ – Just like “Amazon EC2” in 2006!
- 48. 48 Appendix
- 49. 49 Case: Remind
- 50. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Eric Holmes & Michael Barrett, Remind October 2015 DVO308 Docker & ECS in Production How We Migrated Our Infrastructure from Heroku to AWS
- 51. Remind • A messaging platform for teachers. • Chat/announcements/files • Over 30 million users • Used actively in ~50% of U.S. public schools • Over 2 billion messages delivered • ~50 employees. ~30 engineers.
- 52. Heroku was great, but… • Every app on Heroku is publicly accessible • Databases need to be exposed to Internet traffic • Limited visibility and control
- 53. What we want from a PaaS • AWS • Flexibility • Shared patterns for deployment • Easy service operation • Containers/Docker
- 55. Design Goals • Easy to operate • Open source • Support 12-factor stateless apps (12factor.net) • Swappable scheduling back-ends • Stability! • Docker images as a unit of deployment
- 56. Empire :: V2 Scheduler Router Control Plane ECS ELB Heroku Platform API Spec + emp CLI
- 57. Empire :: V2 An open-source, self-hosted PaaS for running twelve-factor Docker apps backed by AWS services
- 58. Twelve-Factor Tenants I. Codebase II. Dependencies III. Config IV. Backing Services V. Build, release, run VI. Processes VII. Port binding VIII.Concurrency IX. Disposability X. Dev/prod parity XI. Logs XII. Admin processes
- 59. 12factor :: Dependencies “Explicitly declare and isolate dependencies” FROM ruby RUN apt-get install imagemagick RUN bundle install
- 60. 12factor :: Build, release, run “Strictly separate build and run stages” Empire
- 61. 12factor :: Build $ git push
- 62. 12factor :: Release, Run Config {} Release Amazon ECS
- 63. 12factor :: Release, Run $ cat Procfile web: ./bin/web worker: ./bin/worker $ aws ecs list-services arn:aws:ecs:us-east-1:***:service/api--web arn:aws:ecs:us-east-1:***:service/api--worker $ emp deploy org/api:latest Status: Created v1 release.
- 64. Service Discovery $ aws ecs describe-services --service api--web "loadBalancers": [{ "containerName": "web”, "containerPort": 9001, "loadBalancerName”: "2888...a31d4c” }] $ curl http://api Ok
- 65. 12factor :: Concurrency “Scale out via the process model” $ emp scale web=10 $ aws ecs describe-service --service api--web “desired-count”: 10
- 66. 12factor :: Dev/prod parity “Keep development, staging, and production as similar as possible” $ docker run --env-file <(emp env -a api) org/api
- 67. 12factor :: Logs “Treat logs as event streams” $ emp log “GET / HTTP/1.1” 200 STDOUT Amazon Kinesis
- 68. 12factor :: Admin processes “Run admin/management tasks as one-off processes” $ emp run rake db:migrate Migrated
- 70. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Frank Chen, Coursera Brennan Saeta, Coursera October 2015 CMP406 Amazon ECS at Coursera Powering a general-purpose near-line execution microservice, while defending against untrusted code
- 73. Bad Old Days of Batch Processing @ Coursera Cascade • PHP-based job runner • Originally ran in screen sessions • Polled APIs for new jobs • Forced restarts on regular basis due to unidentified memory leaks • Fragile and unreliable The early days…
- 74. Bad Old Days of Batch Processing @ Coursera Saturn • Scala scheduled batch job runner • Powered by Quartz Scheduler library • Better than Cascade, but… • All jobs ran on same JVM, causing interference The not- so early days?
- 75. What Else Did We Look At? Home-grown Tech • Tried, but proved to be unreliable • Difficult to handle coordination and synchronization • Powerful, but hard to productionize • Needs developers with experience • Designed for GCE first • Not a managed service, higher Ops load
- 76. Amazon ECS to the Rescue Little maintenance Integrated with rest of AWS Easy to develop for
- 77. However… Amazon ECS is a great building block, but we still need to build tools around it for our purposes.
- 78. What We Built: Iguazú Marissa Strniste (https://www.flickr.com/photos/mstrniste/5999464924) CC-BY-2.0 • Batch Job Scheduler for Amazon ECS • Immediately • Deferred (run once at X time) • Scheduled recurring (cron-like) • Programmatically accessible internally via our standard APIs and clients • Named for Iguazú falls • World’s largest waterfall by volume • We hope Iguazú handles a similar volume of jobs
- 79. Iguazú Frontend Iguazú Scheduler Iguazú Backend Iguazú: Architecture CassandraServices Services Iguazú Admin ECS Workers SQS ECS API Devs Users
- 80. Iguazú: Developer / Ops User Interface
- 81. Deploying Jobs Easy Deployment 1. Developers Merge into master. Done! Jenkins Build Steps: 1. Builds zip package from master 2. Prepares Docker image with zip file 3. Pushes image into Docker registry 4. Registers updated jobs with Amazon ECS API
- 82. Since April 2015… 65 jobs in production >1000 runs per day 44 different scheduled jobs
- 83. Programming Assignments at Coursera
- 84. The Security Challenge Compiling and running untrusted, arbitrary code in Amazon EC2 Would you like to compile and run C code from random people on the Internet on your servers?
- 85. What We Built: GrID Patrick Hoesly (https://www.flickr.com/photos/zooboing/5665221326/) CC-BY-2.0 • Service + architecture for grading programming assignments • Builds on Amazon ECS and Iguazú • Named for Tron’s “digital frontier” • Backronym: Grading Inside Docker
- 86. High-level GrID Architecture Learners GrID Iguazú S3 Bucket ECS API Grading Machines VPC Firewalls Production Acct GrID Grading Account