Scala, ECS, Docker: Delayed Execution @Coursera
0 likes477 views
The document discusses the evolution and implementation of Coursera's asynchronous execution systems, focusing on the transition to a next-generation framework called Iguazú designed for better resource management through Docker on Amazon ECS. It outlines the architectural changes made to enhance the secure execution of user-submitted code, including grading programming assignments while maintaining a focus on security and performance. Key features and lessons learned from building the platform are also highlighted, emphasizing the need for thorough monitoring and security measures.
Scala, ECS, Docker: Delayed Execution @Coursera
- 1. ECS & Docker: Secure Async Execution @ Brennan Saeta
- 2. InfoQ.com: News & Community Site • 750,000 unique visitors/month • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • News 15-20 / week • Articles 3-4 / week • Presentations (videos) 12-15 / week • Interviews 2-3 / week • Books 1 / month Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ scala-ecs-docker-coursera
- 3. Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide Presented at QCon London www.qconlondon.com
- 5. The Beginnings — 2012 10 courses 1 million learners worldwide 4 partners
- 6. Education at Scale 1,800 courses 18 million learners worldwide 140 partners
- 7. Outline • Evolution of Coursera’s nearline execution systems • Next-generation execution framework: Iguazú • Iguazú application deep dive: GrID — evaluating programming assignments
- 8. Key Takeaways • What is nearline execution, and why it is useful • Best practices for running containers in production in the cloud • Hardening techniques for securely operating container infrastructure at scale
- 9. A history of nearline execution
- 12. Early days - Requirements • Video re-encoding for distribution • Grade computation for 100,000+ learners • Pedagogical data exports for courses
- 16. Upgrading to Scala Re-architecting delayed execution for our 2nd generation learning platform.
- 17. Upgrading to the JVM • Leverage mature Scala & JVM ecosystems for code sharing • JVM much more reliable (no memory leaks) • New job model: scheduled recurring jobs. • Named: Saturn
- 18. Saturn Architecture Service A Service B Service C C* Online Serving Scala/micro-service architecture C*
- 19. Saturn Architecture Service A Service B Service C C* Online Serving Scala/micro-service architecture Saturn C*
- 20. Saturn Architecture Service A Service B Service C C* Saturn C* ZK Ensemble
- 21. Saturn Architecture Saturn Leader ZK Ensemble Service A Service B Service C C*C*
- 22. Problems with Saturn • Single master meant naïve implementation ran all jobs in same JVM • Huge CPU contention @ top of the hour • OOM Exceptions & GC issues
- 23. Enter: Docker Containers allow for resource isolation! CC-by-2.0 https://www.flickr.com/photos/photohome_uk/1494590209
- 24. Supported Features Platform Saturn Docker Amazon ECS Iguazú Run code ✅ ✅ ✅ ✅ Resource Isolation ❌ ✅ ✅ ✅ Clusters / HA ☑️ ❌ ✅ ✅ Great developer workflow ✅ ❌ ❌ ✅ Scheduled Jobs ✅ ❌ ❌ ✅
- 25. Supported Features Platform Saturn Docker Amazon ECS Iguazú Run code ✅ ✅ ✅ ✅ Resource Isolation ❌ ✅ ✅ ✅ Clusters / HA ✅ ❌ ✅ ✅ Great developer workflow ✅ ❌ ❌ ✅ Scheduled Jobs ✅ ❌ ❌ ✅
- 26. Supported Features Platform Saturn Docker Amazon ECS Iguazú Run code ✅ ✅ ✅ ✅ Resource Isolation ❌ ✅ ✅ ✅ Clusters / HA ✅ ❌ ✅ ✅ Great developer workflow ✅ ❌ ❌ ✅ Scheduled Jobs ✅ ❌ ❌ ✅
- 27. Supported Features Platform Saturn Docker Amazon ECS Iguazú Run code ✅ ✅ ✅ ✅ Resource Isolation ❌ ✅ ✅ ✅ Clusters / HA ✅ ❌ ✅ ✅ Great developer workflow ✅ ❌ ❌ ✅ Scheduled Jobs ✅ ❌ ❌ ✅
- 28. Supported Features Platform Saturn Docker Amazon ECS ??? Run code ✅ ✅ ✅ ✅ Resource Isolation ❌ ✅ ✅ ✅ Clusters / HA ✅ ❌ ✅ ✅ Great developer workflow ✅ ❌ ❌ ✅ Scheduled Jobs ✅ ❌ ❌ ✅
- 29. Solution: Iguazú Marissa Strniste (https://www.flickr.com/photos/mstrniste/5999464924) CC-BY-2.0
- 30. Solution: Iguazú • Framework & service for asynchronous execution • Optimized Scala developer experience for Coursera • Unified scheduler supports: • Immediate execution (nearline) • Scheduled recurring execution (cron-like) • Deferred execution (run once @ time X) Marissa Strniste (https://www.flickr.com/photos/mstrniste/5999464924) CC-BY-2.0
- 31. Iguazú Architecture Iguazú Frontend Iguazú Scheduler Iguazú Backend CassandraServices Services Iguazú Admin Iguazú Workers SQS ECS API Devs Users
- 32. Iguazú Architecture Iguazú Frontend Iguazú Scheduler Iguazú Backend CassandraServices Services Iguazú Admin Iguazú Workers SQS Queue ECS API Devs Users
- 33. Iguazú Architecture Iguazú Frontend Iguazú Scheduler Iguazú Backend CassandraServices Services Iguazú Admin Iguazú Workers ECS API Devs Users SQS Queue
- 34. Iguazú Architecture Iguazú Frontend Iguazú Scheduler Iguazú Backend CassandraServices Services Iguazú Admin Iguazú Workers ECS API Devs Users ZK Ensemble SQS Queue
- 35. Iguazú Architecture Iguazú Frontend Iguazú Scheduler Iguazú Backend CassandraServices Services Iguazú Admin Iguazú Workers ECS API Devs Users ZK Ensemble SQS Queue
- 37. Autoscaling ⇄ Iguazú ⇆ ECS Iguazu ECS APIAutoscaling EC2 Worker EC2 Worker Shutdown Lifecycle Notification Poll Worker Job Status All finished Proceed Term- inate EC2 Worker
- 38. Failure in Nearline Systems • Most jobs are non-idempotent • Iguazú: At most once execution • Time-bounded delay • Future: At least once execution • With caveats
- 39. Iguazú adoption by the numbers ~100 jobs in production >1000 runs per day >100 different job schedules
- 40. Iguazú Applications Nearline Jobs • Pedagogical Instructor Data Exports • System Integrations • Course Migrations Scheduled Recurring Jobs • Course Reminders • System Integrations • Payment reconciliation • Course translations • Housekeeping • Build artifact archival • A/B Experiments
- 41. While containers may help you on your journey, they are not themselves a destination.CC-by-2.0 https://www.flickr.com/photos/usoceangov/5369581593
- 42. Writing an Iguazu Job class AbReminderJob @Inject() (abClient: AbClient, email: EmailAPI) extends AbstractJob { override val reservedCpu = 1024 // 1 CPU core override val reservedMemory = 1024 // 1 GB RAM def run(parameters: JsValue) = { val experiments = abClient.findForgotten() logger.info(s"Found ${experiments.size} forgotten experiments.") experiments.foreach { experiment => sendReminder(experiment.owners, experiment.description) } } }
- 43. Writing an Iguazu Job class AbReminderJob @Inject() (abClient: AbClient, email: EmailAPI) extends AbstractJob { override val reservedCpu = 1024 // 1 CPU core override val reservedMemory = 1024 // 1 GB RAM def run(parameters: JsValue) = { val experiments = abClient.findForgotten() logger.info(s"Found ${experiments.size} forgotten experiments.") experiments.foreach { experiment => sendReminder(experiment.owners, experiment.description) } } }
- 44. Writing an Iguazu Job class AbReminderJob @Inject() (abClient: AbClient, email: EmailAPI) extends AbstractJob { override val reservedCpu = 1024 // 1 CPU core override val reservedMemory = 1024 // 1 GB RAM def run(parameters: JsValue) = { val experiments = abClient.findForgotten() logger.info(s"Found ${experiments.size} forgotten experiments.") experiments.foreach { experiment => sendReminder(experiment.owners, experiment.description) } } }
- 45. Writing an Iguazu Job class AbReminderJob @Inject() (abClient: AbClient, email: EmailAPI) extends AbstractJob { override val reservedCpu = 1024 // 1 CPU core override val reservedMemory = 1024 // 1 GB RAM def run(parameters: JsValue) = { val experiments = abClient.findForgotten() logger.info(s"Found ${experiments.size} forgotten experiments.") experiments.foreach { experiment => sendReminder(experiment.owners, experiment.description) } } }
- 46. Writing an Iguazu Job class AbReminderJob @Inject() (abClient: AbClient, email: EmailAPI) extends AbstractJob { override val reservedCpu = 1024 // 1 CPU core override val reservedMemory = 1024 // 1 GB RAM def run(parameters: JsValue) = { val experiments = abClient.findForgotten() logger.info(s"Found ${experiments.size} forgotten experiments.") experiments.foreach { experiment => sendReminder(experiment.owners, experiment.description) } } }
- 47. Testing an Iguazu job
- 48. The Hollywood Principle applies to distributed systems. CC-by-2.0 https://www.flickr.com/photos/raindog808/354080327
- 49. Deploying a new Iguazu Job • Developer • merge into master… done • Jenkins Build Steps • Compile & package job JAR • Prepare Docker image • Pushes image into registry • Register updated job with Amazon ECS API
- 50. Invoking an Iguazú Job // invoking a job with one function call // from another service via REST framework RPC val invocationId = iguazuJobInvocationClient .create(IguazuJobInvocationRequest( jobName = "exportQuizGrades", parameters = quizParams))
- 51. A clean environment increases reliability.CC-by-2.0 https://www.flickr.com/photos/raindog808/354080327
- 52. Evaluating Programming Assignments An application of Iguazú
- 55. Design Goals Elastic Infrastructure No Maintenance Near Real-time Secure Infrastructure
- 56. Design Goals Elastic Infrastructure No Maintenance Near Real-time Secure Infrastructure
- 57. Design Goals Elastic Infrastructure No Maintenance Near Real-time Secure Infrastructure
- 58. Solution: GrID Patrick Hoesly (https://www.flickr.com/photos/zooboing/5665221326/) CC-BY-2.0 • Service + framework for grading programming assignments • Builds on Iguazú • Named for Tron’s “digital frontier” • Backronym: Grading Inside Docker
- 59. High-level GrID Architecture Learners GrID Iguazú S3 Bucket ECS APIs Grading MachinesVPC Firewalls Coursera Production Account Coursera GrID Grading Account
- 60. High-level GrID Architecture Learners GrID Iguazú S3 Bucket ECS APIs Grading MachinesVPC Firewalls Coursera Production Account Coursera GrID Grading Account
- 61. High-level GrID Architecture Learners GrID Iguazú S3 Bucket ECS API Grading MachinesVPC Firewalls Production Acct GrID Grading Account
- 62. High-level GrID Architecture Learners GrID Iguazú S3 Bucket ECS API Grading Machines VPC Firewalls Production Acct GrID Grading Account
- 63. Design Goals Elastic Infrastructure No Maintenance Near Real-time Secure Infrastructure
- 65. The Security Challenge Compiling and running untrusted, arbitrary code on our cluster in near real time. Would you like to compile and run C code from random people on the Internet on your servers?
- 66. FROM redis FROM ubuntu:latest FROM jane’s-image
- 67. Security Assumptions • Run arbitrary binaries • Instructor grading scripts may have vulnerabilities • ∴ Grading code is untrusted • Unknown vulnerabilities in Docker and Linux name-spacing and/or container implementation
- 68. Security Goals Prevent submitted code from: • impacting the evaluation of other submissions. • disrupting the grading environment (e.g., DoS) • affecting the rest of the Coursera learning platform
- 69. Grading assignment submissions CC-by-2.0 https://www.flickr.com/photos/dherholz/4367511580/
- 71. CPU CPU CPU CPU RAM Alice’s Container Alice’s Submission Grader Bob’s Container Bob’s Submission Grader Mallory’s Container Mallory’s Submission Grader Kernel Disk
- 72. CPU CPU CPU CPU RAM Alice’s Container Alice’s Submission Grader Bob’s Container Bob’s Submission Grader Mallory’s Container Mallory’s Submission Grader Kernel Disk
- 73. CPU cgroups CPU cgroups RAM — cgroups Alice’s Container Alice’s Submission Grader Bob’s Container Bob’s Submission Grader Mallory’s Container Mallory’s Submission Grader Kernel Disk
- 74. CPU cgroups CPU cgroups RAM — cgroups Alice’s Container Alice’s Submission Grader Bob’s Container Bob’s Submission Grader Mallory’s Container Mallory’s Submission Grader Kernel Disk
- 75. CPU cgroups CPU cgroups RAM — cgroups Alice’s Container Alice’s Submission Grader Bob’s Container Bob’s Submission Grader Mallory’s Container Mallory’s Submission Grader Kernel Disk — blkio limits & btrfs quotas
- 76. CPU cgroups CPU cgroups RAM — cgroups Alice’s Container Alice’s Submission Grader Bob’s Container Bob’s Submission Grader Mallory’s Container Mallory’s Submission Grader Kernel Disk — blkio limits & btrfs quotas
- 77. Attacks: Kernel Resource Exhaustion • Open file limits per container (nofile) • nproc Process limits • Limit kernel memory per cgroup • Limit execution time
- 79. CPU cgroups CPU cgroups RAM — cgroups Alice’s Container Alice’s Submission Grader Bob’s Container Bob’s Submission Grader Mallory’s Container Mallory’s Submission Grader Kernel — cgroups, ulimits Disk — blkio limits & btrfs quotas Network
- 80. Attacks: Network attacks Attacks: • Bitcoin mining • DoS attacks on other systems • Access Amazon S3 and other AWS APIs Defense: • Deny network access
- 81. Docker Network Modes NetworkDisabled too restrictive • Some graders require local loopback • Feature also deprecated --net=none + deny net_admin + audit network • Isolation via Docker creating an independent network stack for each container github.com/coursera/amazon-ecs-agent
- 85. Defense in Depth • Mandatory Access Control (App Armor) • Allows auditing or denying access to a variety of subsystems • Drop capabilities from bounding set • No need for NET_BIND_SERVICE, CAP_FOWNER, MKNOD • Deny root within container
- 86. Deny Root Escalations • We modify instructor grader images before allowing them to be run • Clears setuid • Inserts C wrapper to drop privileges from root and redirect stdin/stdout/stderr • Run cleaning job on another Iguazú cluster • Run Docker in Docker! • Docker 1.10 adds User Namespaces
- 87. If all else fails… • Utilizes VPC security measures to further restrict network access • No public internet access • Security group to restrict inbound/outbound access • Network flow logs for auditing • Separate AWS account • Run in an Auto Scaling group • Regularly terminate all grading EC2 instances
- 88. Other Security Measures • Utilize AWS CloudTrail for audit logs • Third-party security monitoring (Threat Stack) • No one should log in, so any TTY is an alert • Penetration testing by third-party red team (Synack)
- 89. Lessons Learned - GrID • Building a platform for code execution is hard! • Carefully monitor disk usage • Run the latest kernels • Latest security patches • btrfs wedging on older kernels • Default Ubuntu 14.04 kernel not new enough!
- 90. Reliable deploy tooling pays for itself.
- 91. Thank you! Brennan Saeta github/saeta @bsaeta saeta@coursera.org Frank Chen github/frankchn @frankchn frankchn@coursera.org GrID lead Iguazú Lead
- 92. Questions? Brennan Saeta github/saeta @bsaeta saeta@coursera.org Frank Chen github/frankchn @frankchn frankchn@coursera.org GrID lead Iguazú Lead
- 93. Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/scala- ecs-docker-coursera