How Yelp uses Mesos to Power its SOA Infrastructure
- 1. How Yelp Uses Mesos to Power Its SOA Architecture
- 2. Basic Requirements of a Service- Oriented Architecture
- 3. Scheduling Decide which hosts run a service
- 4. Delivery Put the code on the host and run it
- 5. Discovery Tell clients where your service is running
- 6. Alerting Tell humans when things are broken
- 7. ● Scheduling: puppet manifests ● Delivery: ○ service code: rsync + cron jobs ○ dependencies: puppet + apt ● Discovery: hand-crafted LB configs + VIPs ● Alerting: hand-crafted nagios configs Ye Olde Yelp SOA
- 8. ● Scheduling: yelpsoa-configs ● Delivery: ○ service code: rsync + cron jobs ○ dependencies: virtualenv, some puppet ● Discovery: Smartstack ● Alerting: Sensu based on yelpsoa-configs Not-so-Olde Yelp SOA
- 9. This works, but has some problems
- 10. ● All services on all machines? ○ tough if your memory footprint is too big ● Some subset on each? ○ What are the biggest resource users? ● Which services actually need resources? What if you buy more machines? How do you redistribute load?
- 11. ● Hope you have a box with spare capacity ● Spin up more boxes? See previous slide. What if a service needs more CPU?
- 12. ● virtualenv for python libs ● what about C shared libraries? What if services have conflicting dependencies?
- 13. ● virtualenv for python libs ● What about different interpreters? ● Different HTTP server? What if a service has unusual dependencies?
- 14. PaaSTA: Yelp's Mesos SOA glue
- 15. PaaSTA ● Scheduling: Mesos+Marathon ● Delivery: Docker ● Discovery: Smartstack ● Alerting: Sensu based on yelpsoa-configs
- 16. Scheduling in PaaSTA: Mesos and Marathon ● Mesos is an "SDK for distributed systems", batteries not included. ● Requires a framework ○ Marathon (like ASG for Mesos) ○ Chronos (Periodic tasks) ● Supports Docker as task executor
- 17. ● Containers: like lightweight VMs ● Provide language (Dockerfile) for describing container image ● Reproducible builds (mostly) ● Provides software flexibility Delivery in Paasta: Docker
- 18. just kidding, there's more to it than that All done!
- 19. How do we configure Marathon?
- 20. How do we produce & distribute Docker images?
- 21. How does SmartStack know about our containers?
- 22. How do we monitor this?
- 23. How do we configure Marathon? ● Need a wall around Marathon: it has root on your entire cluster. ● Cron job ● Combines yelpsoa-configs and currently- blessed docker image
- 24. ● # instances ● cpu ● mem ● healthcheck timeouts, grace period ● bounce strategy ● cmd / args marathon.yaml
- 25. ● Distribution with private registry ● S3 bucket shared among all environments ● Bless images by creating git tags ○ 1:1 git commit <-> docker image Building/shipping Docker images
- 26. ● configure_nerve.py queries local mesos- slave API ● This means registrations work even when Mesos master or Marathon is down ● Provides backwards compatibility with older SOA Registering with SmartStack
- 28. ● Describe end goal, not path ● Helps achieve fault tolerance "Deploy 12abcd34 to prod" vs. "Commit 12abcd34 should be running in prod" Gas pedal vs. Cruise Control Declarative control
- 29. Demo
- 30. Reading Comprehension Question A. To describe how cool Yelp's PaaSTA system is B. To tease viewers about a system that is not open source yet? D. To confuse viewers into making them consider yet another docker-based PaaS C. To Inspire viewers to build their own bespoke PaaS based on Mesos with some of these ideas.