ContainerDayVietnam2016: Docker at scale with Mesos
1 like573 views
Mesos is an open source cluster manager that provides efficient resource isolation and sharing across distributed applications or frameworks. The document discusses how to run Docker containers on Mesos at scale, including using Marathon for container orchestration. Some key lessons learned are around choosing a container style, service discovery with Mesos-DNS, networking with Calico, and managing secrets, deployments, and Docker limitations.
ContainerDayVietnam2016: Docker at scale with Mesos
- 1. Docker at scale with Mesos Phạm Tuấn Anh Lead engineer @ VCCloud Platform Services
- 2. $ whoami • Lead engineer at VCCloud Platform Services • Build & work with large-scale/production platform services for about 6 years • Services: Storage, DNS, CDN, DDoS Protection, Logging, Private PaaS
- 3. Agenda 1. What is Mesos? 2. How to run docker containers on Mesos 3. Lessons learned 4. Q&A
- 4. 1. What is Mesos? 2. How to run docker containers on Mesos 3. Lessons learned 4. Q&A
- 12. Powered by Mesos: • Apple • Atlassian • CERN • Cisco • Cloudflare • Foursquare • Groupon • HubSpot • Netflix • Opera • Paypal • Shopee • Shopify • Saleforce • Twitter • Uber • Vimeo • Verizon • Weibo • Yelp • Xiaomi http://mesos.apache.org/documentation/latest/powered-by-mesos/
- 13. Solomon Hykes, Founder & CTO of Docker, said at DockerCon EU 12/2014: Mesos is the “gold standard” for large-scale production clusters running containers
- 14. Mesos is: • Open source, top-level Apache project • Proven at massive scale • Multi-resource (CPU, RAM, Disk, GPU) scheduling • Simplified operational model • Cross platform • Native Docker support
- 17. • PaaS: Marathon, Aurora, Kubernetes, Swarm • Big Data: Hadoop, Spark, Storm • Database: Cassandra, ArangoDB, • ElasticSearch, Chronos, Jenkins, Kafka… Mesos Frameworks:
- 18. 1. What is Mesos? 2. How to run docker containers on Mesos 3. Lessons learned 4. Q&A
- 19. • Apache Aurora • Google Kubernetes • HubSpot Singularity • Mesosphere Marathon
- 26. Features: • Start, stop, scale, update, rollback apps • Rolling deploy / restart • Constraints • Health checks • App dependencies • Highly available, no SPoF
- 30. 1. What is Mesos? 2. How to run docker containers on mesos 3. Lessons learned 4. Q&A
- 31. – Immutable Infrastructure / CodeShip “Some of the major challenges today when building infrastructure are predictability, scalability and automated recovery.”
- 32. • Container style • Service discovery • Networking Lessons Learned • Managing secrets • App deployment • Docker annoyances
- 33. Choose a container style: • "thin" single-process container • "fat" container • sshd in container
- 34. • Container style • Service discovery • Networking Lessons Learned • Managing secrets • App deployment • Docker annoyances
- 35. Services Discovery • Mesos-DNS: • <app_id>.<app_group>.marathon.mesos • record types: A / SRV • simple & stateless
- 37. • Container style • Service discovery • Networking Lessons Learned • Managing secrets • App deployment • Docker annoyances
- 38. Networking • Project Calico: • IP-per-container • Pure Layer 3
- 42. • Container style • Service discovery • Networking Lessons Learned • Managing secrets • App deployment • Docker annoyances
- 47. • Container style • Service discovery • Networking Lessons Learned • Managing secrets • App deployment • Docker annoyances
- 48. App deployment • ansible-playbook • marathon.json • docker tag name = git commit hash • 1 mesos cluster (3 VMs) per dev • deploy to servers → deploy to marathon
- 50. marathon.json
- 60. Monitoring • Container’s RAM used
- 61. Stateful Containers • Local persistent volumes
- 62. • Container style • Service discovery • Networking Lessons Learned • Managing secrets • App deployment • Docker annoyances
- 63. Docker 👎 • restart / upgrade docker without stopping the running containers • old images / tags cleanup • container’s stdout / stderr log files • docker pull • docker hub
- 64. "Normal" container vs cloud-native container: • No manual steps • Service discovery friendly • Config files should be optional • Using env vars for config
- 65. 1. What is Mesos? 2. How to run docker containers on Mesos 3. Lessons learned 4. Q&A
- 67. Links • Mesos: http://mesos.apache.org/ • Marathon: https://mesosphere.github.io/marathon/ • Project Calico: https://www.projectcalico.org/ • Mesos-DNS: https://github.com/mesosphere/mesos-dns • A Healthy Platform Checklist: https://gist.github.com/andy-pham/ 04dedff872b33c07a31277afb900950b