Mesos and the Architecture of the New Datacenter
2 likes820 views
The document discusses Apache Mesos, a top-level Apache project designed as a cluster resource negotiator for scalable data centers. It covers its architecture, functionalities, and the evolution of the Data Center Operating System (DC/OS) which supports modern distributed applications. Key features include resource management, fault tolerance, and support for multiple frameworks such as Hadoop, Spark, and Docker.
Mesos and the Architecture of the New Datacenter
- 1. 1 Mesos and the Architecture of the New Datacenter @joerg_schad
- 2. © 2016 Mesosphere, Inc. All Rights Reserved. 2
- 3. Flink Cassandra Rails Spark memcached
- 4. Flink Cassandra Rails Spark memcached
- 5. © 2015 Mesosphere, Inc. All Rights Reserved. 5 A naive approach to handling varied app requirements: static partitioning. This can cope with heterogeneity, but is very expensive. KEEP IT STATIC time
- 6. © 2015 Mesosphere, Inc. All Rights Reserved. 6 Maintaining sufficient headroom to handle peak workloads on all partitions leads to poor utilization overall. KEEP IT STATIC time
- 7. © 2015 Mesosphere, Inc. All Rights Reserved. 7 Multiple frameworks can use the same cluster resources, with their share adjusting dynamically. SHARED RESOURCES time
- 8. © Gerard Julien/AFP Run everything in containers!
- 9. © 2015 Mesosphere, Inc. All Rights Reserved. What about container management? 9
- 10. © 2015 Mesosphere, Inc. All Rights Reserved. What about container management? 10
- 11. © 2015 Mesosphere, Inc. All Rights Reserved. 11 MESOS: ORIGINS
- 12. © 2015 Mesosphere, Inc. All Rights Reserved. 12 THE BIRTH OF MESOS TWITTER TECH TALK The grad students working on Mesos give a tech talk at Twitter. March 2010 APACHE INCUBATION Mesos enters the Apache Incubator. Spring 2009 CS262B Ben Hindman, Andy Konwinski and Matei Zaharia create “Nexus” as their CS262B class project. MESOS PUBLISHED Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center is published as a technical report. September 2010 December 2010 DC/OS April 2016
- 13. © 2015 Mesosphere, Inc. All Rights Reserved. 13 Sharing resources between batch processing frameworks ● Hadoop ● MPI ● Spark What does an operating system provide? ● Resource management ● Programming abstractions ● Security ● Monitoring, debugging, logging TECHNOLOGY VISION
- 14. © 2015 Mesosphere, Inc. All Rights Reserved. • A top-level Apache project • A cluster resource negotiator • Scalable to 10,000s of nodes • Fault-tolerant, battle-tested • An SDK for distributed apps • Native Docker support 14 Apache Mesos
- 15. © 2015 Mesosphere, Inc. All Rights Reserved. 15 ARCHITECTURE MESOS FUNDAMENTALS
- 16. © 2015 Mesosphere, Inc. All Rights Reserved. 16 ARCHITECTURE MESOS FUNDAMENTALS ● Agents advertise resources to Master ● Master offers resources to Framework ● Framework rejects/uses resources ● Agents report task status to Master
- 17. © 2015 Mesosphere, Inc. All Rights Reserved. 17 ARCHITECTURE MESOS FUNDAMENTALS ● Agents advertise resources to Master ● Master offers resources to Framework ● Framework rejects/uses resources ● Agents report task status to Master
- 18. © 2015 Mesosphere, Inc. All Rights Reserved. 18 ARCHITECTURE MESOS FUNDAMENTALS ● Agents advertise resources to Master ● Master offers resources to Framework ● Framework rejects/uses resources ● Agents report task status to Master
- 19. © 2015 Mesosphere, Inc. All Rights Reserved. 19 ARCHITECTURE MESOS FUNDAMENTALS ● Agents advertise resources to Master ● Master offers resources to Framework ● Framework rejects/uses resources ● Agents report task status to Master
- 20. © 2015 Mesosphere, Inc. All Rights Reserved. 20 ARCHITECTURE MESOS FUNDAMENTALS ● Agents advertise resources to Master ● Master offers resources to Framework ● Framework rejects/uses resources ● Agents report task status to Master
- 21. © 2016 Mesosphere, Inc. All Rights Reserved. 21 YARN
- 22. © 2016 Mesosphere, Inc. All Rights Reserved. 22 YARN
- 23. © 2015 Mesosphere, Inc. All Rights Reserved. 23 Distributed Systems SDK„Developers will ask for an API not for a VM to run their software“
- 24. © 2015 Mesosphere, Inc. All Rights Reserved. 24 ● Focus on application logic, not on data-center structure ● Avoid networking-related code ● Reuse of built-in fault-tolerance and high availability ● Reuse distributed (infrastructure) frameworks (e.g., storage) Distributed System SDK Mesos
- 25. © 2015 Mesosphere, Inc. All Rights Reserved. 25 Scheduler Interface ● Utilise resources ● ResourceOffer ● React on Tasks Events ● TaskStatus Updates Executor Interface (optional) ● Task life-cycle and monitoring ● Provided executors ● Command Executor ● Docker Executor First Framework/Service Mesos
- 26. © 2015 Mesosphere, Inc. All Rights Reserved. 26
- 27. 27 WHERE DC/OS CAME FROM PHYSICAL (x86) VIRTUAL HYPERSCALEMAINFRAME SERVER VIRTUAL MACHINEPARTITION (LPAR) UNIT OF INTERACTION ● ERP, CRM, PRODUCTIVITY, MAIL & WEB SERVER ● LINUX, WINDOWS ● DATA / TRANSACTION PROCESSING ● UNIX, IBM OS/360 DEFINITIVE APPS AND OS ● ERP, CRM, PRODUCTIVITY, MAIL & WEB SERVER ● HYPERVISOR + GUEST OS ● BIG DATA, INTERNET OF THINGS, MOBILE APPS ● ??? ???DATACENTER NEW FORM FACTOR FOR DEVELOPING AND RUNNING APPS ● BIG DATA, INTERNET OF THINGS, MOBILE APPS ● DATACENTER OPERATING SYSTEM (DC/OS)
- 28. Datacenter Operating System (DC/OS) Distributed Systems Kernel (Mesos) DC/OS ENABLES MODERN DISTRIBUTED APPS Big Data + Analytics EnginesMicroservices (in containers) Streaming Batch Machine Learning Analytics Functions & Logic Search Time Series SQL / NoSQL Databases Modern App Components Distributed systems kernel to abstract resources Ecosystem of frameworks & apps Consistent architecture to run on top of kernel User Interface (GUI & CLI) Core system services (e.g., distributed init, cron, service discovery, package mgt & installer, storage) Any Infrastructure (Physical, Virtual, Cloud) 28
- 29. 29 THE BASICS DC/OS is … ● 100% open source (ASL2.0) + A big, diverse community ● An umbrella for ~30 OSS projects + Roadmap and designs + The build tool chain + Docs and tutorials ● Not limited in any way ● Familiar, with a few new features
- 30. 30
- 31. 31 THE UNIVERSE
- 32. 32
- 33. © 2015 Mesosphere, Inc. All Rights Reserved. 33 Questions? Thank you!