Apache Mesos: Architecture, Design and Code Review
Download as PPTX, PDF0 likes493 views
This document provides an overview of the Apache Mesos distributed systems framework. It discusses Mesos' architecture for sharing computer clusters between multiple frameworks like Hadoop and MPI. Mesos uses a master-slave model with the master offering resources to frameworks, which can then launch tasks on slave nodes. The document reviews Mesos' design decisions, implementation in C++, strengths like modularity, and weaknesses like some duplicate abstractions. It also analyzes Mesos' source code structure and identifies areas for design improvement.
![25
References
• [1] Hindman, B., Konwinski, A., Matei, Z., Ghodsi, A., D.Joseph, A., Katz, R., …
Stoica, I. (2011). Mesos: A platform for fine-grained resource sharing in the data
center. Proceedings of the …, 32. https://doi.org/10.1109/TIM.2009.2038002
• [2] Apache Mesos. (n.d.). Retrieved from http://mesos.apache.org/ ,2018
• [3] Understand. (n.d.). Retrieved from https://scitools.com/, 2017
• [4] G. Suryanarayana, G. Samarthyam, and T. Sharma, Refactoring for Software
Design Smells: Managing Technical Debt, 1st ed. San Francisco, CA, USA: Morgan
Kaufmann Publishers Inc., 2014.
• [5] I. Sommerville, Software Engineering, 9st ed. 2010.
October 2018 Apache Mesos - Morteza Zakeri - Software Architectures 24](https://image.slidesharecdn.com/zakerimesosarchitecture970730-181023084035/85/Apache-Mesos-Architecture-Design-and-Code-Review-24-320.jpg)
Apache Mesos: Architecture, Design and Code Review
- 1. Apache Mesos: Architecture, Design and Code Review By: Morteza Zakeri Software Architectures Course Instructor: Dr. Mehrdad Ashtiani Iran University of Science and Technology School of Computer Engineering Fall 2018
- 2. 25 Outline • Introduction • Architectural review • Implementation • Design pros and cons • Reverse engineering Mesos • Conclusion • References October 2018 Apache Mesos - Morteza Zakeri - Software Architectures 2
- 3. 25 What is Apache Mesos? • A platform for sharing clusters between multiple diverse cluster computing frameworks. • A distributed system for running and building other distributed systems! • E.g. Hadoop + MPI + … October 2018 Apache Mesos - Morteza Zakeri - Software Architectures Apache Mesos Node Node Node Node Hadoop MPI … Node Node Hadoop Node Node MPI … Introduction Architecture Implementation Design pros & … Reversing Mesos Conclusion 3
- 4. 25 Motivation to sharing a cluster • We live in the high performance computing (HPC) era! • There is no single framework that optimal for all applications. • We want to run multiple frameworks in a single cluster. • Sharing improves cluster utilization. • Allows applications to share access to large datasets. • May be too costly to replicate across distinct clusters! October 2018 Apache Mesos - Morteza Zakeri - Software Architectures Introduction Architecture Implementation Design pros & … Reversing Mesos Conclusion 4
- 5. 25 Common solutions for sharing a cluster 1. Statically partition the cluster and run one framework per partition. 2. Allocate a set of VMs to each framework. • Disadvantages: • No high utilization, • No efficient data sharing. • Better solution: • Dynamic fine-grained sharing Mesos! Introduction Architecture Implementation Design pros & … Reversing Mesos Conclusion October 2018 Apache Mesos - Morteza Zakeri - Software Architectures 5
- 6. 25 Fine-grained sharing Framework 1 Framework 2 Framework 3 321 654 987 Storage System (e.g. HDFS) 321 654 987 Storage System (e.g. HDFS) Fw. 1 Fw. 1Fw. 3 Fw. 3 Fw. 2Fw. 2 Fw. 2 Fw. 1 Fw. 3 Fw. 2Fw. 3 Fw. 1 Fw. 1 Fw. 2Fw. 2 Fw. 1 Fw. 3 Fw. 3 Fw. 3 Fw. 2 Fw. 2 Static partitioning Mesos: dynamic sharing Introduction Architecture Implementation Design pros & … Reversing Mesos Conclusion October 2018 Apache Mesos - Morteza Zakeri - Software Architectures 6
- 7. 25 Mesos design decisions • Small microkernel-like core • Implements fine-grained sharing • Allocation at the level of tasks within a job • High-availability and fault-tolerance • Simple and scalable application-controlled scheduling mechanism. Introduction Architecture Implementation Design pros & … Reversing Mesos Conclusion October 2018 Apache Mesos - Morteza Zakeri - Software Architectures 7
- 8. 25 Mesos design choices • Centralized scheduler vs. Resource Offer • Resource Offer: • Delegating control over scheduling to the frameworks. • Overall architecture (communication model): master-slave • RMI (on TCP/IP) Introduction Architecture Implementation Design pros & … Reversing Mesos Conclusion October 2018 Apache Mesos - Morteza Zakeri - Software Architectures 8
- 9. 25 Apache Mesos architecture Mesos architecture diagram, showing two running frameworks (Hadoop and MPI). Introduction Architecture Implementation Design pros & … Reversing Mesos Conclusion October 2018 Apache Mesos - Morteza Zakeri - Software Architectures User Interface (Dashboard) 9
- 10. 25 Apache Mesos architecture: Resource Offer • The master implements fine-grained sharing across frameworks using resource offers. • Each Resource Offer is a list of free resources on multiple slaves. • The master decides how many resources to offer to each framework according to an organizational policy. • Mesos lets organizations define their own policies via a pluggable allocation module. Introduction Architecture Implementation Design pros & … Reversing Mesos Conclusion October 2018 Apache Mesos - Morteza Zakeri - Software Architectures 10
- 11. 25 Apache Mesos architecture: Resource Offer • Each framework running on Mesos consists of two components: • A scheduler that registers with the master to be offered resources. • An executor process that is launched on slave nodes to run the framework’s tasks. • Frameworks’ schedulers select which of the offered resources to use. Introduction Architecture Implementation Design pros & … Reversing Mesos Conclusion October 2018 Apache Mesos - Morteza Zakeri - Software Architectures 11
- 12. 25 Resource Offer: an example Introduction Architecture Implementation Design pros & … Reversing Mesos Conclusion October 2018 Apache Mesos - Morteza Zakeri - Software Architectures 12
- 13. 25 Apache Mesos API (early versions) Introduction Architecture Implementation Design pros & … Reversing Mesos Conclusion October 2018 Apache Mesos - Morteza Zakeri - Software Architectures 13
- 14. 25 Implementation • About 10,000 lines of C++ code (early versions). • Today version (full): > 250,000 • Runs on Linux, Solaris and Mac OS X. • Frameworks ported: Apache Hadoop, MPICH2, Torque • Open Source :) Introduction Architecture Implementation Design pros & … Reversing Mesos Conclusion October 2018 Apache Mesos - Morteza Zakeri - Software Architectures 14
- 15. 25 Implementation: why C++? I. Performance II. Maintainability (static typing) III. Object oriented (and its benefits) IV. Interfaces to low-level OS (for isolation, etc.) V. Interoperability with other languages and frameworks Introduction Architecture Implementation Design pros & … Reversing Mesos Conclusion October 2018 Apache Mesos - Morteza Zakeri - Software Architectures 15
- 16. 25 Design good points • Uses object oriented design elements: • Abstraction, modularization, and encapsulation. • Uses some design patterns: • E.g. strategy pattern. • + Provides a full development guide: • http://mesos.apache.org/documentation/latest/developer-guide/ • http://mesos.apache.org/documentation/latest/c++-style-guide/ Strategy pattern Introduction Architecture Implementation Design pros & … Reversing Mesos Conclusion October 2018 Apache Mesos - Morteza Zakeri - Software Architectures 16
- 17. 25 Design weak points • Backward incompatible • E.g. changed function signature for create(). • Unutilized abstractions • Duplicate abstractions • High cyclomatic complexity • master.cpp • Negative effects on: • Understandability, testability, and reliability Cyclomatic complexity spectrum Introduction Architecture Implementation Design pros & … Reversing Mesos Conclusion October 2018 Apache Mesos - Morteza Zakeri - Software Architectures Blue color intensity = 𝑀𝑎𝑥𝐶𝑦𝑐𝑙𝑜𝑚𝑎𝑡𝑖𝑐 𝐶𝑜𝑢𝑛𝑡𝐿𝑖𝑛𝑒 17
- 18. 25 Dive into source code • Code breakdown • About 2% inactive code • 9806 lines • Smell: unutilized abstractions Remove them! Introduction Architecture Implementation Design pros & … Reversing Mesos Conclusion October 2018 Apache Mesos - Morteza Zakeri - Software Architectures 18
- 19. 25 Dive into source code • Method breakdown • 86% public methods • Weak (leaky) encapsulation • Public methods in the same class sometimes call each other Make them protected! If possible :) Introduction Architecture Implementation Design pros & … Reversing Mesos Conclusion October 2018 Apache Mesos - Morteza Zakeri - Software Architectures 19
- 20. 25 Dive into source code • Largest methods • _accept() • 1147 lines Break them into sub methods! • Improve some aspects of SRP Introduction Architecture Implementation Design pros & … Reversing Mesos Conclusion October 2018 Apache Mesos - Morteza Zakeri - Software Architectures 20
- 21. 25 More on abstraction mesos::internal::master::Master::Http mesos::internal::checks::check::Http Smell: duplicate abstraction Introduction Architecture Implementation Design pros & … Reversing Mesos Conclusion October 2018 Apache Mesos - Morteza Zakeri - Software Architectures 21
- 22. 25 Conclusion • Mesos shares clusters efficiently among diverse frameworks • A framework ensemble / orchestration builder! • Master-slave communication model • Object-oriented design and programming • Have some design smells and many points to improve • What is the next? • Refactoring design smells! Introduction Architecture Implementation Design pros & … Reversing Mesos Conclusion October 2018 Apache Mesos - Morteza Zakeri - Software Architectures 22
- 23. 25 Conclusion • What do you think about this? Hardware OS MESOS Introduction Architecture Implementation Design pros & … Reversing Mesos Conclusion October 2018 Apache Mesos - Morteza Zakeri - Software Architectures 23
- 24. 25 References • [1] Hindman, B., Konwinski, A., Matei, Z., Ghodsi, A., D.Joseph, A., Katz, R., … Stoica, I. (2011). Mesos: A platform for fine-grained resource sharing in the data center. Proceedings of the …, 32. https://doi.org/10.1109/TIM.2009.2038002 • [2] Apache Mesos. (n.d.). Retrieved from http://mesos.apache.org/ ,2018 • [3] Understand. (n.d.). Retrieved from https://scitools.com/, 2017 • [4] G. Suryanarayana, G. Samarthyam, and T. Sharma, Refactoring for Software Design Smells: Managing Technical Debt, 1st ed. San Francisco, CA, USA: Morgan Kaufmann Publishers Inc., 2014. • [5] I. Sommerville, Software Engineering, 9st ed. 2010. October 2018 Apache Mesos - Morteza Zakeri - Software Architectures 24
- 26. 25 Fine-grained outline! • Introduction • What is Mesos? • Motivation to sharing a cluster • Problems and solutions • Architectural review • Design decisions and choices • Master-slave architecture • Mesos API • Implementation • C++ • Design pros and cons • Good points • Weak points • Reverse engineering Mesos • Dive into Mesos source code • Conclusion • References Backup slides October 2018 Apache Mesos - Morteza Zakeri - Software Architectures +1
- 27. 25 Cluster computing main applications • High Performance Computing (HPC) Large internet services • E.g. social networks, online shopping, … Data-Intensive scientific applications • E.g. physics, astronomy, molecular sciences, … M4 globular cluster Backup slides October 2018 Apache Mesos - Morteza Zakeri - Software Architectures +2
- 28. 25 Terminology • Cluster • A collection of distributed machines that collaborate to present a single integrated computing resource to the user. • Node • Each computing element (single machine itself) within cluster. • Framework • A software system that manages and executes one or more jobs on a cluster. Backup slides October 2018 Apache Mesos - Morteza Zakeri - Software Architectures +3
- 29. 25 Terminology • Job • A unit of work run on the cluster nodes. • Slot • Nodes are subdivided into “slots”. • (In some framework such as Hadoop and Dryad) • Task • Jobs are composed of short tasks that are matched to slots. • (In some framework such as Hadoop and Dryad) Backup slides October 2018 Apache Mesos - Morteza Zakeri - Software Architectures +4
- 30. 25 Terminology • Elastic framework • Can scale its resources up and down during execution tasks (dynamically). • Such as Hadoop and Dryad. • Rigid framework • Can start running its jobs only after it has acquired a fixed quantity of resources. • Can not scale (up/down) dynamically. • Such as MPI. Backup slides October 2018 Apache Mesos - Morteza Zakeri - Software Architectures +5
- 31. 25 Goals to Design Mesos • High utilization • Efficient data sharing • Support diverse frameworks (current and future) • Scalability to 10,000’s of nodes • Reliability in face of failures (robust to failures) Backup slides October 2018 Apache Mesos - Morteza Zakeri - Software Architectures +6
- 32. 25 Background • Diverse array of cluster computing frameworks: • Hadoop/ Map-Reduce (multi-terabyte data-sets) • Dryad (machine learning) • Pregel (graph computation) • MPI • … Dryad Pregel Backup slides October 2018 Apache Mesos - Morteza Zakeri - Software Architectures +7
- 33. 25 Problem • Rapid innovation in cluster computing frameworks. There is no single framework that optimal for all applications. Clusters typically run single framework at a time. Backup slides October 2018 Apache Mesos - Morteza Zakeri - Software Architectures +8
- 34. 25 Problem • Hadoop data warehouse at Facebook: • Include 2000-node Hadoop cluster, • Uses a fair scheduler for Hadoop, • Takes advantage of the fine-grained nature of the workload, • Allocate resources at the level of tasks and to optimize data locality. ↓ • This cluster can only run Hadoop jobs! • Can it run an MPI program efficiently? Backup slides October 2018 Apache Mesos - Morteza Zakeri - Software Architectures +9
- 35. 25 Better solution: Mesos! Hadoop Pregel MPI Shared cluster Today: static partitioning Mesos: dynamic sharing Backup slides October 2018 Apache Mesos - Morteza Zakeri - Software Architectures +10
- 36. 25 Resource Offer: Intuition Backup slides October 2018 Apache Mesos - Morteza Zakeri - Software Architectures +11
- 37. 25 Isolation • Isolate resources using OS container technologies • Linux Containers (LXC) • Solaris Projects • Limit the CPU, memory, network bandwidth, and (in new Linux kernels) I/O usage of a process tree. • Not perfect, but much better than no isolation! Backup slides October 2018 Apache Mesos - Morteza Zakeri - Software Architectures +12
- 38. 25 Fault Tolerance • Master’s only state is the list of active slaves, active frameworks, and running tasks. • Run multiple masters in a hot-standby configuration • Using ZooKeeper. • To deal with scheduler failures, Mesos allows a framework to register multiple schedulers. • when one fails, another one is notified by the Mesos master to take over. Backup slides October 2018 Apache Mesos - Morteza Zakeri - Software Architectures +13
- 39. 25 More on abstraction routing::filter::icmp::Classifier routing::filter::ip::Classifier Smell: duplicate abstraction Backup slides October 2018 Apache Mesos - Morteza Zakeri - Software Architectures +14
- 40. 25 The End • See my previous presentation about Apache Mesos: • Winter 2017 • Cluster, grid and cloud (CGC) computing course • https://www.slideshare.net/MortezaZakeri/introduction-to- apache-mesos-73001184 • Do you have any questions? • Contact me: • m - z a k e r i @ l i v e . c o m October 2018 Apache Mesos - Morteza Zakeri - Software Architectures +15
Editor's Notes
- #2: Apache MESOS: Architecture, Design and Code Review By: Morteza Zakeri Software Architectures Course – Fall 2018 Instructor: Dr. Mehrdad Ashtiani School of Computer Engineering Iran University of Science and Technology Final slides
- #3: Outline Agenda Page numbering forget!
- #4: I love to say that Apache Mesos is a framework ensemble / orchestration builder! What do you think???
- #5: HPC: High performance computing
- #8: Two Key Element: Fine-grained sharing Allocation at the level of tasks within a job Improves utilization, latency, and data locality Resource offer Simple, scalable application-controlled scheduling mechanism
- #9: One approach would be for Mesos to implement a centralized scheduler that takes as input framework requirements, resource availability, and organizational policies, and computes a global schedule for all tasks. Instead, Mesos takes a different approach: delegating control over scheduling to the frameworks. This is accomplished through a new abstraction, called a resource offer, which encapsulates a bundle of resources that a framework can allocate on a cluster node to run tasks.
- #14: The “callback” columns list functions that frameworks must implement, while “actions” are operations that they can invoke.
- #15: Today version versus early version? Where are we?!!
- #17: PHAME: Principle of hierarchy, abstraction, modularization and encapsulation. What about hierarchy? I did not find anything
- #21: SRP: Single responsibility principle
- #23: Apache Mesos is a framework ensemble / orchestration builder
- #24: Mesos come from Mesosphere, is it true Morteza?
- #25: Order by most using…
- #26: Thank you for your attention! Apache MESOS: Architecture, Design and Code Review By: Morteza Zakeri See backup slides section for more information. Last edit: 1397 07 30 – Monday – presentation day; after presentation. Edit 2: 1397 07 27 Edit 1: 1397 07 24
- #29: Cluster Computing Taxonomy (Glossary):
- #30: Cluster Computing Taxonomy (Glossary): Job A unit of work run on the cluster nodes. Slot Nodes are subdivided into “slots” (In Hadoop and Dryad) Task Jobs are composed of short tasks that are matched to slots.
- #31: We characterize workloads along two dimensions: elasticity and task duration distribution. An elastic framework, such as Hadoop and Dryad, can scale its resources up and down, i.e., it can start using nodes as soon as it acquires them and release them as soon its task finish. In contrast, a rigid framework, such as MPI, can start running its jobs only after it has acquired a fixed quantity of resources, and cannot scale up dynamically to take advantage of new resources or scale down without a large impact on performance.