Mesos: Cluster Management System
0 likes671 views
Mesos is an open source cluster management framework that provides efficient resource isolation and sharing across distributed applications or frameworks. It divides resources into CPU, memory, storage, and other compute resources and shares those resources dynamically and efficiently across applications. Mesos abstracts the underlying infrastructure to provide a unified API to applications while employing operating system-level virtualization through interfaces like Docker to maximize resource utilization. It works by having a Mesos master that negotiates resources among Mesos slaves to run applications or frameworks, which are made up of a scheduler to negotiate for resources and executors to run tasks. Common frameworks that run on Mesos include Spark, Hadoop and Docker containers.
Mesos: Cluster Management System
- 2. • to understand target problem domain. • to know the operating system for the Datacenter. • to gain an architectural overview of a Mesos cluster. • to learn how to run applications in a Mesos cluster. • to understand the Service Discovery & Load Balancing • to get an overview on Mesos Frameworks. http://mesos.apache.org Agenda Your Expectations
- 3. Mesos is the name of the cluster management framework and the Apache Project. Mesosphere (Mesosphere, Inc.) is the company which is commercializing the open source project Apache Mesos, and provides consulting services. http://mesos.apache.org Mesos Mesos vs Mesosphere
- 4. http://mesos.apache.org Problem Domain Static Partitioning
- 5. http://mesos.apache.org Problem Domain Static partitioning Dynamic resource sharing Resource Management
- 7. • Realignment of the partitions in the clusters means un- deploy/redeploy of the running software on the instances and requires error-prone and tedious manual effort. • It is hard to predict the workload and fluctuations of the load makes an efficient resource utilisation impossible. • Changes in infrastructure may cause changes in the applications so the coupling between Developers and Devops does remain strong. http://mesos.apache.org Problem Domain Problem Description
- 8. Datacenter as a Computer
- 9. • Process scheduling. • Fairly resource (on the same CPU, Memory, Disk, etc) sharing. • Package managers (aptitude, yum, etc.) • Unified API between apps and hardware. http://mesos.apache.org Datacenter OS DC as a Computer • Cluster-wide application scheduling. • Fairly resource sharing in a distributed environment. • Docker as package manager tool. • Unified API between apps and hardware. Operating Systems Mesos as a DC OS
- 10. • orchestrates the infrastructure so that it can be extended by adding new hardware or shrunk to downscale, dynamically. • abstracts the underlying infrastructure while providing a unified API to the applications. • employs the LXC (Linux Containers) interface to maximize resource utilisation via operating system level virtualization (like Docker or using Docker). http://mesos.apache.org Problem Domain Apache Mesos
- 11. http://mesos.apache.org Architecture Architecture Coordination over Zookeeper Mesos CLI Ports are ranged resources
- 12. http://mesos.apache.org Datacenter OS How does it work ? Mesos Framework = Scheduler + Executor
- 13. http://mesos.apache.org Datacenter OS How does it work ? Resource negotiation between components
- 14. • You can develop your own custom Frameworks for Mesos. • or you can deploy open source Mesos Frameworks like Spark, Hadoop, etc. • or, getting even better, you can deploy Docker containers. • Mesos can orchestrate Docker containers in a Mesos cluster, as well. http://mesos.apache.org Frameworks Applications on Mesos
- 15. http://mesos.apache.org Frameworks Mesos Frameworks OSS Frameworks as Mesos Frameworks
- 17. http://mesos.apache.org Marathon Marathon • Marathon is the PaaS on Apache Mesos. • Marathon provides REST API to automate your workflows. • Marathon UI allows Dev/ Ops to manage the frameworks by point-n-click.
- 18. http://mesos.apache.org Architecture Load Balancing thru HAProxy User Requests
- 19. http://mesos.apache.org Architecture Service Discovery User Requests Mesos DNS provides its clients a REST endpoint for Service Discovery
- 20. http://mesos.apache.org Hands-on Hands-on • Introduction: Mesos Dashboard • Introduction: Marathon UI and REST API. • Case Study: Elastic Jenkins on Mesos.