Mesosphere and the Enterprise: Run Your Applications on Apache Mesos - Steve Wong - Dell EMC World 2017
Download as PPT, PDF1 like2,553 views
The document discusses Mesosphere DC/OS, an infrastructure abstraction layer that simplifies the management of distributed technologies, facilitating application deployment and scheduling. It highlights the benefits of containerization, utilization efficiency, and the integration of tools like Marathon and Rex-Ray for optimal workload management and storage orchestration. Additionally, it outlines the history, performance metrics, and operational capabilities of using Mesos for container management in enterprise environments.
Mesosphere and the Enterprise: Run Your Applications on Apache Mesos - Steve Wong - Dell EMC World 2017
- 1. Mesosphere and the Enterprise: Run Your Applications on Apache Mesos Steve Wong Open Source Engineer @cantbewong {code} by Dell EMC
- 2. © Copyright 2017 Dell Inc.2 Open source at Dell EMC – Contribute to meaningful OSS projects – Create new thought leading OSS applications – Drive awareness of OSS opportunities with Dell EMC product teams – Participate in relevant community engagement projects – Act in the interest of building a community {code} by Dell EMC is a group of passionate open source engineers and advocates working to build a community around software-based infrastructure. Platinum SponsorPlatinum Sponsor
- 3. © Copyright 2017 Dell Inc.3 Why Containers? They make managing applications easier • Speed up software innovation • Deployments are simple • Creates portable, shareable and reusable units of code and requirements Code Code and requirements Container
- 4. © Copyright 2017 Dell Inc.4 What is Mesosphere DC/OS? From a resource perspective: It’s an infrastructure abstraction layer: – Aggregates systems into a single resource pool. From an application perspective: It simplifies management of distributed technologies – A scheduler (= dispatcher). – Dispatches workloads which consume the pooled resources.
- 5. © Copyright 2017 Dell Inc.5 History Started at UC Berkeley in 2009, became an Apache project in 2013 after nearly two years in incubation
- 6. © Copyright 2017 Dell Inc.6 Project Objective Deliver Google’s Borg to the rest or world See Wired: How Twitter rebuilt Google’s Borg March 2013
- 7. © Copyright 2017 Dell Inc.7 Two levels of Application Support Generic Run anything that can be launched in a standard Linux shell + Docker images Framework Allows you to implement a sub-scheduler – Extends Mesos to provide custom workload placement and resource allocation › For example, reserve 10% of the cluster’s resources to Hadoop jobs, framework manages prioritized dispatch
- 8. © Copyright 2017 Dell Inc.8 With Marathon & Metronome, all types of jobs are managed
- 9. © Copyright 2017 Dell Inc.9 Marathon Marathon acts as a service dispatcher You provide instructions and Marathon does its best to keep the service up on the cluster, providing automated scaling and high availability restarts
- 10. © Copyright 2017 Dell Inc.10 Metronome – a “chron” for the datacenter Specify: •The amount of CPU your job will consume. •The amount of memory your job will consume. •The disk space your job will consume. •The schedule for your job, in cron format. You can also set the time zone and starting deadline.
- 11. © Copyright 2017 Dell Inc.11 Overview of Enterprise DC/OS
- 12. © Copyright 2017 Dell Inc.12 Where can you run Mesos?
- 13. © Copyright 2017 Dell Inc.13 Any limitations on what I can use for cluster nodes? • They do not have to be identical • Processor model, speed or memory size variations are OK • They can concurrently run non-Mesos services • don’t have to give Mesos control over 100% of node’s CPU, memory or disk. • Elastic scaling is supported • nodes can come and go
- 14. © Copyright 2017 Dell Inc.14 Production Users & Customers Government Agencies
- 15. © Copyright 2017 Dell Inc.15 Efficiency and Utilization Typical Datacenter siloed, over-provisioned servers, low utilization DCOS Datacenter automated schedulers, workload multiplexing onto the same machines DCOS Multiplexing 30-40% utilization, up to 96% at some customers 4XIndustry Average 12-15% utilization
- 16. © Copyright 2017 Dell Inc.16 How big can this really go? Performance Characteristics
- 17. © Copyright 2017 Dell Inc.17 Performance metric at scale 50,000 containers deployed live on stage 1 minute 12 seconds
- 18. © Copyright 2017 Dell Inc.18 Architecture
- 19. © Copyright 2017 Dell Inc.19 How {code} by Dell EMC fits with Mesos
- 20. © Copyright 2017 Dell Inc.20 Back in 2012 container aps must be stateless?
- 21. © Copyright 2017 Dell Inc.21 The picture today
- 22. © Copyright 2017 Dell Inc.22 External Volume Mount Solution
- 23. © Copyright 2017 Dell Inc.23 External Volume Mount Addresses Availability Risk •Migrate container to another host – your storage is gone •Host goes down – your service goes down Scale Limitation •Need more storage than the host has? Sorry…
- 24. © Copyright 2017 Dell Inc.24 Components REX-Ray rexray.codedellemc.com
- 25. © Copyright 2017 Dell Inc.25 REX-Ray REX-Ray The leading container storage orchestration engine enabling persistence for cloud native workloads rexray.codedellemc.com Cloud Native Interoperability Open Source Enterprise Ready – High Availability – CLI Intuitiveness – Effortless Deployment – Architectural Choices Multi-Platform Storage Management – Storage agnostic (block/file/object)
- 26. © Copyright 2017 Dell Inc.29 Mesos Container Framework Integration • Marathon from Mesosphere • Apache Aurora (originally used by Twitter) • ElasticSeach
- 27. © Copyright 2017 Dell Inc.30 End-to-End Container Persistence Leadership for Mesos What does it do Enables external storage to be created / mounted / unmounted with each agent task Abstracts Docker Volume Drivers for Mesos Storage orchestration engine Cool factor • First Mesos Agent external storage module • Merged into Mesos 1.0 (2016) • Allows the use of any Docker volume driver with Mesos! • Vendor agnostic • DC/OS 1.7+ framework (2016) mesos-module-dvdi Docker Volume Driver Client CLI DVDCLI REX-Ray
- 28. © Copyright 2017 Dell Inc.31 ScaleIO Framework for Mesos • A Software-based Storage Framework using ScaleIO to turn commidy hardware and direct attached storage (DAS) into a globally accessible and scalable storage platform that can be deployed anywhere • Framework installs and configures ScaleIO on all Mesos Agent (compute) nodes. As more Agents are added to the cluster, ScaleIO is installed and adds storage resources to the cluster. • Deploy and Configure ScaleIO without knowing operational details • Centralized Monitoring of the storage platform • Persistent storage native to the container scheduling platform using REX-Ray • https://github.com/codedellemc/scaleio-framework
- 29. © Copyright 2017 Dell Inc.32 {code} + Mesos Mesos Module DVDI is officially merged into Mesos 1.0 and allows any Docker Volume Driver to be used with Mesos Frameworks REX-Ray is installed on every host and performs storage orchestration MARATHON ScaleIO-Framework installs ScaleIO software on all Mesos Agents SUPPORTED STORAGE PLATFORMS WITH REX-RAY
- 30. © Copyright 2017 Dell Inc.33 Demo! DC/OS / Marathon Mesos Database Deploy
- 31. © Copyright 2017 Dell Inc.34 codedellemc.com community.codedellemc.com @codeDellEMC blog.codedellemc.com {code} by Dell EMC is a group of passionate open source engineers and advocates working to build a community around software-based infrastructure. rexray.codedellemc.com github.com/codedellemc/labs Steve Wong @cantbewong github.com/cantbewong #codeopen HOL01 Use REX-Ray & ScaleIO w/ Docker, Mesos and Kubernetes
Editor's Notes
- #3: Add talking points to each point OSS projects: Docker, Mesos, Kubernetes, Cloud Foundry
- #4: Mesos is a system for managing containers, in production, at scale.This is a beginner session, and because we have limited time I am going to assume you are familiar with containers. If not I’m doing this one slide as a recap, but you better “buckle up” for the rest of this hour.
- #5: Two perspectives Often described as a Data Center Operating System (DC/OS)
- #22: When somebody trots out that old adage that only cattle should ride in the container ship.. This of this image Bull + Ship
- #23: External volume mount allows to have storage to be independent of the container using it.Storage has its own, potentially indefinite life cycle
- #25: Note RexRay is built into the DC/OS distribution
- #26: Kenny
- #31: Docker Volume Driver Isolator Module No remote storage capability using Mesos before September 2015 {code} was first to bring an abstracted persistent storage model Freedom to use the Mesos Containerizer with ANY docker volume driver (that means you don’t need Docker) Merged upstream into Mesos 1.0 to provide storage platform support for any vendor who has written a docker volume driver, including REX-Ray Every company that boasts storage integration with Mesos uses the {code} contributed module. DVDCLI DC/OS 1.7 Release ships with REX-Ray embedded DC/OS 1.8 Release ships with DVDI embedded for Mesos 1.0 Featured in the documentation - https://dcos.io/docs/1.8/usage/storage/external-storage/
- #35: First, a few things about the team that has made this possible. The Dell EMC {code} team is a team made up of open source software engineers and developer advocates, focused on making EMC a well-known name within the open source community. We will focus on one of their projects, REX-Ray, in this presentation.