EMC World 2016 - Introduction to Mesos and Mesosphere

2© Copyright 2016 EMC Corporation. All rights reserved. 2© Copyright 2016 EMC Corporation. All rights reserved.
INTRODUCTION TO
MESOS AND MESOSPHERE
MAY 2016

• Somik Behera, DCOS Product Manager - Mesosphere
– Twitter: @strikesme
– Blog: http://strikesme.com
• David vonThenen, Developer Advocate - EMC {code}
– Twitter: @dvonthenen
– Blog: http://dvonthenen.com
PRESENTERS

• What is Mesos?
• What will Mesos do for me?
• Who is Mesosphere?
• How EMC{code} fits with Mesos
• Demo
• Q&A
AGENDA

WHAT IS MESOS?

• From a resource perspective, it’s a cluster manager:
– Pools servers resources to be centrally managed as a single
unit
• From an application perspective, it’s a scheduler:
– Dispatches workloads to consume pooled resources
• Often described as a Data Center Operating System
(DCOS)
MESOS IS…

• Master
– Manages Mesos Agents
– Enables fine-grained sharing of resources (CPU, RAM, …)
across frameworks in the form of resource offers.
• Zookeeper
– Enables highly reliable distributed coordination (cluster
state)
• Agent (aka Slave)
– The entity that will be running the task
MESOS COMPONENTS

MESOS ARCHITECTURE

EFFICIENT RESOURCE UTILIZATION

WHO DOESN’T LIKE EFFICIENCY?

• Generic
– Anything that can be run on a standard Linux shell
– Provides a mechanism for deploying applications packages
– Supports Docker workloads
• Framework
– A plug-in to Mesos that allows you to implement a sub-
scheduler
– specialize a workload by providing custom placement and
resource allocation
APPLICATION SUPPORT

GENERIC APPLICATIONS

MESOS FRAMEWORKS

BRINGING EVERYTHING TOGETHER

WHAT WILL MESOS DO FOR ME?

• Bare metal x86 hardware
• VMs (your favorite hypervisor)
• Public, Private, Hybrid Cloud
MULTIPLE DEPLOYMENT VECTORS

MESOS WILL RUN ON…

• Heterogeneous nodes (different CPU models,
memory footprints, etc)
• Supports elastic scaling
– Nodes can be added for capacity
– Nodes can be removed for maintenance
CLUSTER NODES

• Has been in production use for 3+ years
• 10k+ node scale
• Recognized by users and the community
• Tasks dispatching scale linearly over # nodes (2010)
STABILITY AND PERFORMANCE

POWERED BY MESOS

• Marathon
– manages long-running applications.
Provides both a UI and REST API
• Chronos
– is a replacement for cron to schedule batch
jobs and other workloads.
• Mesos-DNS
– enables DNS based service discovery
• Etc, etc, etc…
RICH APPLICATIONS AND ECOSYSTEM

• Runs a wide variety of workloads
• On commodity hardware
• Efficient utilization of resources
• High Availability / Fault Tolerant
• UI and API management
WHAT WILL MESOS DO FOR ME?

OR TRY TO BUILD ONE FROM SCRATCH

WHO IS MESOSPHERE?

MESOSPHERE: BORN OF THE HYPERSCALE ERA
Mesosphere is well funded
$50M by Tier 1 investors:
Andreessen Horowitz & Khosla Ventures
2009 2013 2014 2015
First Mesosphere DCOS
lighthouse customers
Tens of thousands of containers launched
Expanded Operations
San Francisco (2013)
Hamburg (2014)
New York (2015)
Berlin (2015)
Mesosphere Founded
Key engineering leaders from Twitter,
Airbnb - companies behind open-
source tech
Apache Mesos built at UC Berkeley
● Core technology from AMPlab
● Corporate large-scale production deployments
begin (e.g., Apple, Twitter, Salesforce)
Growing Ecosystem &
DCOS General Availability
Now a breeze to install modern app
services (e.g., Hadoop, Spark,
Cassandra)

DELIVERING OUTCOMES FOR ENTERPRISES
DEVELOPER
AGILITY
HYPERSCALE
OPERATIONS
DATA
AGILITY
Highly Efficient & Scalable IT
Distributed Framework
Pooling
Container Orchestration
Cont. Integration/Delivery
Developer Cloud
Self-Service Container
Runtime
Big Data
Internet Of Things
Reactive applications

THE DATACENTER IS THE NEW SERVER
PHYSICAL (x86) VIRTUAL HYPERSCALEMAINFRAME
SERVER
VIRTUAL
MACHINE
PARTITION
(LPAR)
UNIT OF
INTERACTION
● ERP, CRM,
PRODUCTIVITY, MAIL &
WEB SERVER
● LINUX, WINDOWS
● DATA /
TRANSACTION
PROCESSING
● UNIX, IBM OS/360
DEFINITIVE
APPS AND OS
● ERP, CRM,
PRODUCTIVITY, MAIL &
WEB SERVER
● HYPERVISOR + GUEST
OS
● BIG DATA, INTERNET
OF THINGS, MOBILE
APPS
● ???
???DATACENTER
NEW FORM FACTOR FOR
DEVELOPING AND
RUNNING APPS
● BIG DATA, INTERNET
OF THINGS, MOBILE
APPS
● DATACENTER
OPERATING SYSTEM
(DCOS)

MESOSPHERE DCOS ACCELERATES ENTERPRISES TO
A MODERN CLOUD STACK WITH THE DATACENTER FORM
Datacenter Operating System (DCOS)
Distributed Systems Kernel (Apache Mesos)
Big Data + Analytics EnginesMicroservices (in containers)
Streaming
Batch
Machine Learning
Analytics
Functions &
Logic
Search
Time Series
SQL / NoSQL
Databases
Modern Cloud App Components
Distributed systems kernel to
abstract resources
Killer “apps” ecosystem
Cont Integration/Cont Delivery
Framework “app” architecture
to consistently run on top of
kernel
User Interface (GUI & CLI)
Core system services
(e.g., distributed init, cron,
service discovery, package mgt &
installer, storage)
Any Infrastructure (Physical, Virtual, Cloud)

DCOS BRINGS HYPERSCALE TO ENTERPRISES
Desktop Apps (e.g., Safari,
Adobe Photoshop, Itunes)
GUI (Aqua)/CLI
Personal computer
- App store
- OpenGL
- Advanced UI gestures
BSD Unix
Mobile Apps (e.g., Spotify,
Evernote, WhatsApp)
Android GUI
Mobile devices
- Telephony Manager
- Battery management
- External storage support
Linux
Distributed systems
(e.g., Docker, Spark,
Hadoop, Cassandra)
GUI/CLI (DCOS CLI)
Full Datacenter / Cloud
- Container orchestration
- Distributed batch jobs
- Persistent storage mgmt
Apache MesosKernel
Applications
Form Factor
User Interface
OS Services
(highlights)
Computing for the masses Mobile apps for the
masses
Datacenter-scale apps for
mainstream enterprises
Benefits

HOW EMC {CODE} FITS WITH MESOS

• Containers are ephemeral therefore stateless
• No state? Means boring Application
• Container vendors started introducing state by way
of local disks
• Something better?
• EMC {code} brings storage expertise to the table
HOW EMC {CODE} FITS WITH MESOS?

• Enable external storage
• Enhance frameworks to embrace and consume
external storage
• Run EMC storage platforms as scale out data
services
EMC {CODE} GOALS WITH MESOS

• Persistent
– Containers can come and go,
but continue where they left off
– Application Data, Logs, Audit
Trail, etc
• External
– Storage lives outside the Mesos
Agent node
– High Availability
– Tolerates node failure
PERSISTENT EXTERNAL STORAGE

• REX-Ray
– Provides a vendor agnostic storage orchestration engine
– AWS, GCE, ScaleIO, XtremIO, many many more
– Github: https://github.com/emccode/rexray
• mesos-module-dvdi
– Provides a hook to Mesos agent nodes to provision, attach,
and detach external storage
– Github: https://github.com/emccode/mesos-module-dvdi
HOW DO WE ACHIEVE THIS?

• We can run “real world” applications
• Persistence – DBs
• Highly Available, Fault Tolerant
• Audit trail – Transaction Logs
• Not worry about the “2am call”
STATELESS TO STATEFUL

THE 2AM CALL…

DEMO

DEMO 1: TWEETER - ARCHITECTURE
Datacenter Operating System (DCOS)
Containerized
Microservice
Kafka Distributed
Messaging
system
Cassandra
Distributed
Datastore
Spark Data
Analytics JobsDistributed
Software Load
Balancers
“Tweeter” Service

• Running on Amazon EC2
– Mesos
– Marathon
– Mesos DNS
• Production HA Cluster
– 3 x Master + Zookeeper Nodes (t2.micros)
– 4 x Slave Nodes with Docker Support (t2.medium)
DEMO 2: CONFIGURATION

• Production Website (MVC)
– Web UI (Golang)
• Github: https://github.com/dvonthenen/goprojects/tree/master/src/restclient
– REST API (Golang)
• Github: https://github.com/dvonthenen/goprojects/tree/master/src/restserver
– PostgreSQL (Docker Container) backed by Persistent
External Storage
• Using REX-Ray
• Using mesos-module-dvdi
• Simple Web App - User Account Management
DEMO 2: PRODUCTION WEBSITE

• Deploy Model, View, Controller using REST API
– Observe creation of each component
– Observe PostgreSQL DB attaching to EBS volumes
– Using browser, log into Account Management UI and
add/delete records
– Observe data from Web UI being added to PostgreSQL table
• Simulate Failure
– Kill EC2 Instance
– Watch task recreation and reattach to EBS volumes
DEMO 2: PRODUCTION WEBSITE

DEMO 2: INITIAL DEPLOY

DEMO 2: SCALE IT UP

Data Persistence in the New Container World
Wednesday 3PM
Joshua Bernstein
VP of Technology for ETD
Tobi Knaup
CEO & Co-Founder of Mesosphere
Guru Session

@EMCcode
@strikesme
@dvonthenen
emccode.com
community.emccode.com
Come visit us at
Booth #1044
or in the vLab
Questions?

EMC World 2016 - Introduction to Mesos and Mesosphere

EMC World 2016 - Introduction to Mesos and Mesosphere

More Related Content

What's hot (18)

Viewers also liked (20)

Similar to EMC World 2016 - Introduction to Mesos and Mesosphere (20)

More from David vonThenen (20)

Recently uploaded (20)

EMC World 2016 - Introduction to Mesos and Mesosphere

Editor's Notes