Federated mesos clusters for global data center designs

HUAWEI TECHNOLOGIES CO. LTD.
www.huawei.com
Federated Mesos Clusters for Global Data Centers Designs
Krishna M Kumar, Lead Architect, Huawei Cloud

HUAWEI TECHNOLOGIES CO. LTD. 2
 Why Federation?
 Multi-Master Federation Approach
 Demo
Contents

What is Federation in General?
“A federation is a group of computing or network providers agreeing upon standards of operation in a collective fashion.” - wiki
Regional Authority: Autonomously working body.
Federal Layer: Helps the regional Authorities co-operate with each other.
So Cloud Federation is the union of multiple co-operating data Centers across the geography solving a common purpose.
Federal
Layer
Regional
Authority
Regional
Authority
Regional
Authority
Regional
Authority

Federating with Different Service Providers

Federating Datacenters within a Service Provider

Why Federation?
 High Availability.
 No Vendor Lock-in.
 Cloud bursting(accommodating spikes in demand).
 Load balancing across geographies.
 Application Upgrade/Migrate
 Policy Based Deployment.
 Economic benefits among providers.

Cloud Federation
Ubernetes (Google)
Ubernetes

Cloud Federation
Nomad (Hashicorps)
Nomads
Gossip
Gossip
Similar to Google’s Borg

Docker Swarm (Cluster Federation)
Cloud Federation

What about Federation for Mesos Clusters?

Some Mesos federation designs considered
in our research lab but dropped………
 Design 1 : Nested/Proxy approach
 Design 2 : Multi-Zone Mesos Super Cluster
 Design 3 : Mesos Global Manager (MGM)

Preferred Model: Multi-Master Federation Approach

Why Multi-Master?
Its really hard to control Super-Hero’s if you are not one. Ask this man!!!
Phew!!!
Big Day…

Benefits of Multi-Master
Each Data Center is a Super Hero, that will co-operate with each other.
• No single point of failure.
• DC co-operate with each other using gossip protocol.
• The frameworks gets fast feedbacks because it is connected to all the masters directly. The framework to be federated.
• Centralized data store layer.
• A simple policy Engine to demonstrate cloud bursting.

How it works?

Data Center 3
Data Center 1
Data Center 4Data Center 2
Hashicorp’s Consul will store all the
Policy information
Each Mesos Master is accompanied
by a ‘Gossiper’. Who will be the
representative of this Mesos run
Datacenter in the federation.
‘Gossipers’ talks to each other in the
federation and understand the
current policy
Gossiper
Gossipers negotiate with each other
and informs their respective master
what framework deserves the
offers.
Framework
Gossiper
Gossiper
Gossiper
ConsulMaster Master
Master
Master
Broad Overview

Data Center 3
Data Center 1
Data Center 4Data Center 2
Gossiper
Gossiper
Gossiper
Gossiper
Consul
Consul
Consul
Consul
Gossipers talk to each other using
hashicorp’s Member List library
Hashicorp’s Consul uses the same
MemberList Library
Overview of
Consul and
Gossiper
Interaction

Federated Master
FedAlloc: An Allocation module inherited from the default DRF module of Master
FedComm: A Mesos module of type Anonymous to which gossiper will talk to.
FedAlloc FedComm
Master
Gossiper
Allocation Module Anonymous Module

Internals of Federated Master
FedAlloc FedComm
Mesos Master
(Write only)(Read only)
Plug-in Plug-in
(Conditional
Wait)
F. Id
Suppress by FW
Suppress by
Federation
1122001 True True
1122007 True False
1122005 False True
1122004 False False
Gossiper

Internals of Federated Master (Cont.)
F. Id Suppress by
FW
Suppress by
Federation
1122001 True True
1122007 True False
1122005 False True
1122004 False False
FedAlloc FedComm
Mesos Master
(Write only)(Read only)
Plug-in Plug-in
(mutex)
FedComm (TCP read on Gossiper)
Lock Table
Write
Unlock Table
Signal Condition
FedAlloc (Conditional Variable)
Lock Table
Read
Call suppress( )/revive( )
unlock
Fedcomm automatically gets invoked
once the condition variable is set.
Gossiper

Gossiper
Anon Client: This instructs the master when to start and when to stop sending the Offers.
MasterInfo: This module periodically performs http GET on its respective Mesos master to update its statistical information
HTTP: Http Server that exposes some REST API’s
Member List (ML): Module that actually implements gossip layer.
Consul Lib: Library to talk to Consul and Replicate to other DC’s. Also implements a watch if there is any update on the policy.
Policy Engine: Read from Consul and interprets two policies:
1. Max Threshold
2. Next Max DC
HTTPMaster InfoAnon Client
Policy Engine
Consul Lib
ML
Gossiper

Master-Gossiper Interaction
Consul
Data Center 2
Data Center 3
Data Center 4
Data Center 5
FedAlloc FedComm
Master
HTTPMaster InfoAnon Client
Policy Engine
Consul Lib
ML
Gossiper

Framework
Protocol
M1: Mesos Master managing our DC1
Sample Policy: If we run out of resource in our
DC burst into Next Cloud
Register to Master 1;
M2
Register to Master 2
Offer 1
Launch Task 1
OOR
Offer 2
Launch Task 2
Offer 3
Launch Task 3
M3
Protocol
Register to Master 3
OOR
OOROOR
M1
Sequence Diagram

Gossiper - Exchange Framework
Broadcast{
Framework 11
Framework 7
Framework 5
}
Broadcast{
Framework 1
Framework 7
Framework 10
}
Broadcast{
Framework 8
Framework 7
Framework 4
}
Broadcast{
Framework 8
Framework 7
Framework 4
}
Gossiper 4
Gossiper 3
Gossiper 2
Gossiper 1

Gossiper - Exchange Resource Information
Gossiper 4
Gossiper 3
Gossiper 2
Gossiper 1
Broadcast{
CPU: 4
RAM: 16GB
Disk: 2TB
}
Broadcast{
CPU: 4
RAM: 8GB
Disk: 80GB
}
Broadcast{
CPU: 2
RAM: 4GB
Disk: 1TB
}
Broadcast{
CPU: 8
RAM: 4GB
Disk: 1.2TB
}

Gossiper - Exchange Out Of Resource
Gossiper 4
Gossiper 2
Gossiper 1
Out of Resource(OOR)
Gossiper 3

Minimal Policy Engine Implemented for this Experiment
• We needed a minimal Policy Engine to demonstrate cloud-busting scenario
• This Policy Engine is embedded as a part of Gossiper and can interpret only two simple rules
• The content of the Policy Engine in an array of Policy objects.
• Each Policy object has set of rules which needs to be applied.
• We use Hashicorp’s Consul to store Policy which is replicated across datacenter to avoid single point failure.
• Any update in the policy in one DC is instantly propagated to others. Gossiper watches Consul KeyStore and keeps the
latest copy of the policy.
{
"Name": "Policy_One",
"Rules": [{
"Name": "MinMax",
"Priority": 1,
"Scope": "",
"Content": {
"MinOrMax": "MAX"
}
}, {
"Name": "Threshold",
"Priority": 4,
"Scope": "",
"Content": {
"ResourceLimit": 90
}
}]
}
Simple Policy with two Rules
Rule 1:
• If Cloud busting which DC to choose ?
• One with Max Resources or Min Resources?
Rule 2:
• When should you perform Cloud busting?
• At what Resource Percentage?

Demo

When All Federated Data Centre’s are up and running

Map Legends for User Interface

When one of the Federated Data Centre’s goes Out of Resource

When all the Federated Data Centre’s are down

Resource Information of Selected Data Center

Resource Utilization of all the Data Center’s

Challenges / Future Work Planned
 Policy Engine with enhanced load balancing/Affinity
 Optimize the Gossip protocol for data consistency across clusters.
 Network throughput/Latency
 Service Discovery (i.e. DNS, etc.)
 Consolidated Monitoring, health, alerts, etc.
 Security & compliance in the Federation
 Work with the Mesos community for further refinement……….

Thank you
www.huawei.com
Copyright©2016 Huawei Technologies Co. Ltd. All Rights Reserved.
The information in this document may contain predictive statements including, without limitation, statements regarding the future financial and
operating results, future product portfolio, new technology, etc. There are a number of factors that could cause actual results and developments to
differ materially from those expressed or implied in the predictive statements. Therefore, such information is provided for reference purpose only and
constitutes neither an offer nor an acceptance. Huawei may change the information at any time without notice.

Federated mesos clusters for global data center designs

More Related Content

What's hot (20)

Viewers also liked (16)

Similar to Federated mesos clusters for global data center designs (20)

More from Krishna-Kumar (20)

Recently uploaded (20)

Federated mesos clusters for global data center designs