Next Generation Scheduling for YARN and K8s: For Hybrid Cloud/On-prem Environment to run Mixed Workloads

© Cloudera, Inc. All rights reserved.
NEXT GENERATION SCHEDULING for YARN & K8s:
For Hybrid Cloud/On-prem Environment to run Mixed Workloads
Sunil Govindan & Weiwei Yang

© Cloudera, Inc. All rights reserved. 2
SPEAKER
SUNIL GOVINDAN
Engineering Manager @Cloudera
@sunilgovind
Apache Hadoop PMC & Committer
WEIWEI YANG
Staff Software Engineer @Cloudera
@abvclouds
Apache Hadoop PMC & Committer

AGENDA
Journey through Big Data Experience
Where are we today on scheduling experience ?
Motivation to improve scheduling
Introduction to YuniKorn
Demo
Future & Open Source Story

BATCH
WORKLOADS
DEEP LEARNING APPS
CUSTOMER JOURNEY - BIG DATA ECOSYSTEM
BIG DATA ECOSYSTEM - TODAY
PUBLIC CLOUD
STORAGE
100100100
101001000
010010101
STORAGECOMPUTE (on-prem/on-cloud)
HIVE on LLAP
SERVICES

BIG DATA ECOSYSTEM
NEW Trends
Big Data &
Containerization
• Portability for apps
• Resource Isolation
• Dependency
Management
Moving to Cloud for
better elasticity
• Budget Control
• Infinite Resources
• Cloud Burst
Decoupled Compute &
Storage
• Low Cost
• Hybrid in nature
• Agility
• Mixed Workloads

AGENDA
Demo

WHERE ARE WE TODAY ?
RESOURCE ORCHESTRATOR PERSPECTIVE - STRENGTHS
APACHE YARN
Big Data Ecosystem
KUBERNETES
Cloud Ecosystem
CLOUD NATIVE
Public Cloud
Big Data—Optimized to run Big
Data workloads.
Batch Workloads—High Throughput
scheduling for batch workloads.
SLA—Better SLA for Big Data
workloads.
Multi tenant—Quota Management.
Services—Optimized for
containerized microservices.
Networking—Strong Network
management support.
Cloud Aware—Better tuned for
cloud use cases.
Storage—Persistent volumes
Cost—Budget Centric
∞—Infinite resource for infinite $$.

QUICK RECAP TO THE NEW TRENDS
Moving to Cloud
Big Data &
Containerization
Mixed Workloads

CHALLENGES - APACHE YARN
APACHE YARN WORLD
• Challenges in managing THREE YARN schedulers for different use cases
• Should Deploy & Manage containerized microservices in the same Hadoop Cluster
• Need strong networking and persistent volume support
• Need more powerful Auto scaling and budget control

CHALLENGES - KUBERNETES
KUBERNETES WORLD
• Challenges in running Big Data workloads along with microservices together
• Need much better quota management and better SLA
• No first class Application management concept for Big Data workloads

“WHERE IS MY SILVER BULLET
TO SOLVE ALL THESE
CHALLENGES? ”

“SORRY, THERE ISN’T ANY”

SO HOW TO SOLVE THIS ?
Assessment
Native Big Data Apps — Moving
batch workloads (MR, TEZ .. ) from
YARN to Kubernetes looking into
High throughput, Low Latency and
With notion of job etc is costly.
Adaptability — Optimized for
services and running batch
workloads exposes hard-to-bridge
gaps such as run few workloads
with or without docker
Services on YARN — Services and
web farms can run on YARN
however it is not as feature-rich as
Kubernetes.
Multiple Schedulers (YARN) —
Different schedulers are focussed
on specific use cases and not very
easy to drive continuous feature
enhancements .
One cannot replace another —
Neither Kubernetes can replace
YARN or vice versa in the near
future considering some of the
fundamental architecture
differences.
EXPENSIVE
Higher cost to achieve the goal

AGENDA
Demo

“WE NEED TO IMPROVE ON
WHAT THEY ARE NOT GOOD AT
TODAY ”

© Cloudera, Inc. All rights reserved. 16© Cloudera, Inc. All rights reserved.
APACHE YARN WORLD
Strengthening Dynamic
Environments
• Improve YARN to work
well for cloud (public &
private)
• Focus on Autoscaling,
Smarter Scheduling etc.
Refer: YARN-9548
Improving capabilities for
persistent volumes
• Added CSI (Container
Storage Interface) support
• Enhancing CSI
implementation to expand
and support storages such
S3, Ozone etc as mounted
volume to YARN
containers
Native Service
enhancements
• Improving native
services support in
YARN
• Micro Service upgrades
What’s happening now in YARN today ?

KUBERNETES WORLD
Demand for Better support of
batch workloads
• Early efforts to support batch
scheduling is in progress by
K8s community.
Efforts on running Spark
on K8s
• Spark and Kubernetes
community is working
towards Spark on K8s
deployments.
• Gaps in running Spark
such dynamic resource
allocation, security etc is
still open.
CDP and Kubernetes
Cloudera
Few CDP microservices will
be running on Kubernetes
What’s happening now in KUBERNETES today ?

MOTIVATION & EFFORTS TO IMPROVE
Not Optimized — to balance use-
cases like batch workloads to the
needs like running web farms or
services with respect to scheduling
challenges.
Poor Resource Utilization — Not able
to effectively utilize complete resources
in cluster for services and workloads.
FRAGMENTED
Multiple YARN & Kubernetes schedulers are
UNIFIED RESOURCE SCHEDULER AND
APPLICATION MANAGEMENT
What we need is an effort to improve both YARN and
Kubernetes scheduling worlds.
Multiple schedulers power YARN & Kubernetes for different use cases

AGENDA
Demo

PROPOSAL
YuniKorn (/ˈyo͞ onəˌkôrn/, Y for YARN, K for K8s, uni- for
Unified)
• A common resource scheduler
• Platform independent
• Enhanced scheduling capabilities

WHAT YuniKorn IS (IS NOT) ?
YuniKorn is
• A better scheduler for the K8s world, for services and batch workloads
• A unified scheduler for the YARN world (FiFo, Fair and Capacity Scheduler)
• Providing unified resource scheduling experience across the YARN and K8s (and beyond)
• Suitable for both finite resource (datacenter) and infinite resources/dollars (cloud) worlds
Is NOT
• A system to port YARN applications to run on K8s w/o modification, or vice versa

YuniKorn - A UNIFIED RESOURCE SCHEDULER
Capacity Planning
Capacity Planning
Divide cluster resources into resource pools
(queues), define capacity range based on needs.
Enforce resource quotas and limits.
Resource scheduling
Resource fairness, preemption, high-throughput,
multi-tenant, placement, etc.
Application Management
A central place to monitor application states
Resource Monitoring
A unified view of cluster resources, a dashboard to
easy track resource usage by queue, user or
organization.
Resource
Scheduling
Application
Management
Resource
Monitoring
...
Explore the feature set

CAPACITY PLANNING
Hierarchy of queues
Queues can be organized in honor of user groups or
organizations, with multiple levels.
Elastic Capacity
Each queue has its min-max capacity, usage is elastic
within this range for multiple users.
Resource Quotas
Resource cap for queues or users. Limited amount of
resources, number of applications etc.
Partition
A set of instances (nodes) that are physically isolated

RESOURCE SCHEDULING
Resource Fairness
Queue/User/App level fairness
ensures each entity gets its own fair
share of resources.
Priorities
Queue priority + App priority
Preemption
Queue demands for more resources
have the chance to preempt resources
from other queues for high priority
apps.
Placement Constraints
Affinity/anti-affinity, node constraints
etc
Services
Low Latency
Long Running
Batch
High-throughput
Short-lived
YuniKorn

RESOURCE MONITORING
Common dashboard to monitor resources
Hierarchy of
queues
Cluster
resources are
divided into
hierarchy of
queues, all
queue state is
visible
Common View
A common view
of resources,
cross platform.
Resource Centric
Focus on resources,
total/available/used,
and all
Resource
Dashboar
d

APPLICATION MANAGEMENT
Track applications in a consistent fashion
Application originated
GUI to manage
workloads (instead of
individual pods in
K8s).
Entire application
lifecycle is visible and
trackable.

ARCHITECTURE
YuniKorn Core
Scheduler
Shim
Master
Scheduler Interface
Api-server etcd
kubelets
Resource
Manager
Node Managers
GPRC/API
Resource requests, new application,
node updates
GPRC/API
Container allocation, preemption
Master Node
Slave nodes
MR
Spark
Flink
Tez
MySQL
Spark
Web
Server
Kafka
Client API
Allocate, release container

YuniKorn SCHEDULER vs. OTHERS
Disclosure: this table is summarized based on speakers’ analysis
Scheduler
Capabilities
Resource Sharing
Resource
Fairness Preemption Throughput Gang Scheduling
Hierarchy
queues
Queue
prioritoy
Queue elastic
capacity
Cross queue
fairness
User level
fairness
App level
fairness
Basic
preemption With fairness With priority
Kube-default x x x x x x v x v 100+ allocs/s x
Kube-batch x x x x x v v x v ? v
YARN CS/FS v v v v v v v v v 4k+ allocs/s x
YuniKorn v x (wip) v v v v v v x (wip) ? x (wip)
Key capabilities of a resource scheduler from our perspective

AGENDA
Demo

DEMO
K8shim API Server
YuniKorn Core
etcd
yunikorn-pod
Master
Kubelet
● YuniKorn is deployed in ONE pod
● YuniKorn core only talks to K8shim via scheduler-interface, K8shim talks to api-server
Let’s explore what YuniKorn can do for KUBERNETES !!!
HERE WE GO...
YuniKorn UI

AGENDA
Demo

YARN cluster
TAKE AWAY
BEFORE AFTER
Scenario 1
Scenario 2
Scenario 3
K8s cluster on cloud/prem
Existing K8s
schedulers
K8s cluster on cloud/prem
YuniKorn
scheduler
K8s cluster
Existing K8s
schedulers
YARN cluster
Capacity
Scheduler
Fair
Scheduler
K8s cluster
YuniKorn Scheduler
YARN cluster
Capacity
Scheduler
Fair
Scheduler
YARN cluster
YuniKorn
scheduler

OPEN SOURCE
Yes, We are going to Open Source soon.
STAY TUNED...

ACKNOWLEDGMENTS
A big shout out to the folks who helped to design, develop and make this project
possible.
❏ Vinod Kumar Vavilapalli
❏ Wangda Tan
❏ Wilfred Spiegelenburg
❏ Akhil PB
❏ Suma Shivaprasad
❏ and many others...

Next Generation Scheduling for YARN and K8s: For Hybrid Cloud/On-prem Environment to run Mixed Workloads

More Related Content

What's hot (20)

Similar to Next Generation Scheduling for YARN and K8s: For Hybrid Cloud/On-prem Environment to run Mixed Workloads (20)

More from DataWorks Summit (20)

Recently uploaded (20)

Next Generation Scheduling for YARN and K8s: For Hybrid Cloud/On-prem Environment to run Mixed Workloads