The Evolution of Big Data Frameworks
5 likes2,526 views
This document discusses the evolution of cluster computing and resource management. It describes how: 1) Early clusters were single-purpose and used technologies like MapReduce. General purpose cluster OSes like YARN emerged to allow multiple applications on a cluster. 2) YARN improved on Hadoop by decoupling the programming model from resource management, allowing more flexibility and better performance/availability. 3) REEF aims to further improve frameworks by factoring out common functionalities around communication, configuration, and fault tolerance.
The Evolution of Big Data Frameworks
- 1. From the dive bars of silicon valley to the World Tour Carlo Curino
- 2. PhD + PostDoc in databases “we did all of this 30 years ago!” Yahoo! Research “webscale, webscale, webscale!” Microsoft – CISL “enterprise + cloud + search engine + big-data” My perspective
- 3. Cluster as an Embedded systems (map-reduce) single-purpose clusters General purpose cluster OS (YARN, Mesos, Omega, Corona) standardizing access to computational resources Real-time OS for the cluster !? (Rayon) predictable resource allocation Cluster stdlib (REEF) factoring out common functionalities Agenda
- 4. Cluster as an Embedded Systems “the era of map-reduce only clusters”
- 5. Purpose-built technology Within large web companies Well targeted mission (process webcrawl) à scale and fault tolerance The origin Google leading the pack Google File System + MapReduce (2003/2004) Open-source and parallel efforts Yahoo! Hadoop ecosystem HDFS + MR (2006/2007) Microsoft Scope/Cosmos (2008) (more than MR)
- 6. In-house growth What was the key to success for Hadoop?
- 7. In-house growth (following Hadoop story) Access, access, access… All the data sit in the DFS Trivial to use massive compute power à lots of new applications But… everything has to be MR Cast any computation as map-only job MPI, graph processing, streaming, launching web-servers!?!
- 8. Popularization Everybody wants Big Data Insight from raw data is cool Outside MS and Google, Big-Data == Hadoop Hadoop as catch-all big-data solution (and cluster manager)
- 9. Not just massive in-house clusters New challenges? New deployment environments Small clusters (10s of machines) Public Cloud
- 10. New deployment challenges Small clusters Efficiency matter more than scalability Admin/tuning done by mere mortals Cloud Untrusted users (security) Users are paying (availability, predictability) Users are unrelated to each other (performance isolation)
- 14. Classic Hadoop (1.0) Architecture Client Job1 JobTracker Scheduler TaskTracker TaskTracker TaskTracker Map Reduce Map Reduce Map Map Map Map Map Map Map Red. Red. Red. Red. Red. Red. Reduce Handles resource management Global invariants fairness/capacity Determines who runs / resources / where Manages MapReduce application flow maps before reducers, re-run upon failure, etc..
- 15. What are the key shortcomings of (old) Hadoop? Hadoop 1.0 Shortcomings
- 16. Programming model rigidity JobTracker manages resources JobTracker manages application workflow (data dependencies) Performance and Availability Map vs Reduce slots lead to low cluster utilization (~70%) JobTracker had too much to do: scalability concern JobTracker is a single point of failure Hadoop 1.0 Shortcomings (similar to original MR)
- 17. General purpose cluster OS “Cluster OS (YARN)”
- 18. YARN (2008-2013, Hadoop 2.x, production at Yahoo!, GA)* Request-based central scheduler Mesos (2011, UCB, open-sourced, tested at Twitter)* Offer-based two level scheduler Omega (2013, Google, simulation?)* Shared-state-based scheduling Corona (2013, Facebook, production) YARN-like but offered-based Four proposals * all three were best-papers or best-student-paper
- 19. YARN Ad-hoc app Ad-hoc app Ad-hoc app Ad-hoc Apps YARN MR v2 Tez Giraph Storm Dryad REEF ... Hive / Pig Hadoop 1.x (MapReduce) MR v1 Hive / Pig Users Application Frameworks Programming Model(s) Cluster OS (Resource Management) Hadoop 1 World Hadoop 2 World File System HDFS 1 HDFS 2 Hardware Ad-hoc app Ad-hoc app
- 20. A new architecture for Hadoop Decouples resource management from programming model (MapReduce is an “application” running on YARN) YARN (or Hadoop 2.x)
- 21. YARN (Hadoop 2) Architecture Client Job1 Resource Manager Scheduler NodeManager NodeManager NodeManager Task Task Task Task TaskApp Master Task Task Negotiate access to more resources (ResourceRequest)
- 22. Flexibility, Performance and Availability Multiple Programming Models Central components do less à scale better Easier High-Availability (e.g., RM vs AM) Why does this matter? System Jobs/Day Tasks/Day Cores pegged Hadoop 1.0 77k 4M 3.2 YARN 125k (150k) 12M (15M) 6 (10)
- 23. Anything else you can think?
- 24. Maintenance, Upgrade, and Experimentation Run with multiple framework versions (at one time) Trying out a new ideas is as is as launching a job Anything else you can think?
- 25. Real-time OS for the cluster (?) “predictable resource allocation”
- 26. YARN (Cosmos, Mesos, and Corona) support instantaneous scheduling invariants (fairness/capacity) maximize cluster throughput (eye to locality) Current trends New applications (require “gang” and dependencies) Consolidation of production/test clusters + Cloud (SLA jobs mixed with best-effort jobs) Motivation
- 27. Job/Pipeline with SLAs: 200 CPU hours by 6am (e.g., Oozie) Service: daily ebb/flows, reserve capacity accordingly (e.g., Samza) Gang: I need 50 concurrent containers for 3 hours (e.g., Giraph) Example Use Cases
- 28. In a consolidated cluster: Time-based SLAs for production jobs (completion deadline) Good latency for best-effort jobs High cluster utilization/throughput (Support rich applications: gang and skylines) High-Level Goals
- 29. Decompose time-based SLAs in resource definition: via RDL predictable resource allocation: planning + scheduling Divide and Conquer time-based SLAs
- 30. Expose to planner application needs time: start (s), finish (f) resources: capacity (w), total parallelism (h), minimum parallelism (l), min lease duration (t) Resource Definition Language (RDL) 1/2
- 31. Skylines / pipelines: dependencies: among atomic allocations (ALL, ANY, ORDER) Resource Definition Language (RDL) 2/2
- 32. Important classes Framework semantics: Perforator modeling of Scope/Hive Machine Learning: gang + bounded iterations (PREDict) Periodic jobs: history-based resource definition Coming up with RDL specs prediction
- 33. Root 100% Staging 15% Production 60% J1 10% J2 40% J3 10% Post 5% Best Effort 20% Planning vs Scheduling Plan Follower J3 J1 J3 J1 J3 J1 J3 J1 J3 ?? Scheduling (fine-grained but time-oblivious) Resource Definition Planning (coarse but time-aware) Preemption Plan Sharing Policy ReservationService Resource Manager Sys Model / Feedback
- 34. Some example run: lots of queues for gridmix Microsoft pipelines Dynamic queues
- 35. Improves production job SLAs best-effort jobs latency cluster utilization and throughput Comparing against Hadoop CapacityScheduler
- 36. Under promise, over deliver Plan for late execution, and run as early as you can Greedy Agent GB
- 37. Coping with imperfections (system) compensate RDL based on black-box models of overheads Coping with Failures (system) re-plan (move/kill allocations) in response of system- observable resource issues Coping with Failures/Misprediction (user) continue in best-effort mode when reservation expires re-negotiate existing reservations Dealing with “Reality”
- 38. Sharing Policy: CapacityOverTimePolicy constrains: instantaneous max, and running avg e.g., no user can exceed an instantaneous 30% allocation, and an average of 10% in any 24h period of time single partial scan of plan: O(|alloc| + |window|) User Quotas (trade-off flexibility to fairness)
- 39. Introduce Admission Control and Time-based SLAs (YARN-1051) New ReservationService API (to reserve resources) Agents + Plan + SharingPolicy to organize future allocations Leverage underlying scheduler Future Directions Work with MSR-India on RDL estimates for Hive and MR Advanced agents for placement ($$-based and optimal algos) Enforcing decisions (Linux Containers, Drawbridge, Pacer) Conclusion
- 40. Cluster stdlib: REEF “factoring out recurring components”
- 41. Dryad DAG computations Tez DAG computations (focus on interactive and Hive support) Storm stream processing Spark interactive / in-memory / iterative Giraph graph-processing Bulk Synchronous Parallel (a la’ Pregel) Impala scalable, interactive, SQL-like query HoYA Hbase on yarn Stratoshpere parallel iterative computations REEF, Weave, Spring-Hadoop meta-frameworks to help build apps Focusing on YARN: many applications
- 42. Lots of repeated work Communication Configuration Data and Control Flow Error handling / fault-tolerance Common “better than hadoop” tricks: Avoid Scheduling overheads Control Excessive disk IO Are YARN/Mesos/Omega enough?
- 43. The Challenge YARN / HDFS SQL / Hive … … Machine Learning u Fault Tolerance u Row/Column Storage u High Bandwidth Networking
- 44. The Challenge YARN / HDFS SQL / Hive … … Machine Learning u Fault Awareness u Local data caching u Low Latency Networking
- 45. SQL / Hive The Challenge YARN / HDFS … … Machine Learning
- 46. SQL / Hive REEF in the Stack YARN / HDFS … … Machine Learning REEF
- 47. REEF in the Stack (Future) YARN / HDFS SQL / Hive … … Machine Learning REEF Operator API and Library Logical Abstraction
- 48. REEF Client Job1 Resource Manager Scheduler NodeManager NodeManager NodeManager Task Task Task Task TaskApp Master Task Task Negotiate access to more resources (ResourceRequest)
- 49. REEF Client Job1 Resource Manager Scheduler NodeManager NodeManager NodeManager Evaluator Task services Evaluator Task services REEF RT Driver Name- based User control flow logic Retains State! User data crunching logic Fault- detection Injection-based checkable configuration Event-based Control flow
- 50. REEF: Computation and Data Management Extensible Control Flow Data Management Services Storage Network State Management Job Driver Control plane implementation. User code executed on YARN’s Application Master Activity User code executed within an Evaluator. Evaluator Execution Environment for Activities. One Evaluator is bound to one YARN Container.
- 51. Control Flow is centralized in the Driver Evaluator, Tasks configuration and launch Error Handling is centralized in the Driver All exceptions are forwarded to the Driver All APIs are asynchronous Support for: Caching / Checkpointing / Group communication Example Apps Running on REEF MR, Asynch Page Rank, ML regressions, PCA, distributed shell,…. REEF Summary (Open-sourced with Apache License)
- 52. Big-Data Systems Ongoing focus Future work Leverage high-level app semantics Coordinate tiered-storage and scheduling Conclusions
- 53. Adding Preemption to YARN, and open-sourcing it toApache
- 54. Limited mechanisms to “revise current schedule” Patience Container killing To enforce global properties Leave resources fallow (e.g., CapacityScheduler) à low utilization Kill containers (e.g., FairScheduler) à wasted work (Old) new trick Support work-preserving preemption (via) checkpointing à more than preemption State of the Art
- 55. Changes throughout YARN Client Job1 RM Scheduler NodeManager NodeManager NodeManager App Master Task Task Task Task Task Task Task PreemptionMessage { Strict { Set<ContainerID> } Flexible { Set<ResourceRequest>, Set<ContainerID> } } Collaborative application Policy-based binding for Flexible preemption requests Use of Preemption Context: Outdated information Delayed effects of actions Multi-actor orchestration Interesting type of preemption: RM declarative request AM bounds it to containers
- 56. Changes throughout YARN Client Job1 RM Scheduler NodeManager NodeManager NodeManager MR AM Task Task Task Task Task Task Task When can I preempt? tag safe UDFs or user-saved state @Preemptable public class MyReducer{ … } Common Checkpoint Service WriteChannel cwc = cs.create(); cwc.write(…state…); CheckpointID cid = cs.commit(cwc); ReadChannel crc = cs.open(cid);
- 57. 57 CapacityScheduler + Unreservation + Preemption: memory utilization CapacityScheduler (allow overcapacity) CapacityScheduler (no overcapacity)
- 58. Client Job1 RM Scheduler NodeManager NodeManager NodeManager App Master Task Task Task Task Task Task Task MR-5192 MR-5194 MR-5197 MR-5189 MR-5189 MR-5176 YARN-569 MR-5196 (Metapoint) Experience contributing to Apache Engaging with OSS talk with active developers show early/partial work small patches ok to leave things unfinished
- 59. With @Preemptable tag imperative code with semantic property Generalize this trick expose semantic properties to platform (@PreserveSortOrder) allow platforms to optimize execution (map-reduce pipelining) REEF seems the logical place where to do this. Tagging UDFs
- 60. (Basic) Building block for: Enables efficient preemption Dynamic Optimizations (task splitting, efficiency improvements) Fault Tolerance Other uses for Checkpointing
- 61. 61