Dynamic Resource Allocation Spark on YARN
20 likes9,371 views
Spark on YARN allows Spark jobs to run efficiently on YARN clusters. It supports two modes: yarn-client mode where the driver runs locally, and yarn-cluster mode where the driver runs in a YARN container. Dynamic resource allocation allows Spark to dynamically allocate containers based on workload, launching and killing executors as needed. This improves resource utilization by avoiding inefficient allocation where containers remain unused after tasks complete. Configuration changes are required to enable the external shuffle service to store RDD state externally rather than within executors.
Dynamic Resource Allocation Spark on YARN
- 1. Dynamic Resource Allocation for Spark on YARN ozawa@apache.org Tsuyoshi Ozawa
- 2. What s YARN • A resource manager implementation for computer cluster
- 4. YARN overview • All resources are managed by ResourceManager • All tasks are launched on NodeManager • Client submit jobs via ResourceManager NodeManager NodeManager ResourceManager client
- 5. Spark on YARN • 2 mode • yarn-cluster • yarn-client
- 6. yarn-cluster mode • Launching Spark driver on YARN container • Working well with spark-submit NodeManager NodeManager NM container1 container2Spark AppMaster clientResource Manager 1 submit 2 launching master 3 launching executers spark driver
- 7. yarn-client mode • Launching Spark driver at client side • Working well with spark-shell NodeManager NodeManager NM container1 container2Spark AppMaster client Resource Manager 1 submit 2 launching master 3 launching executers spark driver 4. send commands
- 8. Spark on YARN • yarn-cluster mode Node1 Node2 Node3 container 1 container 2 AppMaster container 2
- 9. Problem • Inefficient resource management • containers cannot exit until job exits Node1 Node2 container container container container stage1 stage2 100% 100% 100% 100% 100%0%0% 0%
- 10. Dynamic resource allocation(since v1.2) • Allocating containers more dynamically • number of executers are decided by workload NodeManager NodeManager NM container1 container2Spark AppMaster clientResource Manager 1 submit 2 launching master 3 launching executers/ kill executors spark driver
- 11. Yak shaving • Where should we hold the state of Spark RDD? • If executers are killed, it ll be lost… NodeManager executer executer RDD RDD
- 12. external shuffle • Saving Spark RDD to NodeManager • NodeManager has a interface, external shuffle plugin • Now executers are stateless! NodeManager executer executer external shuffle plugin RDD (IntermediateFile) RDD (IntermediateFile)
- 13. How to install (with Apache Hadoop) • Copy shuffle plugin to nodemanager s classpath • Edit yarn-site.xml • Edit spark-defaults.conf
- 14. Copy shuffle jar to nodemanager s classpath $ cp lib/spark-*-yarn-shuffle.jar /home/ubuntu/hadoop/share/hadoop/yarn/
- 15. Edit yarn-site.xml • Adding shuffle plugin • Note that documentation for 1.2 includes typo - I PRed :-) • See documentation for 1.4
- 17. We re ready!! • num-executers are defined automatically
- 18. Demo
- 19. Summary • Spark on YARN • yarn-client mode • yarn-cluster mode • Spark can launch jobs efficiently on YARN with dynamic allocation