GPU Support in Spark and GPU/CPU Mixed Resource Scheduling at Production Scale
- 1. GPU Support in Spark and GPU/CPU Mixed Resource Scheduling at Production Scale Yonggang Hu, IBM, Distinguished Engineer Junfeng Liu, IBM, Architect
- 2. About us • Yonggang Hu Distinguished Engineer, IBM Chief Architect at Spectrum Computing, IBM. Vice President and Application Architect at JPMorgan Chase Working on distributed computing, grid, cloud and big data for the past 20 years. • Junfeng Liu IBM Spectrum Computing Architect, focusing on Big data platform design and implementation. Successfully delivering solutions to key customers. 2
- 3. Agenda • GPU and Spark integration motivation • The challenges of production deployments • Solutions • Demo • IBM® Spectrum Conductor with Spark 3
- 4. Spark and GPU Spark apps are CPU intensive Need to handle more data and bigger models Machine Learning Predictive analytics, Logistic regression, ALS Kmeans, etc. Graph Analytics Security, Fraud Detection Social Network Analytics GraphX Video/Speech Analytics Object Recognition Dialog Financial Risk Analytics Market simulation Credit risk, home-grown, apps from Murex, Misys Spark-enable existing GPU appsGPU-enable Spark apps 4
- 5. Various ways to integrate Spark and GPUs • Use GPUs for accelerating Spark Libraries and operations without changing interfaces and programming model – IBM Open source project: https://github.com/IBMSparkGPU – Nvidia GTC 2016 talk, • http://on-demand.gputechconf.com/gtc/2016/presentation/s6280-rajesh-bordawekar-accelerating-spark.pdf • http://on-demand.gputechconf.com/gtc/2016/video/S6211.html • Automatically generate CUDA code from the source Spark Java code – Nvidia GTC 2016 talk: http://on-demand.gputechconf.com/gtc/2016/presentation/s6346-kazuaki- ishizaki-gpu-programming-java-programmers.pdf • Integrate Spark with GPU-enabled application and system (e.g., Spark integrated with Caffe, TensorFlow and customer applications) 5
- 6. Production challenges • However – Identification of GPU execution vs. CPU execution in DAG – Data preparation for GPU execution – Low resource utilization for CPU or GPU or both • Cannot assume all compute hosts are identical and have GPU resource available • GPU is a lot more expensive !!! – Overload and contention when running mixed GPU and CPU workload – Long tail and GPU / CPU tasks failover – Monitoring and management Stage 2Stage 1 collect reduceByKey Stage GPU GPU Group CPU Group 6
- 7. A typical example Personalized Medicine – Adverse Drug Reaction Workload - 30X faster at learning speed and 4X speed up end-to-end - Need to fully utilize both GPU and CPU resources to get economic benefits 7 Source: http://on-demand.gputechconf.com/gtc/2016/presentation/s6280-rajesh-bordawekar-accelerating-spark.pdf 4X is not enough
- 8. Scheduling granularity • Scheduling at application level – Mesos and YARN tag the GPU machine with label – Schedule the application on GPU hosts based on resource requirement of application – Coarse-grained scheduling leads to low utilization of CPU/GPU • Scheduling at DAG level – Need fine-grained sharing for GPU resources rather than reserving entire GPU machines – Identify GPU operation – Optimize the DAG tree by decoupling GPU operations from CPU operations and by inserting new GPU stages – Reduce GPU wait time, enable sharing of GPUs among different jobs and therefore improve the overall GPU utilization 8
- 9. GPU task recognition • GPU and CPU tasks mixed together • Need to separate the workloads for scheduling control GPUFunction() Python-C Wrapper to Invoke Native Function Function implemented by CUDA/OpenCL GPU libraryPython-C/C++ Wrapper 9
- 10. GPU task recognition • Mark the GPU workload by DAG operation – Go through the DAG tree to identify stages with GPU requirement – Optimize the distribution by inserting GPU stage 10
- 11. Policies • Resource manager needs to be able to identify the GPU hosts and manage along with CPU resources • Prioritization policy - share GPU resources among applications • Allocation policy – control GPU and CPU allocation independently – multi-dimensional scheduling • Fine-grained policy to schedule tasks according to GPU-optimized DAG plan 11
- 12. Adaptive scheduling • CPU and GPU tasks are convertible in many applications • Scheduling needs adaptive capability – If GPU is available, use a portion of GPU – Otherwise run rest of tasks on CPU dstInBlocks.join(merged).mapV alues { …. if (useGPU) { loadGPULib() callGPU () } else { //CPU version } } 12
- 13. Adaptive scheduling CPU CPU CPU GPU GPU GPU Driver Executors Executors Executors Tasks -------------- Node 0 Node 1 Node n 13
- 14. Efficiency considerations • Do we need to wait for a GPU resource if there is a CPU available? • Do we need rerun the CPU tasks on GPU if tasks on CPU are long-tail? • Do we need to have failover cross-resource type? 14
- 15. Defer scheduling • Traditional defer scheduling – Wait for data locality – Cache, Host, Rack • Resource-based defer scheduling – Necessary if the GPU can greatly speed up task execution – Wait time is acceptable 15
- 16. Evaluation • Machine Learning – ALS on GPU – IBM Spark on GPU project – Various ML implementation on GPU – https://github.com/IBMSparkGPU • MovieLensALS Example – Spark Example, without modification – Binary compatible with Spark Application – Data Source • http://grouplens.org/datasets/movielens/
- 17. Evaluation Single Application Multiple Applications
- 18. Future work • Global optimization – Consider the cost of additional shuffle stage – Consider data locality of CPU and GPU stages – Add time dimension to MDS (multi-dimensional scheduling) – Optimize global DAG tree execution – Use historical data to optimize future execution, e.g, future iteration 18
- 19. Fine grain, dynamic allocation of resources maximizes efficiency of Spark instances sharing a common resource pool. Multi-tenant, multi-framework support. Eliminates cluster sprawl. 2 Run Spark natively on a shared infrastructure without the dependency of Hadoop. Reduce application wait time, improving time to results. 1 Building a Spark-Centric Shared Service with IBM spectrum Conductor End-to-End Enterprise Class Solution Improve Time to Results Proven architecture at extreme scale, with enterprise class workload management, multi-version support for Spark, monitoring, reporting, and security capabilities. 3 Reduce Administration Costs Increase Resource Utilization • IBM STC Spark Distribution • IBM Platform Resource Orchestrator / Session Scheduler, application service manager. • IBM Spectrum Scale FPO 4 19 http://www-03.ibm.com/systems/spectrum-computing/products/conductor/
- 20. IBM Spectrum Conductor with Spark IBM Bluemix Spark Cloud Service in production – thousands of users and tenants Third party audited benchmark indicated significant performance / throughput / SLA advantages https://ibm.biz/Bd4Tyw 20
- 21. IBM Spectrum Conductor with Spark Monitor and Reporting with Elastic (ELK) Integrated Elastic Search, Logstash, Kibana for customizable monitoring Built-in monitoring Metrics Cross Spark Instance Groups Cross Spark Applications within Spark Instance Group Within Spark Application Built-in monitoring inside Zeppelin Notebook 21
- 22. Demo 22
- 23. THANK YOU. Visit us at the IBM booth in the expo www.ibm.com/spectrum-conductor yhu@ca.ibm.com jfliu@ca.ibm.com
Editor's Notes
- #5: The synergy is coming from two directions. There are many existing GPU applications, such as deep learning apps – Caffe, Torch, commercial or home grown apps doing risk calculation etc, They could leverage Spark to support data distribution and data parallelization of computation on multiple GPU card. On the other hand, it is already well known that many Spark apps are compute intensive – within each Spark task there are many parallizable elements – Instead of running them on CPU, running them on GPU will greatly improve the performance. For example, we have implemented some of Mllib functions on GPU and achieve great speed. Also if each node does more work, we can reduce communication and network traffic.
- #6: IBM is actively working on and contributing to all three areas. Rajesh had a very talk on our work to accelerate Spark Mllib using GPU without changing Spark interface. Wei Tan presentation demonstrate how to use GPU to accelerate ALS. Our Japan team and Canada team worked on this thread. They had a talk at GTC 2016, and showed how to program GPU in pure Java using new parallel streaming API and JIT Cuda code generation.
- #7: How to effectively map Spark partitions to GPU nodes, then to make sure data will fit into GPU memory and move necessary data to GPU. Data might need to shuffled and certain data format (columnar format) is more suitable for GPU access. Spark memory manager needs to consider GPU memory as well. Spark is not GPU aware – when it creates DAG, when it distributes partition, manages memory. There is no way you can monitor and control it. How many GPUs are there, how GPUs are being used, utilization, etc.?
- #8: Builds a LBFGS – based logistic regression model to classify drug pairs
- #20: Spark centric infrastructure with Conductor
- #22: 21