Running Spark Inside Containers with Haohai Ma and Khalid Ahmed
3 likes2,208 views
The document discusses IBM Spectrum Computing's approach to running Spark within containers, focusing on the benefits of using containers for resource management, isolation, and ease of deployment. It outlines the architecture for deploying Spark workloads on Kubernetes, detailing the structure of Spark instance groups and the dynamic scalability based on workload demands. Future work includes integrating Kubernetes batch workload scheduling and comparing its performance with other Spark on Kubernetes solutions.
![Submitting workload with the container
definition
9IBM Spectrum Computing
spark-submit --class<main-class> --master<master-url> --deploy-mode cluster
--conf spark.ego.driver.docker.definition= MyAppDef
--conf spark.ego.executor.docker.definition= MyAppDef
<application-jar>
[application-arguments]
Cluster Mode:
Define container specifications for
the drivers and executors](https://image.slidesharecdn.com/la5ahmedma-171031223615/85/Running-Spark-Inside-Containers-with-Haohai-Ma-and-Khalid-Ahmed-9-320.jpg)
Running Spark Inside Containers with Haohai Ma and Khalid Ahmed
- 1. Haohai Ma, IBM Khalid Ahmed, IBM RUNNING SPARK INSIDE CONTAINERS
- 2. Myself • “How High” • Software Architect • IBM Spectrum Computing • Toronto Canada 2IBM Spectrum Computing
- 3. Agenda • Why container? • Migrate spark workload to container • Spark instance on Kubernetes – Architecture – Workflow – Multi-tenancy • Future work 3IBM Spectrum Computing
- 4. Why use containers? • To enforce the CPU and memory bounds. – CPU shares are proportional to the allocated slots – spark.driver.memory & spark.executor.memroy • To completely isolate the file system – Solve the dependency conflicts • To create and ship images – Develop once and run everywhere 4IBM Spectrum Computing
- 5. No prebuilt Spark image • A running container needs an application image – Independent to Spark versions • Seamlessly migrate Spark workloads to a container based environment – Assume: Spark is distributed onto the host file system 5IBM Spectrum Computing
- 6. Host Filesystem Spark installation Regular Spark workload IBM Spectrum Computing Spark Master JVM Spark Submit Spark Driver Spark Executor
- 7. Host Filesystem Spark installation Running in containers IBM Spectrum Computing Spark Master Spark Executor container: ubuntu JVM Spark Submit Spark Driver container: ubuntu container: image
- 8. Creating a container definition for an application image IBM Spectrum Computing Extra dependency from host file system
- 9. Submitting workload with the container definition 9IBM Spectrum Computing spark-submit --class<main-class> --master<master-url> --deploy-mode cluster --conf spark.ego.driver.docker.definition= MyAppDef --conf spark.ego.executor.docker.definition= MyAppDef <application-jar> [application-arguments] Cluster Mode: Define container specifications for the drivers and executors
- 10. Host Filesystem Spark installation Running in containers IBM Spectrum Computing Spark Master Spark Executor Container: myappimage:v1 Spark Submit Spark Driver container: myappimage:v1 Infobatch lib
- 11. Spark Instance on Kubernetes • Increase resource utilization – Share nodes between Spark and surrounding ecosystem • Isolation between tenants and apply resource enforcement – Each tenant gets a dedicated Spark working instance – Tenant price plan can directly map to its resource quota • Simplify deployment and roll out 11IBM Spectrum Computing
- 12. Architecture IBM Spectrum Computing IBM Spectrum Conductor with Spark Spark Instance Group Spark Instance Group Spark Master History Server Notebook Shuffle Service Spark Master History Server Notebook Shuffle Service SparkaaS level Tenant level Spark instance group Admin Portal Spark Information Hub Image and deployment
- 13. Architecture IBM Spectrum Computing IBM Spectrum Conductor with Spark Spark Instance Group Spark Instance Group Spark Master History Server Notebook Shuffle Service Spark Master History Server Notebook Shuffle Service Spark Service level Tenant level Spark Information Hub Image and deployment Spark instance group Admin Portal • A Spark instance group is an independent deployment in Kubernetes. • A docker image is built automatically based on the Spark version, configuration, notebook edition, and user application dependencies. • Initially only one container for Spark services of the Spark instance group. • Dynamic scalability based on workload demand.
- 14. Architecture IBM Spectrum Computing IBM Spectrum Conductor with Spark Spark Instance Group Spark Instance Group Spark Master History Server Notebook Shuffle Service Spark Master History Server Notebook Shuffler Service SparkaaS level Tenant level Spark Information Hub Image and deployment Spark instance group Admin Portal • IBM Spectrum Conductor with Spark - End points • Admin manages Spark instance group life cycle • Tenant accesses Spark workloads and notebooks • Deploy by a helm chart and expose as a service with one single container: CWS master • Multiple deployments in a Kubernetes cluster • Cloud: One deployment for one Spaas • On-Prem: One deployment for one BU
- 15. Creating a master container Kubernetes
- 16. Creating a master container Kubernetes Namespace: ns4bu1 container: CWS spaas4bu1_cwsmaster
- 17. Creating a Spark instance group Kubernetes Namespace: ns4bu1 container: CWS spaas4bu1_cwsmaster Registry tenant1 Image
- 18. Deploying a Spark instance group Kubernetes Namespace: ns4bu1 container: CWS spaas4bu1_cwsmaster container: tenant1 spaas4bu1_tenant1 Registry tenant1 Image
- 19. Kubernetes Namespace: ns4bu1 container: tenant1 container: tenant1 container: tenant1 Scaling the Spark instance group based on workload demands container: tenant1 scheduler API Server K8s master Spark Master Spark Driver Spark ExecutorSpark Executor
- 20. Performance • Without Dynamic Scaling
- 21. Performance • With Dynamic Scaling • Without Dynamic Scaling
- 22. Multitenancy with Spark instance groups Kubernetes Namespace: ns4bu1 container: CWS spaas4bu1_cwsmaster Registry container: tenant1 spaas4bu1_tenant1 container: tenant1 spaas4bu1_tenant1 container: tenant1 spaas4bu1_tenant1 container: tenant1 spaas4bu1_tenant1 container: tenant2 spaas4bu1_tenant2 container: tenant2 spaas4bu1_tenant2 container: tenant2 spaas4bu1_tenant2 tenant1 Image tenant2 Image
- 24. Registry tenant1 Image tenant2 Image tenant3 Image tenant4 Image Multi-Spaas Kubernetes Namespace: ns4bu1 container: CWS spaas4bu1_cwsmaster container: tenant1 spaas4bu1_tenant1 container: tenant1 spaas4bu1_tenant1 container: tenant1 spaas4bu1_tenant1 container: tenant1 spaas4bu1_tenant1 container: tenant2 spaas4bu1_tenant2 container: tenant2 spaas4bu1_tenant2 Namespace: ns4bu2 container: CWS spaas4bu2_cwsmaster container: tenant3 spaas4bu2_tenant3 container: tenant4 spaas4bu2_tenant4
- 25. Survey: Spark on Kubernetes SPARK-18278 Standalone IBM Spectrum Conductor with Spark on Kubernetes Dynamic allocation on demand Yes Static Yes K8s interaction granularity Job level Instance level – static Instance level – dynamic Deployment Automation • Simple deploy by helm charts No Yes Yes Spark instance per tenant • Multi-job/workflow/user • Image with user applications • Security No limited Yes
- 26. Future work • Integration with Kubernetes batch workload scheduler – Kube-arbitrator (https://github.com/kubernetes- incubator/kube-arbitrator) • Performance comparation with other Spark on Kubernetes solutions 26IBM Spectrum Computing