One Click to Run Apache Spark as a Service on Kubernetes
0 likes34 views
The document discusses the integration of Apache Spark with Kubernetes, highlighting various challenges and their solutions in deployment and management. Key topics include the complexity of creating and managing Kubernetes clusters, the use of operators to simplify operations, and tools for log aggregation and dynamic resource allocation. The presentation emphasizes the importance of automation and ease of deployment for effective Spark operations on Kubernetes.
One Click to Run Apache Spark as a Service on Kubernetes
- 1. Bo Yang DoK Day Europe 2022 @ KubeCon Apache Spark on Kubernetes - Challenges and Solutions
- 2. DoK Day Europe 2022 @ KubeCon Agenda ● Quick Introduction ● Challenges and Solutions ● Conclusion ● One Click To Run
- 3. DoK Day Europe 2022 @ KubeCon Introduction ● Apache Spark: powerful tool to run data processing and machine learning jobs ● Kubernetes: extensible platform to run containerized workload and services ● Apache Spark + Kubernetes: 1 + 1 > 2 ● It works: after solving various challenges
- 4. DoK Day Europe 2022 @ KubeCon Challenge: Complexity ● A Lot of Moving Pieces ○ Create Kubernetes cluster ○ Create service account and set up permission ○ Run spark-submit ○ Use kubectl to check driver/executor pods
- 5. DoK Day Europe 2022 @ KubeCon Make it Simple: Operator ● Operator Pattern: simplify operation work ● Spark Operator: https://github.com/GoogleCloudPlatform/spark-on-k8s-operator ○ Spark Application CRD ○ No need to deal with details of spark-submit ○ Monitor and manage the application status
- 6. DoK Day Europe 2022 @ KubeCon Spark UI ● Challenge: Cannot Access Spark UI from Outside ● Solution: Spark Operator + Nginx Ingress Controller ○ Automate ingress rule to expose Spark UI ○ Leverage Nginx to modify Spark UI web page on the fly
- 7. DoK Day Europe 2022 @ KubeCon Spark Logs ● Challenges ○ No log aggregation ○ Executor logs gone after application finish ● Solutions ○ Log shipping tools (Fluentd, Fluent Bit, Logstash, etc.) ○ spark.kubernetes.executor.deleteOnTermination=false
- 8. DoK Day Europe 2022 @ KubeCon CPU/Memory Overhead ● Challenges ○ Executor memory not same as pod memory ○ Non trivial overhead ● Solutions ○ Spark conf: spark.kubernetes.memoryOverheadFactor ○ Use larger node in Kubernetes cluster
- 9. DoK Day Europe 2022 @ KubeCon Dynamic Allocation ● Why ○ Dynamically allocate and terminate workers (executors) ○ Increase cluster utilization and reduce cost ● Challenges ○ Issue on Kubernetes: shuffle data prevents terminating executors ○ No External Shuffle Service (v.s. Spark on YARN) ● Solutions ○ Decouple shuffle data from executor ○ Uber Remote Shuffle Service
- 10. DoK Day Europe 2022 @ KubeCon Kubernetes Default Scheduler ● Originally designed to orchestrate long running services ● Problems ○ Missing Job Queue ○ Static Resource Limit ○ Driver Deadlock
- 11. DoK Day Europe 2022 @ KubeCon Batch Friendly Scheduler ● Kubernetes Scheduling Framework ○ Support plugins to enhance behavior ○ Able to totally replace the default scheduler ● Options ○ Volcano: a batch scheduler inspired by machine learning workload ○ Apache YuniKorn: inspired by the YARN scheduler from Hadoop ○ Scheduler Plugin - scheduler plugin to support gang scheduling and run FIFO
- 12. DoK Day Europe 2022 @ KubeCon Auto Scaling ● Automatically scale up/out ● Cluster Autoscaler: horizontally scale the cluster ● Usage Example ○ Create different node groups for different teams in the same cluster ○ Enable Cluster Autoscaler for those node groups ○ Adjust different min and max sizes for different node groups
- 13. DoK Day Europe 2022 @ KubeCon Conclusion ● Spark on Kubernetes: low operation cost (after get it working) ● A lot of work! ● Make Deployment Easy ○ Automate ○ Repeatable
- 14. DoK Day Europe 2022 @ KubeCon Data Punch Project ● One Click to Create Spark Environment on Kubernetes ○ Create IAM Role for EKS Control Plane ○ Create IAM Role for EKS Node Instance ○ Create the actual EKS cluster ○ Create a node group ○ Install Nginx Ingress Controller ○ Install Spark Operator ○ Install Spark REST Service ● Command Example: punch install SparkOnK8s ● https://github.com/datapunchorg/punch
- 15. DoK Day Europe 2022 @ KubeCon Thank you!