Meetup Kubernetes Rhein-Necker
0 likes562 views
The document outlines data-aware scheduling in Spark on Kubernetes, including its significance for both big data and non-big data applications. It discusses various components such as the Quobyte architecture, Hadoop Distributed File System (HDFS) integration, and scheduler architecture, highlighting the importance of data locality and scheduling mechanisms. The presentation serves as a proof of concept and invites community feedback while noting areas for further development and integration.
Meetup Kubernetes Rhein-Necker
- 1. Data-aware scheduling Spark on Kubernetes with HDFS Johannes M. Scheuermann Karlsruhe, 18.10.2017
- 2. Johannes M. Scheuermann IT Engineering & Operations @ inovex 〉 Software-Defined Datacenters 〉 Infrastructure as Code 〉 Cloud technologies 〉 High Availability & Scalability 〉 Want‘s an IBM Z-Frame 〉 @johscheuer 2
- 4. • Why data-aware scheduling • Data-ware for non Big-Data application • Data-ware scheduler • Big Data on Kubernetes • Spark on Kubernetes • HDFS on Kubernetes Agenda
- 5. • I’m not a scheduling expert • Concept is/was a PoC • Share learnings/ideas • Get feedback from the community Spoiler(s) J
- 8. Data-aware scheduling for non Big-Data • Databases • (large) image processing • Video encoding • (Web)-Cache
- 9. • Distributed (parallel) POSIX file system • Any workload with high performance (incl. throughput, databases, small files) • Can be deployed in containers, on kubelet hosts. • Linearly scalable performance. • Fully fault-tolerant, split-brain safe Quobyte – What is Quobyte
- 11. • Metadata servers make placement decisions against policies • on file level • tiering, isolation, … • keep stripes of files on disks of same machine => enable local read • allow preferring writes to local storage servers => enable local write • Locality information can be retrieved per file • that’s where the scheduler hooks in Quobyte - Placement
- 13. • Specify wanted Data • Lookup Data Placement • Remapping if Storage runs in Containers • Schedule Pod Scheduling data-aware (file-based)
- 17. Benchmarks
- 18. (Spark) Big-Data on Kubernetes
- 19. • https://github.com/apache-spark-on-k8s/spark • Not the “faked” Spark on Kubernetes • Still in development • Still not in the official Apache Spark project • Current: v2.2.0-0.4.0 • Alpha/Beta ?! Spark on Kubernetes
- 20. Spark on Kubernetes Spark Core MesosYARNStandaloneKubernetes StreamingMLlibSparkSQLGraphX
- 21. • Integrates with Kubernetes • RBAC • Resource Quotas • Audit logging • Etc. • Only cluster-mode Spark on Kubernetes
- 22. Spark on Kubernetes (cluster-mode)
- 23. Spark + HDFS on Kubernetes Driver 1 Driver 2 Executor 1 Executor 2.1 HDFS NN HDFS DN 1 HDFS DN 2 Pod Network Host Network Executor 2.2
- 24. Demo
- 25. • Rack-locality • Node preferences • Priority-based scheduling (K8s 1.8 alpha) • NameNode HA • Kerberos support Missing pieces
- 26. Conclusions
- 27. • Good starting point • Good integration • Still some points open • Work for better integration (more general) • Play with it! Conclusions
- 29. Q&A
- 30. • https://issues.apache.org/jira/browse/SPARK-18278 • https://www.youtube.com/watch?v=0xRHONrWwvU& feature=youtu.be • https://www.youtube.com/watch?v=DxCDxi08HWo&f eature=youtu.be Further reading
- 31. Johannes M. Scheuermann inovex GmbH Johannes.scheuermann@inovex.de CC BY-NC-ND inovex.de +JohannesScheuermann github.com/johscheu er @johscheuer youtube.com/inovexGmb H