Improving Apache Spark for Dynamic Allocation and Spot Instances
1 like317 views
Holden Karau discusses improvements to Apache Spark for dynamic allocation and spot instances, highlighting challenges related to data resilience and the impact of cloud technology and Kubernetes. The talk reflects on past experiences and personal anecdotes, including recovering from an accident that affected development work. Key features and future improvements for Spark's decommissioning capabilities are also outlined.
Improving Apache Spark for Dynamic Allocation and Spot Instances
- 1. Apple logo is a trademark of Apple Inc. Holden Karau | Data / AI Summi t @holdenkara u Improving Spark for Dynamic Allocation & Spot Instances
- 2. Who am I? • Holden Kara u • She / he r • Apache Spark PMC • Contributor to a lot of other projects • co-author of High Performance Spark, Learning Spark, and Kubeflow for Machine Learning • http://bit.ly/holdenSparkVideos • https://youtube.com/user/holdenkarau
- 3. Apple logo is a trademark of Apple Inc.
- 4. Let us start at the beginning • Spark achieves resilience through re-computation which is part of how we go fas • This poses challenges with removing executors that may contain dat • We "solved" it for YARN/Mesos back in the da • I drank waaaay too much coffee and came up with an alternativ • But no one really liked it because we didn't need it so I closed the Google doc and forgot about i t • Don’t worry, we’ll get to the code soon :)
- 5. But then…. • The "cloud" became really popula r • Kubernetes became popula r • Everything caught on fire :/
- 6. Our Protagonist Remembers • I started drinking a lot of coffee • We dusted off that old design and wrote some cod e • And then I got hit by a ca r • More people wrote more cod e • We had a VOT E • We wrote waaaaay more cod e • Everyone lived happily ever after? Photo by Lukas from Pexels
- 7. How did DA work on YARN? • Scale up is "easy" (add more resources ) • Scale down required a stay resident program to be on each YARN node to serve any file s • Spark stored it's shuffle data as file s • Persist in memory data was still lost when scaling down an executor Photo by Markus Spiske from Pexels
- 8. Why did the cloud impact this? • If you wanted a ~50% cost saving of spot/preemptible instances you might lose entire machine s • Yes Spark can "handle" this, but does so by recomputing data (expensive ) • You can't depend on leaving a program around to serve files when the server is just gon e • So we need to find a way to migrate the data
- 9. Ok sure the cloud, but K8s? • Kubernetes doesn't like like the idea of scheduling a stay resident program on every nod e • Also most people don't like the idea of shared disk here either (accros jobs/ users ) • So we need to find a way to migrate the data
- 10. SPARK-20624 • Yee-haw ! • Ok but more seriously how does it work? Great question lets open up the code • BlockManagerDecomissioner.scala is where most of the magic happens
- 12. Ok what about the car? Getting hit by a car sucks a lot Slowed down dev work while I did rehab to be able to walk & type again Shout out to everyone who helped me recover (from my wife, girlfriend, partners, my friends, to the hospital staff, nursing home, PT, OT, Ambulance, my employer for giving me time off, the Spark community for understanding I needed time off <3)
- 13. It’s early though so please be careful On a Happy Note: You can try this now • Enable the followin g - spark.decommission.enabled - spark.storage.decommission.enabled - spark.storage.decommission.rddBlocks.enabled - spark.storage.decommission.shuffleBlocks.enabled • Want to get fancy? Optionally enable: - spark.shuffle.externalStorage.enabled - And configure a storage backend ( spark.shuffle.externalStorage.backend)
- 14. Future work • Heuristics to migrate dat a • Improve container pre-emption selectio • Better heuristics around when to scale up and down containers
- 15. Please review this talk :)
- 16. TM and © 2021 Apple Inc. All rights reserved.