Optimizing Spark Deployments for Containers: Isolation, Safety, and Performance: Spark Summit East talk by William Benton
1 like2,038 views
The document discusses optimizing Spark deployments within containers, addressing key concerns in architecture, security, and performance. It highlights that Spark executors fit well into microservice architectures and emphasizes the importance of using lightweight isolation methods to enhance performance while ensuring security through practices like SELinux. Conclusions include recommendations for deployment strategies, security measures, and performance optimization techniques.
Optimizing Spark Deployments for Containers: Isolation, Safety, and Performance: Spark Summit East talk by William Benton
- 1. OPTIMIZING SPARK DEPLOYMENTS FOR CONTAINERS: ISOLATION, SAFETY, AND PERFORMANCE William Benton • @willb Red Hat, Inc.
- 2. OPTIMIZING SPARK DEPLOYMENTS FOR CONTAINERS: ISOLATION, SAFETY, AND PERFORMANCE William Benton • @willb Red Hat, Inc.
- 3. Forecast Background and definitions Architectural concerns Security concerns Performance concerns Conclusions and takeaways Background and definitions
- 4. Forecast Background and definitions Architectural concerns Security concerns Performance concerns Conclusions and takeaways Background and definitions Architectural concerns
- 5. Forecast Background and definitions Architectural concerns Security concerns Performance concerns Conclusions and takeaways Background and definitions Architectural concerns Security concerns
- 6. Forecast Background and definitions Architectural concerns Security concerns Performance concerns Conclusions and takeaways
- 8. What is a container? …a lightweight VM? …a way to totally isolate applications? …a packaging format for a container runtime or orchestration platform?
- 13. pid root /tmp/foo net $SPARK_HOME/bin/spark-class org.apache.spark.deploy.worker.Worker master:7077 container runtime
- 14. pid root /tmp/foo net $SPARK_HOME/bin/spark-class org.apache.spark.deploy.worker.Worker master:7077 container runtime SPEED LIMIT 55
- 17. What is a container? …a lightweight VM? …a way to totally isolate applications? …a packaging format for a container runtime or orchestration platform?
- 18. What is a container? …a lightweight VM? …a way to totally isolate applications? …a packaging format for a container runtime or orchestration platform? …a lightweight means to address some of the same use cases as VMs.
- 19. What is a container? …a lightweight VM? …a way to totally isolate applications? …a packaging format for a container runtime or orchestration platform? …a lightweight means to address some of the same use cases as VMs. …a way to provide reasonable, not exhaustive application isolation.
- 20. What is a container? …a lightweight VM? …a way to totally isolate applications? …a packaging format for a container runtime or orchestration platform? …a lightweight means to address some of the same use cases as VMs. …a way to provide reasonable, not exhaustive application isolation. …yes, but really just any Linux process with some special settings!
- 26. High-level app architecture federate events databases file, object storage transform transform transform archive
- 27. High-level app architecture federate trainmodels events databases file, object storage transform transform transform archive
- 28. High-level app architecture federate trainmodels events databases file, object storage management web and mobile reporting developer UItransform transform transform archive
- 29. High-level app architecture federate trainmodels events databases file, object storage management web and mobile reporting developer UItransform transform transform archive
- 30. High-level app architecture federate trainmodels archive events databases file, object storage management web and mobile reporting developer UItransform transform transform
- 31. High-level app architecture federate trainmodels archive events databases file, object storage management web and mobile reporting developer UItransform transform transform
- 32. High-level app architecture federate trainmodels archive events databases file, object storage management web and mobile reporting developer UItransform transform transform Spark is a natural fit for microservice architectures, since executors are microservices!
- 33. Monolithic Spark clusters Cluster scheduler Shared FS / object store Spark executor Spark executor Spark executor Spark executor Spark executor Spark executor Resource manager app 1 app 2 app 4app 3 Databases
- 34. Monolithic Spark clusters Cluster scheduler Shared FS / object store Spark executor Spark executor Spark executor Spark executor Spark executor Spark executor Resource manager app 1 app 2 app 4app 3 Databases
- 35. One cluster per application Resource manager Shared FS / object store app 1 app 2 app 5app 4 app 3 app 6 Databases
- 36. One cluster per application Resource manager Shared FS / object store app 1 app 2 app 5app 4 app 3 app 6 app 2 app 4 Databases
- 37. Security
- 45. spark-class /tmp/foo systemd nginx SELinux limits your exposure to an exploit in a container or a bug in a container runtime. Use SELinux
- 47. Root is root … /
- 53. Keeping secrets … /tmp/foo Shared FS / object store ACCESS_KEY=… SECRET_KEY=…
- 54. Keeping secrets cat <<EOF > secret.txt ACCESS_KEY=… SECRET_KEY=… EOF git add secret.txt
- 55. Keeping secrets cat <<EOF > secret.txt ACCESS_KEY=… SECRET_KEY=… EOF git add secret.txt export ACCESS_KEY=… export SECRET_KEY=…
- 56. Keeping secrets cat <<EOF > secret.txt ACCESS_KEY=… SECRET_KEY=… EOF git add secret.txt export ACCESS_KEY=… export SECRET_KEY=… kubectl create secret generic mysecrets --from-file=… --from-file=…
- 57. Keeping secrets cat <<EOF > secret.txt ACCESS_KEY=… SECRET_KEY=… EOF git add secret.txt export ACCESS_KEY=… export SECRET_KEY=… kubectl create secret generic mysecrets --from-file=… --from-file=…
- 58. Performance
- 60. Potential performance pitfalls Hypervisors introduce overhead. Use more lightweight isolation mechanisms to preserve performance.
- 63. Potential performance pitfalls Virtualized networking likely has minimal impact on overall application performance!
- 64. Potential performance pitfalls Virtualized networking likely has minimal impact on overall application performance! …but measure the performance of your I/O configuration!
- 67. Potential performance pitfalls SPEED LIMIT 55 Quotas mean some ubiquitous techniques can have surprising performance impact. Consider in particular parallel GC and disk buffer cache use.
- 68. Potential performance pitfalls SPEED LIMIT 55 Be sure you set your heap sizes based on your resource limits…or wait for OpenJDK 9!
- 70. Architectural takeaways Spark executors are already microservices. Consider using a single Spark cluster per application for flexible scheduling and easy deployments. Persistent storage lives outside of containers and is probably best accessed via service interfaces rather than through filesystem interfaces.
- 71. Security takeaways It isn’t safe to run arbitrary code just because you put it in a container. Use SELinux to minimize your exposure to error and malice. Don’t run as root unless you absolutely have to (and you probably don’t). Ad hoc mechanisms for configuring secrets are likely to leak information and are almost always a bad idea.
- 72. Performance takeaways Avoid hypervisor overhead by using different approaches to isolation. Measure everything, but virtualized networking likely has a minimal performance impact on real applications. Artificially throttled performance can be a real problem. Experiment with JVM settings, including serial GC, to reduce your chance of getting limited.
- 73. Configuration takeaways If you consume logs from standard output and error, consider using an alternate stack trace formatter to get exceptions in a single log record. If you use ephemeral user IDs, set SPARK_USER or use nss_wrapper so Hadoop file libraries won’t get confused.