Apache Spark Operations
Download as PPTX, PDF7 likes1,921 views
This document discusses Spark operations and architecture. It provides examples of Spark code and describes how Spark is deployed on different cluster managers like standalone, YARN, and Mesos. It discusses features like dynamic resource allocation, multi-tenancy, security, and authorization capabilities in Spark.
Apache Spark Operations
- 1. 1© Cloudera, Inc. All rights reserved. Spark Operations Kostas Sakellis
- 2. 2© Cloudera, Inc. All rights reserved. Me • Software Engineer at Cloudera • Contributor to Apache Spark • Before that, contributed to Cloudera Manager
- 3. 3© Cloudera, Inc. All rights reserved. Building a proof of concept! Courtesy of: http://www.nefloridadesign.com/mbimages/6.jpg
- 4. 4© Cloudera, Inc. All rights reserved. Example sc.textFile(“hdfs://data/u.item”, 4) .map(Movie(_)) .filter(_.month.equals(“Nov”)) .collect()
- 5. 5© Cloudera, Inc. All rights reserved. Example sc.textFile(“hdfs://data/u.item”, 4) .map(Movie(_)) .filter(_.month.equals(“Nov”)) .collect()
- 6. 6© Cloudera, Inc. All rights reserved. Example sc.textFile(“hdfs://data/u.item”, 4) .map(Movie(_)) .filter(_.month.equals(“Nov”)) .collect()
- 7. 7© Cloudera, Inc. All rights reserved. Partitions sc.textFile(“hdfs://data/u.item”, 4) .map(Movie(_)) .filter(_.month.equals(“Nov”)) .collect() HDFS Partition 1 Partition 2 Partition 3 Partition 4
- 8. 8© Cloudera, Inc. All rights reserved. RDDs sc.textFile(“hdfs://data/u.item”, 4) .map(Movie(_)) .filter(_.month.equals(“Nov”)) .collect() …RDD HDFS Partition 1 Partition 2 Partition 3 Partition 4
- 9. 9© Cloudera, Inc. All rights reserved. RDDs sc.textFile(“hdfs://data/u.item”, 4) .map(Movie(_)) .filter(_.month.equals(“Nov”)) .collect() …RDD …RDD HDFS Partition 1 Partition 2 Partition 3 Partition 4 Partition 1 Partition 2 Partition 3 Partition 4
- 10. 10© Cloudera, Inc. All rights reserved. RDDs sc.textFile(“hdfs://data/u.item”, 4) .map(Movie(_)) .filter(_.month.equals(“Nov”)) .collect() …RDD …RDD HDFS Partition 1 Partition 2 Partition 3 Partition 4 Partition 1 Partition 2 Partition 3 Partition 4 …RDD Partition 1 Partition 2 Partition 3 Partition 4
- 11. 11© Cloudera, Inc. All rights reserved. …RDD …RDD RDDs HDFS Partition 1 Partition 2 Partition 3 Partition 4 sc.textFile(“hdfs://data/u.item”, 4) .map(Movie(_)) .filter(_.month.equals(“Nov”)) .collect() Partition 1 Partition 2 Partition 3 Partition 4 …RDD Partition 1 Partition 2 Partition 3 Partition 4 Collect
- 12. 12© Cloudera, Inc. All rights reserved. …RDD …RDD RDD Lineage HDFS Partition 1 Partition 2 Partition 3 Partition 4 sc.textFile(“hdfs://data/u.item”, 4) .map(Movie(_)) .filter(_.month.equals(“Nov”)) .collect() Partition 1 Partition 2 Partition 3 Partition 4 …RDD Partition 1 Partition 2 Partition 3 Partition 4 Collect Lineage
- 13. 13© Cloudera, Inc. All rights reserved. Task …RDD …RDD HDFS Partition 1 Partition 2 Partition 3 Partition 4 Partition 1 Partition 2 Partition 3 Partition 4 …RDD Partition 1 Partition 2 Partition 3 Partition 4 Collect • A pipelined set of transformation on a single thread
- 14. 14© Cloudera, Inc. All rights reserved. Spark Architecture
- 15. 15© Cloudera, Inc. All rights reserved. Spark System Architecture
- 16. 16© Cloudera, Inc. All rights reserved. Deployments • Spark supports pluggable Cluster Managers • local, Standalone, YARN and Mesos • In early 2014, CDH 4.x with Spark 0.9 only supported Standalone • CDH 5.x includes Spark on YARN support
- 17. 17© Cloudera, Inc. All rights reserved. Standalone Master Worker Client Worker Process App Master Process
- 18. 18© Cloudera, Inc. All rights reserved. Standalone • On cluster ./sbin/start-master.sh ./sbin/start-slave.sh <master-spark-URL> • Submit job spark-submit --master <master-spark-URL> …
- 19. 19© Cloudera, Inc. All rights reserved. Container YARN Architecture Resource Manager Node Manager Client Node Manager Container Process App Master Container Process
- 20. 20© Cloudera, Inc. All rights reserved. Container Spark on YARN Architecture Resource Manager Node Manager Client Node Manager Container Process App Master Container Process
- 21. 21© Cloudera, Inc. All rights reserved. Container Spark on YARN Architecture Resource Manager Node Manager Client Node Manager Container Process App Master Container Process
- 22. 22© Cloudera, Inc. All rights reserved. Spark on YARN • Submit job spark-submit --master yarn-client … • Cluster mode spark-submit --master yarn-cluster … • Spark shell only works in client mode!
- 23. 23© Cloudera, Inc. All rights reserved. Customers often have shared infrastructure Courtesy of: https://radioglobalistic.files.wordpress.com/2011/02/lagos-traffic.jpg
- 24. 24© Cloudera, Inc. All rights reserved. Multi-tenancy • Cluster utilization is top metric • Target: 70-80% utilization • Mixed workloads from mixed customers • We recommend YARN • Built in resource manager
- 25. 25© Cloudera, Inc. All rights reserved. Underutilized Clusters Courtesy of: http://media.nbclosangeles.com/images/1200*675/60-freeway-repair-dec16-2-empty.JPG
- 26. 26© Cloudera, Inc. All rights reserved. Dynamic Allocation • Spark applications scale the number of executors based on load • Removes need for: --num-executors • Idle executors get killed • First supported in CDH 5.4 • Ideal for: • Long ETL jobs with large shuffles • shell applications: hive and spark shell
- 27. 27© Cloudera, Inc. All rights reserved. Dynamic Allocation Limitations • Still required to specify cores • --num-cores • Memory • --executor-memory • Includes JVM overhead • Need to do the math yourself • Our customers still get it wrong!
- 28. 28© Cloudera, Inc. All rights reserved. The Future of Dynamic Allocation • Only “task size” needed: --task-size • Eliminates • --num-cores • --num-executors • --executor-memory • Leads to better cluster utilization
- 29. 29© Cloudera, Inc. All rights reserved. Security, now it’s getting serious. Courtesy of: https://www.iti.illinois.edu/sites/default/files/Cybersecurity_image.jpg
- 30. 30© Cloudera, Inc. All rights reserved. Authentication • Kerberos – the necessary evil • Ubiquitous amongst other services • YARN, HDFS, Hive, HBase, etc. • Spark utilizes delegation tokens
- 31. 31© Cloudera, Inc. All rights reserved. Encryption • Control plane • File distribution • Block Manager • User UI / REST API • Data-at-rest (shuffle files) SPARK-6028 (Replace with netty) Replace with netty Spark 1.4 SPARK-2750 (SSL) SPARK-5682
- 32. 32© Cloudera, Inc. All rights reserved. Authorization • Enterprises have sensitive data • Beyond HDFS file permissions • Partial access to data • Column level granularity • Apache Sentry • HDFS-Sentry synchronization plugin • Record Service • Column level security for Spark!
- 33. 33© Cloudera, Inc. All rights reserved. Thank you We’re Hiring!
Editor's Notes
- #2: Lets talk about what we have seen as issues from our customers as issues as they try to get Spark into production.
- #3: In scope - Focus on operational issues - Not on building the code itself Experience from our customer support tickets
- #4: Spark makes building a proof of concept with a subset of data relatively easy. But then things go wrong Plug for my talk at Hadoop Summit
- #5: Lets start with an example program in Spark.
- #6: Lets start with an example program in Spark.
- #7: The sum() call launches a job
- #8: A chunk of data somewhere Could be on Hadoop File System (HDFS) Could be cached in Spark Defines the degree of parallelism
- #9: Describes a way of generating input and output partitions Immutable – very important! RDDs can depend on other RDDs Most have single parent Joins have multiple parents Lineage over replication for fault tolerance https://www.cs.berkeley.edu/~matei/papers/2012/nsdi_spark.pdf
- #10: Describes a way of generating input and output partitions Immutable – very important! RDDs can depend on other RDDs Most have single parent Joins have multiple parents Lineage over replication for fault tolerance https://www.cs.berkeley.edu/~matei/papers/2012/nsdi_spark.pdf
- #11: Describes a way of generating input and output partitions Immutable – very important! RDDs can depend on other RDDs Most have single parent Joins have multiple parents Lineage over replication for fault tolerance https://www.cs.berkeley.edu/~matei/papers/2012/nsdi_spark.pdf
- #12: Describes a way of generating input and output partitions Immutable – very important! RDDs can depend on other RDDs Most have single parent Joins have multiple parents Lineage over replication for fault tolerance https://www.cs.berkeley.edu/~matei/papers/2012/nsdi_spark.pdf
- #13: Describes a way of generating input and output partitions Immutable – very important! RDDs can depend on other RDDs Most have single parent Joins have multiple parents Lineage over replication for fault tolerance https://www.cs.berkeley.edu/~matei/papers/2012/nsdi_spark.pdf
- #15: Lets review the general Spark architecture
- #16: A driver Where the DAG scheduler lives Drives the show Single point of failure Executors Communicates with driver Runs the tasks created by the driver Think of this as a ThreadPoolExecutor in java Pluggable cluster managers YARN, Mesos, standalone
- #17: In scope - Focus on operational issues - Not on building the code itself Experience from our customer support tickets
- #18: Lets review the general Spark architecture
- #19: In scope - Focus on operational issues - Not on building the code itself Experience from our customer support tickets
- #20: Lets review the general Spark architecture
- #21: Lets review the general Spark architecture
- #22: Lets review the general Spark architecture
- #23: In scope - Focus on operational issues - Not on building the code itself Experience from our customer support tickets
- #24: Spark makes building a proof of concept with a subset of data relatively easy.
- #30: Spark makes building a proof of concept with a subset of data relatively easy.
- #32: Control plane File distribution Block Manager User UI / REST API Data-at-rest (shuffle files)