SlideShare a Scribd company logo

Spark after Dark by Chris Fregly of Databricks

Data Con LA, profile picture
Data Con LA

This document provides an extensive overview of Spark technologies, including streaming, machine learning, and graph processing, illustrated through various use cases and demos. It presents the evolution of Spark, key functionalities, execution models, and the integration with big data tools like Hadoop and AWS. Additionally, it covers advanced topics like lambda architecture, approximations, and cluster deployments to enhance performance and scalability.

Technology
Related topics:
Apache Spark
1 of 116
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
after! Dark! ! Spark Streaming, Machine Learning, Graph Processing, Lambda Architecture, Probabilistic Data Structs/Algos, Approximations! ! Chris Fregly!  
Who am I?   Former Netflix’er,! NetflixOSS Committer! (netflix.github.io)   ! ! Spark Contributor! (github.com/apache/spark)!     ! Author! (sparkinaction.com)! (effectivespark.com)! ! ! ! ! Field Engineering! (databricks.com)!  
Spark Streaming-Kinesis Jira  
Not-so-Quick Poll   •  Execution Engines! –  Spark, Hadoop, Hive, Pig! •  File Formats! –  Parquet, Avro, RCFile, ORCFile! •  Streaming! –  Spark Streaming, Storm, Flume, Kafka, Kinesis ! •  AWS Big Data! –  EMR, DynamoDB, Redshift, Kinesis! •  Lambda Architecture! –  Combined Batch and Real-time Processing! •  Machine Learning! –  Recommendations, Clustering, Classification! •  Graph Processing! –  PageRank, Nearest Neighbor, Triangle Count! •  Approximations! –  Bloom Filters, HyperLogLog, Count-min Sketch!
Agenda   •  Spark After Dark Overview! •  Spark Overview! •  Use Cases! •  API and Libraries! •  Machine Learning! •  Graph Processing! •  Execution Model! •  Fault Tolerance! •  Cluster Deployment! •  Monitoring! •  Scaling and Tuning! •  Lambda Architecture! •  Probabilistic Data Structs/Algos! •  Approximations! DEMOS!!! DEMOS!!! DEMOS!!! DEMOS!!! DEMOS!!! DEMOS!!! DEMOS!!! DEMOS!!! DEMOS!!! DEMOS!!! DEMOS!!! DEMOS!!! DEMOS!!! DEMOS!!! DEMOS!!!
Spark After Dark (SAD) Overview! sparkafterdark.com   “Better Dating through Algorithms”!
Spark After Dark (SAD) ~ Tinder*   Swipe Left (Reject) Swipe Right (Accept) *not affiliated with Tinder in any way  
Spark After Dark Sample Dataset  
Goal: Increase Matches  
Matches: Spark SQL, BlinkDB   •  Traditional rating aggregations! – Most-active users (SUM)! – Most-rated users (SUM)! – Top-rated users (AVG)! – Spark SQL! •  Approximate online rating aggregations! – Error-bound, time-bound! – When exact results aren’t needed! – Near-real-time trend analysis! – BlinkDB!
Matches: MLlib, GraphX   •  Find similar based on! interests, religion, etc! –  MLlib: K-Means Cluster, ! DIMSUM Similarity, ! DecisionTree & ! RandomForest Classifier! ! •  Recommendations! –  Fill in missing ratings! –  MLlib: Alternating Least ! Squares (ALS)!
Matches: MLlib + GraphX   •  Profile Text Analysis! –  MLlib: TF/IDF,! N-gram, LDA Topics! -  GraphX: TextRank! •  Top Influencers! –  Most desirable people! –  GraphX: PageRank! ! •  People You May Know! –  More intimate matches! –  GraphX: Shortest Path,! Connected Components!
Matches: Spark Streaming   •  Capture real-time, online, incremental trends!! •  Spark Streaming + MLlib + GraphX! – Clustering: StreamingKMeans.scala! – Logistics Regression: StreamingLogR.scala! – Linear Regression: StreamingLinR.scala! – Naïve-Bayes Classification: TODO! – DecisionTree/RandomForest: TODO! – PageRank: TODO! •  Spark Streaming + Spark SQL! – Lambda Architecture: real-time + batch  
Spark Overview  
What is Spark?   •  Based on 2007 Microsoft Dryad paper! •  Written in Scala! •  Supports Java, Python, SQL, and R! •  Data fits in memory when possible, but not required! •  Improved efficiency over MapReduce! – 100x in-memory, 2-10x on-disk! •  Compatible with Hadoop! – File formats, SerDes, UDFs, Hive, and Pig!
Timeline of Spark Evolution  
Spark and Berkeley AMP Lab! Berkeley Data Analytics Stack (BDAS)!
Advantages of Unified Libraries   •  Advancements in higher-level libraries are pushed down into core and vice-versa! •  Spark Core! –  Highly-optimized, low overhead, network-saturating shuffle! •  Spark Streaming! –  GC, memory management, and cleanup improvements ! •  Spark GraphX! –  IndexedRDD for random access within a partition vs scanning entire partition! •  MLlib ! –  Statistics (Correlations, sampling, heuristics, etc)!
Hadoop Support! •  Shark! –  Hive on Spark! –  Used Hive execution engine – didn’t allow for optimizations! –  *Deprecated in favor of Spark SQL + Thrift RPC Server for JDBC/ODBC connectivity! •  Hive on Spark! –  Cloudera project! •  Spork! –  Pig on Spark! –  Collabortion: Sigmoid Analytics and Twitter! •  Parquet! –  Columnar File! –  Collaboration: Twitter and Cloudera (Impala)!
Spark Use Cases   Spark SQL! •  Ad hoc, exploratory, interactive analytics! •  Traditional ETL, analytics, and reporting! Spark Streaming + MLlib! •  Real-time traffic shifting based on CDN performance (video or music streaming)! •  Proactively notify customers when errors are elevated (SaaS model)! MLlib + GraphX! •  Recommendations, anomaly/spam detection, related news! •  Social analytics: “You May Know”, top influencers! Spark Streaming + Spark SQL! •  Lambda Architecture: batch + real-time!
Explosion of Specialized Systems  
Resilient Distributed Dataset! (RDD)  
RDD Overview   •  Core Spark abstraction! •  Represents partitions ! across the cluster nodes! •  Enables parallel processing ! on data sets! •  Partitions can be in-memory! or on-disk! •  Immutable, recomputable, ! fault tolerant! •  Contains transformation history (“lineage”) for whole data set!
RDD Interface   Lineage (Required)! 1.  Set of partitions! 2.  List of dependencies (0, 1, many)! 3.  Compute function per partition! Execution optimizations (Optional)! 4.  Partitioner (ie. Hash, Range)! 5.  Preferred locations per partition!
Types of RDDs   •  Spark Out-of-the-Box! –  HadoopRDD   –  FilteredRDD! –  MappedRDD! –  PairRDD! –  ShuffledRDD! –  UnionRDD! –  DoubleRDD! –  JdbcRDD! –  JsonRDD! –  SchemaRDD! –  VertexRDD! –  EdgeRDD! •  External! –  CassandraRDD (DataStax)! –  GeoRDD (Esri)! –  EsSpark (ElasticSearch)!
Hadoop RDD   Lineage (Required)! 1.  Set of partitions! 1 per HDFS block! 2.  List of dependencies! None! 3.  Compute function ! per partition! Read HDFS block ! Execution optimizations ! (Optional)! 4.  Partitioner! (ie. Hash, Range)! None! 5.  Preferred locations ! per partition! HDFS block location ! from NameNode!
Hadoop RDD Partitions  
Filtered RDD   Lineage (Required)! 1.  Set of partitions! Same as parent RDD! 2.  List of dependencies! Parent RDD! 3.  Compute function! Compute parent and filter! Execution optimizations ! (Optional)! 4.  Partitioner ! (ie. Hash, Range)! None! 5.  Preferred locations ! per partition! Same as parent RDD!
Mapped RDD   Lineage (Required)! 1.  Set of partitions! Same as parent RDD! 2.  List of dependencies! Parent RDD! 3.  Compute function! Compute parent, ! apply map function! Execution optimizations ! (Optional)! 4.  Partitioner ! (ie. Hash, Range)! None! 5.  Preferred locations ! per partition! Same as parent RDD!
Joined RDD   Lineage (Required)! 1.  Set of partitions! One per reduce task ! (PART- file)! 2.  List of dependencies! Every parent ! (many to many shuffle)! 3.  Compute function! Read and join ! shuffled data! Execution optimizations ! (Optional)! 4.  Partitioner ! (ie. Hash, Range)! HashPartitioner! 5.  Preferred locations! per partition! None!
RDD Operator DAG  
RDD Operator DAG  
Demo: Joining Users and Gender   •  Load user data! •  Load gender data! •  Join user data ! with gender data! •  Analyze lineage  
Join Optimizations   •  When joining large dataset with small dataset (reference data)! •  Broadcast small dataset to each node/partition of large dataset (one broadcast per worker node)!
Spark Core API  
Spark Core API Overview   •  Richer, more expressive than MapReduce! •  “Distributed Scala collections”! •  Native support for Java, Scala, Python, SQL, and R* (coming soon)! •  Unified API across all libraries! •  Operations ! – Transformations (lazy evaluation DAG)! – Actions (execute transformations)!
Why Scala?   •  Java/JVM is used by all popular big data frameworks! –  Lucene, SOLR, Hadoop, Mahout, Flume, Cassandra, HBase, Giraph, Storm, Avro, Protocol Buffers, Thrift, Parquet, etc! •  Supports closures! –  Allows shipping code to data! •  Functional, but not obscure! •  Typing allows performance optimizations! •  Leverage Spark REPL! •  It’s cool!!
Transformations (Lazy)  
Actions (Force Tranformations)  
Spark Execution Model  
Job Scheduler   •  Job! –  Contains many stages! •  Stages! –  Contains many tasks! •  Job resource allocation is aggressive! –  FIFO! •  Long-running jobs may starve resources for other jobs! –  Fair! •  Round-robin to prevent resource starvation! •  Fair Scheduler Pools! –  High-priority pool for important jobs! –  Separate user pools! –  FIFO within the pool! –  Modeled after Hadoop Fair Scheduler!
Spark Execution Model Overview   •  Parallel, distributed! •  DAG-based, lazy evaluation! •  Allows optimizations! –  Reduce disk I/O! –  Reduce shuffle I/O! –  Single pass through dataset! –  Parallel execution! –  Task pipelining! •  Data locality and rack awareness! •  PROCESS, NODE, RACK, ANY! •  Fault tolerance: RDD lineage graphs!
Execution Optimizations   ! ! ! !  
Task Pipelining  
Daytona GraySort Contest   250,000 partitions, on-disk (No caching)!
Improved Shuffle & Sort   •  aka “Sort-based Shuffle”! •  Released across both Spark 1.1 and 1.2! •  Improves performance for large number of partitions/reducers (>10,000)! •  Reduces memory pressure, GC, and OOMs! •  Reduces OS open files! •  Sequential versus random disk I/O! •  Saturate the network link! –  ~125 MB/s (1 GB ethernet)! –  1.25 GB/s (10 GB ethernet)!
Improved Shuffle   •  Sort-based Shuffle (SPARK-2045)! –  1 file for all partitions (grouped by partition key)! •  Versus many files (1 per partition key)! –  Sorted in-memory, sequential file write I/O! •  Versus random file write I/O (1 per partition key)! •  Java-Netty Network Module (SPARK-2468)! –  Replaces old nio socket-level code! –  epoll, zero-copy, kernel-space! –  Self-managed memory pool, less GC! •  External Shuffle Service (SPARK-3796)! –  Decoupled from the Executor JVMs! –  Uses Network Module! –  Can serve shuffle files after an Executor crash or during GC!
Improved Sort   •  TimSort! –  Combo of merge and insertion sort! –  Optimized for partially-ordered datasets! –  Used in both map and reduce phases! •  Reduced Data Movement, Cache Misses! –  Sort using (key, pointer) pairs! –  GraySort keys are 10 bytes (2^3.16227766??)! –  Compressed Oops = 4 bytes! –  10 key bytes + 2 pad bytes + 4 pointer bytes = 16 bytes (2^4!!)! –  AlphaSort Paper ~1995 (Chris Nyberg, Jim Gray)!
Elastic Spark   •  Scaling up/down based on resource idle threshold! •  Common problem! –  Long-running apps hold resources even when idle! –  Spark Shell is a long-running Spark app! –  Spark Streaming is a long running Spark app! •  Best practice! –  Scale up quickly (exponentionall)! –  Scale down slowly!
Spark Cluster Deployments  
Cluster Resource Managers   •  Spark Standalone (AMPLab, Spark default)! –  Suitable for a lot of production workloads! –  Only Spark workloads! •  YARN (Hadoop)! –  Allows hierarchies of resources! –  Kerberos integration! –  Multiple workloads from different execution frameworks! •  Hive, Pig, Spark, MapReduce, Cascading, etc! •  Mesos (AMPLab)! –  Similar to YARN, but allows elastic allocation to disparate execution frameworks! –  Coarse-grained ! •  Single, long-running Mesos tasks runs Spark mini tasks! –  Fine-grained! •  New Mesos task for each Spark task! •  Higher overhead, not good for long-running Spark jobs (ie. Streaming)!
Spark Standalone Deployment  
Spark YARN Deployment  
Spark Master High Availability   •  Multiple Master Nodes! •  ZooKeeper maintains current Master! •  Existing applications and workers will be notified of new Master election   •  New applications and workers need to explicitly specify current Master! •  Alternatives (Not recommended)! – Local filesystem ! – NFS Mount!
Spark Library APIs  
Spark SQL  
HiveQL (v13)  
Hive In-Memory Caching   •  In-memory columnar storage format! •  Physically stores columns together (instead of rows)! •  Optimized for fast column-based aggregations! –  SELECT AVG(age) FROM users! –  Not “SELECT *” which is row-based! •  Predicate pushdowns (WHERE age >= 21)! •  Column filtering (SELECT age FROM users)! •  Partition pruning (WHERE date=“2015-01”)!
Parquet File Format   •  On-disk columnar storage format! •  Heavy compression:! –  Run Length Encoding! –  Dictionary Encoding! –  Bit packing! •  Nested structures! •  Self-describing, ! evolving schema! •  Optimized for:! –  Column-based ! aggregations! –  Data at rest! •  Not optimized for:! –  SELECT * (row-based)! –  INSERTS/UPDATES (entire dataset needs re-encoding)!
Demo: Rating Aggregations (SparkSQL, Parquet)   •  Load CSV! •  Convert to JSON! •  Query JSON using SQL (slow)! •  Convert to Parquet file! •  Save Parquet file to disk! •  Query Parquet using SQL (fast!)  
MLlib and GraphX! (Machine Learning and! Graph Processing)  
Machine Learning Algos   •  Supervised! – Prediction: Train a model with existing data + label, predict label for new data! •  Classification (categorical or numeric)! •  Regression (continuous numeric)! – Recommendation: recommend to similars! •  User-user, item-item, user-item similarity! •  Unsupervised! – Clustering: Find natural clusters in data based on similarities (No training needed)! •  Must be parallelizable!!
ML Model Training   •  Input Data (Instances)! –  Training data (~80%)! –  Hold-out data: model validation (~10%) & testing (~10%)! –  Shape/density: Tall & skinny, short & wide, dense vs. sparse?! •  Features! –  Columns/attributes of instances! –  Types: Categorical, numeric (categorical & continuous)! –  Recommender: user id, product id! –  Clustering: height, religion, interest! –  Combine features to derive new ones! –  Requires domain knowledge! •  Labels! –  Supervised: predictions, classifications! •  Model Hyperparameters! –  Convergence threshold, number of iterations, etc!
ML Model Validation & Testing   •  Use hold-out data (20%)! –  Compare known validation label to model-predicted label! •  Measure the difference with evaluation criteria:! –  Root Mean Squared Error (RMSE)! –  Mean Absolute Error (MAE)! •  Optimize evaluation criteria! –  Choose the features and hyperparameters with the best evaluation criteria! –  ML Optimizer/ML Pipelines (AMPLab)! •  K-folds validation! –  Split input data k-ways! –  Alternate the role of each split: training, validation, testing! –  Especially useful for smaller datasets! •  Avoid overfitting to a particular dataset!!
Shape and Density of Input   •  Convert data to matrix! –  Rows (instances) x Columns (features)! •  SparseVector vs. DenseVector! –  Sparse: O(12nnz+4); ! nnz = number of non-zeros ! –  Dense requires O(8n); ! n = size of vector! •  SparseVector: wide rows! –  Lots of 0’s (most large distributed datasets)! –  LIBSVM Format! •  Some ML algos optimized for tall and skinny, some for short and wide! –  Depends on parallelism/shuffle characteristics!
Categorical Features! •  ML algos require data to be numeric (Double)! •  Categorical is usually not numeric! –  ie. NFL Teams: Bears, 49’ers, Steelers! •  Index? 1->Bears 2->49’ers 3-> Steelers! –  No! ML algo will assign irrelevant meaning to number! •  Instead, use “one-hot” or “1 of n” encoding! –  Convert categories to unique combos of 0’s and 1’s! –  ie. Bears -> 1,0,0! 49’ers -> 0,1,0! Steelers-> 0,0,1! •  Lots of open Spark PR’s for one-hot impls!
Similarity Algos (1/2)   •  DIMSUM! –  “Dimension Independent Matrix Square Using MapReduce”! –  From Twitter! –  Merged into MLlib as RowMatrix.columnSimilarities()! –  Similar to Cosine Similarity, but overcomes O(m*n^2) shuffle by sampling! –  Becomes O(n^2/Epsilon^2); n = columns, Epsilon is threshold! –  Sampling requires ! shuffling! –  40% improvement in ! Twitter Ad similarity ! computation   h#ps://blog.twi#er.com/2014/all-­‐pairs-­‐similarity-­‐via-­‐dimsum  
Similarity Algos (2/2)   •  LSH ! – Locality-Sensitive Hash! – Hashes values and creates buckets of similar values! – Hash function takes into account similarity using euclidean distance! – Requires pre-processing to create buckets! – Could suffer from skew! •  Reduce the space and return approximation! – O(n^2) brute force w/o LSH! – O(n/m * m^2) w/ LSH!
Common LSH Use Case   •  ”I have a 500k x 500k matrix and I want to calculate pair-wise similarities”! –  O(n^2), cartesian, lots of shuffle!! •  Use LSH!! –  Divide similar data into ! m buckets! –  Compare between buckets! –  Reduces down to ! O(n/m * m^2); ! where m = # of buckets! •  O(250,000,000,000) vs.! O(25,000,000); m=50!
Serving Model Results   •  Cold Start Problem! –  New users suffer cold start! –  Need suitable starting points! •  Top-rated users from Spark SQL! •  Most-desirable users from PageRank! •  Randomization! –  Don’t underestimate the power of randomizing! –  Need to surface results that wouldn’t otherwise surface! •  A/B test diff models to determine actual effectiveness! •  Combine model results! –  Track which model the result came from! –  Remember that position in results matters!
Demo: Generating Matches ! (PageRank & ALS)   GraphX: PageRank (TopK)   MLlib: ALS (Recommend)  
Spark Streaming  
“Streaming”   Kinesis   Streaming   Video ! Streaming   Piping   Big Data! Streaming  
Spark Streaming Overview! •  Low latency, high throughput, fault- tolerance (mostly)! •  Long-running Spark application! •  Supports Flume, Kafka, Twitter, Kinesis, Socket, File, etc.! •  Graceful shutdown, in-flight message draining! •  Uses Spark Core, DAG Execution Model, and Fault Tolerance! •  Submits micro-batch jobs to the cluster!
Spark Streaming Overview  
Spark Streaming Use Cases   •  ETL and enrichment ! of streaming data! on injest! –  Spark Streaming,! Spark SQL, MLlib! •  Lambda Architecture! –  Spark Streaming, ! Spark SQL, MLlib, GraphX! –  Everything!!! ! •  Operational dashboards! –  Lambda Architecture! –  Spark Streaming, ! Spark SQL!
Discretized Stream (DStream)   •  Core Spark Streaming abstraction! •  Micro-batches of RDDs! •  Operations similar to RDD! •  Operates on all underlying RDDs of DStream!
Spark Streaming API Overview   •  Rich, expressive API similar to core! •  Operations! –  Transformations (lazy)! –  Actions (execute transformations)! •  Window and Stateful Operations! –  Current state depends on previous state! •  Requires 2 types of durable checkpointing (HDFS) to snip long-running lineages! –  Metadata: DStream DAG, queue of jobs, ! –  Data: input data, window and stateful data! •  Register DStream as a Spark SQL table for querying?! Wow.  
DStream Transformations  
DStream Actions  
Window and State DStream Operations  
DStream Window and State Example  
Spark Streaming ! Cluster Deployment  
Streaming Cluster Deployment  
Adding Receivers    
Adding Workers  
Spark Streaming! +! Kinesis  
Spark Streaming + Kinesis Architecture  
Throughput and Pricing  
Demo: Real-time ALS Matches   Kinesis   Streaming   Scala:    …/scala/org/apache/spark/examples/streaming/KinesisWordCountASL.scala   Java:    …/java/org/apache/spark/examples/streaming/JavaKinesisWordCountASL.java     h#ps://github.com/apache/spark/blob/master/extras/kinesis-­‐asl/src/main/…  
Kinesis Best Practices   •  Avoid resharding! –  Over-provision shards to avoid splitting/merging & increase throughput! –  Remember to include egress rate when provisioning – not just ingest! –  Choose high-cardinality partition hash key to avoid hot spots/shards (ie. CustomerID, etc)! •  Avoid excess PUTs! –  Batch vs. single records! –  Kinesis Mobile Client supports batch to reduce battery consumption! –  Increases throughput! –  Filter on client-side! –  Async producer like FluentD, Log4J, Flume! •  Avoid data loss! –  Use an archiver consumer app to move the data into S3 for longer-term durability past 24 hours! –  Write to durable storage (Spark 1.2 WAL) before checkpointing! –  De-dupe as deep into the pipeline as possible – allows retry! –  Use deterministic checkpoint strategy to get repeatable failover results ! (ie. number of records since last checkpoint)! –  Idempotent processing where possible!
Spark Streaming! Fault Tolerance  
Types of Streaming Sources   •  Batched! –  Flume, Kafka, Kinesis! –  Improves throughput at expense of possible duplication during failure replay! •  Partitioned! –  Kafka, Kinesis! –  Different receivers dedicated to different partitions! –  Improves throughput through parallelism! –  Order is only guaranteed within a single partition! •  Buffered! –  Flume, Kafka, Kinesis! –  Allows back pressure, rate limiting, etc! •  Reliable (Checkpointed)! –  Kafka, Kinesis! –  Allows replay from specific checkpoint! –  Requires ACK after each batch is durably stored within Spark Streaming! •  Transactional! –  Flume! –  Start, Commit semantics!
Message Delivery Guarantees   •  Exactly once [1]! –  No loss! –  No redeliver! –  Perfect delivery! –  Incurs higher latency for transactional semantics! –  Spark default per batch using DStream lineage! –  Degrades to less guarantees depending on source! •  At least once [1..n]! –  No loss! –  Possible redeliver! •  At most once [0,1]! –  Possible loss! –  No redeliver! –  *Best configuration if some data loss is acceptable! •  Ordered! –  Per partition: Kafka, Kinesis! –  Global across all partitions: Hard to scale!
Types of Checkpoints   Spark! 1.  Metadata: Spark checkpointing of StreamingContext DStreams and metadata ! 2.  Data: Lineage of input data, state, and window DStream operations! Kinesis! 3.  Kinesis Client Library (KCL) checkpoints current position within shard! –  Checkpoint info is stored in DynamoDB per Kinesis application keyed by shard!
Fault Tolerance   •  Points of Failure! –  Receiver! •  Should restart, replace automatically! •  Possible data loss if not using Write Ahead Log (WAL)! –  Driver! •  Application is torn down including worker allocations! –  Worker/Executor! •  Normal Spark partition-level recovery occurs! •  Possible Solutions! –  Use HDFS File Source for durability   –  Data Replication! –  Secondary/Backup Nodes! –  Checkpoints! •  Stream, Window, and State info!
Streaming Receiver Failure   •  Use multiple receivers pulling from multiple shards! •  Use checkpoint-enabled, reliable, sharded source (ie. Kafka, Kinesis) for replay! •  Data is replicated to 1 other node in-memory immediately upon ingestion! –  2nd node is chosen at random! –  Spills to disk if data doesn’t fit in memory! •  Write Ahead Log (WAL)! –  Prevents losing in-flight metadata and data (received, but not processed)! –  Stored in HDFS! –  ACK back to source when safely stored in WAL! –  Spark processes the stored-but-not-processed data upon failover!
Streaming Driver Failure   •  Use a backup Driver ! –  Use DStream metadata checkpoint info to recover! –  StreamingContext.getOrCreate()! •  Single point of failure! –  Interrupts stream processing! –  State and Window data is lost! –  Use checkpointing to avoid data loss! •  Streaming Driver is a long-running Spark application! –  Schedules long-running receivers! –  Checkpointing prevents infinite DStream lineage for long-running streaming app!
Stream Worker/Processor Failure   •  No problem!! •  DStream RDD partitions will be recalculated from lineage! •  Causes temporary processing blip during failover!
Spark Streaming! Monitoring and Tuning  
Monitoring   •  Monitor driver, receiver, worker nodes, and streams! •  Alert upon failure or unusually high latency! •  Spark Web UI! – Streaming tab! •  Ganglia, CloudWatch (AWS)! •  Custom StreamingListener callback hooks during DStream processing!
Spark Web UI  
Tuning (1/2)   •  DStream batch time interval! –  Higher: reduces overhead by submitting less tasks overall! –  Lower: keeps latencies low! –  Sweet spot: DStream job time (scheduling + processing) is steady and less than batch interval! ! •  Checkpoint time interval! –  Higher: reduces load on checkpoint overhead! –  Lower: reduces amount of data loss on failure! –  Recommendation: 5-10x sliding window interval!
Tuning (2/2)   •  Use DStream.repartition() ! –  Increase parallelism of processing DStream jobs across more cluster nodes! •  Use spark.streaming.unpersist=true! –  Let’s the Streaming Framework figure out when to unpersist out-of-scope resources! •  Use Concurrent Mark and Sweep (CMS) GC Strategy! –  For consistent processing times! –  Note: Default stores data as serialized byte arrays to minimize GC! •  Consistency is key!!!
Spark and the Lambda Architecture  
Lambda Architecture Overview   •  Batch Layer! –  Immutable, ! Batch read,! Append-only write! –  Source of truth! –  ie. HDFS! •  Speed Layer! –  Mutable,! Random read/write! –  Most complex! –  Recent data only! –  ie. Cassandra! •  Serving Layer! –  Immutable,! Random read, ! Batch write! –  ie. ElephantDB!
Spark + AWS + Lambda  
Spark + Lambda + GraphX + MLlib  
Approximations  
Approximation Overview   •  Required for scaling efficiently! •  Speed up analysis of large datasets! •  Reduce size of working dataset! •  Data is messy! •  Collection of data is messy! •  Exact isn’t always necessary! •  “Approximate is the new Exact”! !
Probabilistic Data Structs/Algos   •  HyperLogLog! –  Count distinct elements for a date range! –  16K for 99% accuracy! •  Count-min Sketch! –  Heavy hitters, top-k, most-frequent elements! –  Supports streaming data! –  Fixed amount of space for all counts! •  Bloom Filter! –  Search/filter elements! –  Bitset representation of hash! –  “Maybe yes, definitely no”!
Benefits of Approximations                 Figure:    Memory  Savings  with  ApproximaLon  Techniques   (h#p://highlyscalable.wordpress.com/2012/05/01/probabilisLc-­‐structures-­‐web-­‐analyLcs-­‐data-­‐mining/)  
Spark’s Approximation Methods   •  BlinkDB! –  Approximate error-bound, time-bound “online aggregation” queries! •  RDD: sample(), takeSample()! –  Bernouilli and Poisson Sampling! •  PairRDD: countApproxDistinctByKey()! –  HyperLogLog   •  Spark Streaming and Twitter Algebird! •  Count-min Sketch! •  MLlib RowMatrix.columnSimilarities()! –  DIMSUM approximation! •  Bloom Filters! –  Everywhere!!
Spark’s Statistics Library   •  Correlations! –  Dependence between 2 random variables! –  Pearson, Spearman! •  Hypothesis Testing! –  Measure of statistical significance! –  Chi-squared test! •  Stratified Sampling! –  Sample separately from different sub-populations! –  Bernoulli and Poisson sampling! –  With and without replacement! •  Random data generator! –  Uniform, standard normal, and Poisson distribution  
Upcoming Spark Conferences  
Summary   •  Spark, Spark Streaming Overview ! •  Use Cases! •  API and Libraries! •  Machine Learning! •  Graph Processing! •  Execution Model! •  Fault Tolerance! •  Cluster Deployment! •  Monitoring! •  Scaling and Tuning! •  Lambda Architecture! •  Probabilistic Data Structs/Algos! •  Approximations! http://sparkinaction.com! http://effectivespark.com   Thanks!!! ! Chris Fregly! @cfregly!

More Related Content

PDF
Sparkcamp @ Strata CA: Intro to Apache Spark with Hands-on Tutorials
Databricks
 
PDF
Introduction to Apache Spark
Anastasios Skarlatidis
 
PDF
Fast Data Analytics with Spark and Python
Benjamin Bengfort
 
PPTX
Introduction to real time big data with Apache Spark
Taras Matyashovsky
 
PDF
Unified Big Data Processing with Apache Spark (QCON 2014)
Databricks
 
PDF
How Apache Spark fits into the Big Data landscape
Paco Nathan
 
PPTX
Introduction to Apache Spark Developer Training
Cloudera, Inc.
 
PDF
Apache Spark Tutorial
Ahmet Bulut
 
Sparkcamp @ Strata CA: Intro to Apache Spark with Hands-on Tutorials
Databricks
 
Introduction to Apache Spark
Anastasios Skarlatidis
 
Fast Data Analytics with Spark and Python
Benjamin Bengfort
 
Introduction to real time big data with Apache Spark
Taras Matyashovsky
 
Unified Big Data Processing with Apache Spark (QCON 2014)
Databricks
 
How Apache Spark fits into the Big Data landscape
Paco Nathan
 
Introduction to Apache Spark Developer Training
Cloudera, Inc.
 
Apache Spark Tutorial
Ahmet Bulut
 

What's hot (20)

PDF
Intro to apache spark stand ford
Thu Hiền
 
PDF
Apache Spark - Intro to Large-scale recommendations with Apache Spark and Python
Christian Perone
 
PDF
Apache Spark 2.0: Faster, Easier, and Smarter
Databricks
 
PDF
Spark overview
Lisa Hua
 
PPTX
Spark SQL
Caserta
 
PDF
Boston Apache Spark User Group (the Spahk group) - Introduction to Spark - 15...
spinningmatt
 
PDF
Spark SQL
Joud Khattab
 
PPTX
Introduction to Spark ML
Holden Karau
 
PPTX
Processing Large Data with Apache Spark -- HasGeek
Venkata Naga Ravi
 
PDF
Apache Spark Introduction
sudhakara st
 
PDF
Introduction to Spark Internals
Pietro Michiardi
 
PDF
Spark DataFrames: Simple and Fast Analytics on Structured Data at Spark Summi...
Databricks
 
PPTX
Spark After Dark: Real time Advanced Analytics and Machine Learning with Spark
Chris Fregly
 
PDF
A really really fast introduction to PySpark - lightning fast cluster computi...
Holden Karau
 
PPTX
Apache Spark sql
aftab alam
 
PPTX
Intro to Apache Spark
Robert Sanders
 
PDF
Spark ETL Techniques - Creating An Optimal Fantasy Baseball Roster
Don Drake
 
PPTX
Rapid Prototyping in PySpark Streaming: The Thermodynamics of Docker Containe...
Richard Seymour
 
PPTX
The Pushdown of Everything by Stephan Kessler and Santiago Mola
Spark Summit
 
PDF
Spark Summit East 2015 Advanced Devops Student Slides
Databricks
 
Intro to apache spark stand ford
Thu Hiền
 
Apache Spark - Intro to Large-scale recommendations with Apache Spark and Python
Christian Perone
 
Apache Spark 2.0: Faster, Easier, and Smarter
Databricks
 
Spark overview
Lisa Hua
 
Spark SQL
Caserta
 
Boston Apache Spark User Group (the Spahk group) - Introduction to Spark - 15...
spinningmatt
 
Spark SQL
Joud Khattab
 
Introduction to Spark ML
Holden Karau
 
Processing Large Data with Apache Spark -- HasGeek
Venkata Naga Ravi
 
Apache Spark Introduction
sudhakara st
 
Introduction to Spark Internals
Pietro Michiardi
 
Spark DataFrames: Simple and Fast Analytics on Structured Data at Spark Summi...
Databricks
 
Spark After Dark: Real time Advanced Analytics and Machine Learning with Spark
Chris Fregly
 
A really really fast introduction to PySpark - lightning fast cluster computi...
Holden Karau
 
Apache Spark sql
aftab alam
 
Intro to Apache Spark
Robert Sanders
 
Spark ETL Techniques - Creating An Optimal Fantasy Baseball Roster
Don Drake
 
Rapid Prototyping in PySpark Streaming: The Thermodynamics of Docker Containe...
Richard Seymour
 
The Pushdown of Everything by Stephan Kessler and Santiago Mola
Spark Summit
 
Spark Summit East 2015 Advanced Devops Student Slides
Databricks
 
Ad

Viewers also liked (20)

PDF
DSP開発におけるSpark MLlibの活用
Kotaro Tanahashi
 
PPT
Dot pab forum september 2011
The Social Executive
 
PDF
Big Data Day LA 2015 - Using data visualization to find patterns in multidime...
Data Con LA
 
PDF
Big Data Day LA 2015 - What's New Tajo 0.10 and Beyond by Hyunsik Choi of Gruter
Data Con LA
 
PPTX
Big Data Day LA 2015 - The Big Data Journey: How Big Data Practices Evolve at...
Data Con LA
 
PPT
101129 tokyopref bochibochi
redgang
 
PPTX
Big Data Day LA 2015 - Transforming into a data driven enterprise using exist...
Data Con LA
 
PDF
Big Data Day LA 2015 - Big Data Day LA 2015 - Applying GeoSpatial Analytics u...
Data Con LA
 
PPTX
An evening with Jay Kreps; author of Apache Kafka, Samza, Voldemort & Azkaban.
Data Con LA
 
PPTX
Big Data Day LA 2015 - Lessons learned from scaling Big Data in the Cloud by...
Data Con LA
 
PDF
Big Data Day LA 2016/ Data Science Track - Data Storytelling for Impact - Dav...
Data Con LA
 
PDF
Do you know how the ultra affluent use social media? Find out.
The Social Executive
 
PPTX
Spark Streaming& Kafka-The Future of Stream Processing by Hari Shreedharan of...
Data Con LA
 
PDF
Keeping Movies Running Amid Thunderstorms!
Sid Anand
 
PDF
6 damaging myths about social media and the truths behind them
The Social Executive
 
PPTX
Big Data Day LA 2016/ Use Case Driven track - Shaping the Role of Data Scienc...
Data Con LA
 
PPTX
Big Data Day LA 2016/ Use Case Driven track - From Clusters to Clouds, Hardwa...
Data Con LA
 
PPTX
Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Building an Event-oriented...
Data Con LA
 
PDF
Big Data Day LA 2016/ NoSQL track - Big Data and Real Estate, Jon Zifcak, CEO...
Data Con LA
 
PPTX
Big Data Day LA 2016/ Data Science Track - Intuit's Payments Risk Platform, D...
Data Con LA
 
DSP開発におけるSpark MLlibの活用
Kotaro Tanahashi
 
Dot pab forum september 2011
The Social Executive
 
Big Data Day LA 2015 - Using data visualization to find patterns in multidime...
Data Con LA
 
Big Data Day LA 2015 - What's New Tajo 0.10 and Beyond by Hyunsik Choi of Gruter
Data Con LA
 
Big Data Day LA 2015 - The Big Data Journey: How Big Data Practices Evolve at...
Data Con LA
 
101129 tokyopref bochibochi
redgang
 
Big Data Day LA 2015 - Transforming into a data driven enterprise using exist...
Data Con LA
 
Big Data Day LA 2015 - Big Data Day LA 2015 - Applying GeoSpatial Analytics u...
Data Con LA
 
An evening with Jay Kreps; author of Apache Kafka, Samza, Voldemort & Azkaban.
Data Con LA
 
Big Data Day LA 2015 - Lessons learned from scaling Big Data in the Cloud by...
Data Con LA
 
Big Data Day LA 2016/ Data Science Track - Data Storytelling for Impact - Dav...
Data Con LA
 
Do you know how the ultra affluent use social media? Find out.
The Social Executive
 
Spark Streaming& Kafka-The Future of Stream Processing by Hari Shreedharan of...
Data Con LA
 
Keeping Movies Running Amid Thunderstorms!
Sid Anand
 
6 damaging myths about social media and the truths behind them
The Social Executive
 
Big Data Day LA 2016/ Use Case Driven track - Shaping the Role of Data Scienc...
Data Con LA
 
Big Data Day LA 2016/ Use Case Driven track - From Clusters to Clouds, Hardwa...
Data Con LA
 
Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Building an Event-oriented...
Data Con LA
 
Big Data Day LA 2016/ NoSQL track - Big Data and Real Estate, Jon Zifcak, CEO...
Data Con LA
 
Big Data Day LA 2016/ Data Science Track - Intuit's Payments Risk Platform, D...
Data Con LA
 
Ad

Similar to Spark after Dark by Chris Fregly of Databricks (20)

PPTX
Global Big Data Conference Sept 2014 AWS Kinesis Spark Streaming Approximatio...
Chris Fregly
 
PPTX
East Bay Java User Group Oct 2014 Spark Streaming Kinesis Machine Learning
Chris Fregly
 
PDF
Liferay & Big Data Dev Con 2014
Miguel Pastor
 
PDF
Introduction to apache spark
UserReport
 
PDF
Apache Spark: The Next Gen toolset for Big Data Processing
prajods
 
PPTX
Apache Drill
Ted Dunning
 
PDF
OCF.tw's talk about "Introduction to spark"
Giivee The
 
PPTX
Big Data tools in practice
Darko Marjanovic
 
PDF
Apache Spark 101 - Demi Ben-Ari - Panorays
Demi Ben-Ari
 
PPTX
Apache Spark Fundamentals
Zahra Eskandari
 
PPTX
Unit II Real Time Data Processing tools.pptx
Rahul Borate
 
PPTX
Hadoop Ecosystem
Lior Sidi
 
PDF
Scaling Spark Workloads on YARN - Boulder/Denver July 2015
Mac Moore
 
PPTX
Kinesis and Spark Streaming - Advanced AWS Meetup - August 2014
Chris Fregly
 
PDF
An introduction to apache drill presentation
MapR Technologies
 
PDF
Apache Spark for Everyone - Women Who Code Workshop
Amanda Casari
 
PPTX
Apache Spark on HDinsight Training
Synergetics Learning and Cloud Consulting
 
PDF
Intro to Apache Spark
Marius Soutier
 
PPTX
Microsoft Openness Mongo DB
Heriyadi Janwar
 
PPTX
Apache Spark in Industry
Dorian Beganovic
 
Global Big Data Conference Sept 2014 AWS Kinesis Spark Streaming Approximatio...
Chris Fregly
 
East Bay Java User Group Oct 2014 Spark Streaming Kinesis Machine Learning
Chris Fregly
 
Liferay & Big Data Dev Con 2014
Miguel Pastor
 
Introduction to apache spark
UserReport
 
Apache Spark: The Next Gen toolset for Big Data Processing
prajods
 
Apache Drill
Ted Dunning
 
OCF.tw's talk about "Introduction to spark"
Giivee The
 
Big Data tools in practice
Darko Marjanovic
 
Apache Spark 101 - Demi Ben-Ari - Panorays
Demi Ben-Ari
 
Apache Spark Fundamentals
Zahra Eskandari
 
Unit II Real Time Data Processing tools.pptx
Rahul Borate
 
Hadoop Ecosystem
Lior Sidi
 
Scaling Spark Workloads on YARN - Boulder/Denver July 2015
Mac Moore
 
Kinesis and Spark Streaming - Advanced AWS Meetup - August 2014
Chris Fregly
 
An introduction to apache drill presentation
MapR Technologies
 
Apache Spark for Everyone - Women Who Code Workshop
Amanda Casari
 
Apache Spark on HDinsight Training
Synergetics Learning and Cloud Consulting
 
Intro to Apache Spark
Marius Soutier
 
Microsoft Openness Mongo DB
Heriyadi Janwar
 
Apache Spark in Industry
Dorian Beganovic
 

More from Data Con LA (20)

PPTX
Data Con LA 2022 Keynotes
Data Con LA
 
PPTX
Data Con LA 2022 Keynotes
Data Con LA
 
PDF
Data Con LA 2022 Keynote
Data Con LA
 
PPTX
Data Con LA 2022 - Startup Showcase
Data Con LA
 
PPTX
Data Con LA 2022 Keynote
Data Con LA
 
PDF
Data Con LA 2022 - Using Google trends data to build product recommendations
Data Con LA
 
PPTX
Data Con LA 2022 - AI Ethics
Data Con LA
 
PDF
Data Con LA 2022 - Improving disaster response with machine learning
Data Con LA
 
PDF
Data Con LA 2022 - What's new with MongoDB 6.0 and Atlas
Data Con LA
 
PDF
Data Con LA 2022 - Real world consumer segmentation
Data Con LA
 
PPTX
Data Con LA 2022 - Modernizing Analytics & AI for today's needs: Intuit Turbo...
Data Con LA
 
PPTX
Data Con LA 2022 - Moving Data at Scale to AWS
Data Con LA
 
PDF
Data Con LA 2022 - Collaborative Data Exploration using Conversational AI
Data Con LA
 
PDF
Data Con LA 2022 - Why Database Modernization Makes Your Data Decisions More ...
Data Con LA
 
PDF
Data Con LA 2022 - Intro to Data Science
Data Con LA
 
PDF
Data Con LA 2022 - How are NFTs and DeFi Changing Entertainment
Data Con LA
 
PPTX
Data Con LA 2022 - Why Data Quality vigilance requires an End-to-End, Automat...
Data Con LA
 
PPTX
Data Con LA 2022-Perfect Viral Ad prediction of Superbowl 2022 using Tease, T...
Data Con LA
 
PPTX
Data Con LA 2022- Embedding medical journeys with machine learning to improve...
Data Con LA
 
PPTX
Data Con LA 2022 - Data Streaming with Kafka
Data Con LA
 
Data Con LA 2022 Keynotes
Data Con LA
 
Data Con LA 2022 Keynotes
Data Con LA
 
Data Con LA 2022 Keynote
Data Con LA
 
Data Con LA 2022 - Startup Showcase
Data Con LA
 
Data Con LA 2022 Keynote
Data Con LA
 
Data Con LA 2022 - Using Google trends data to build product recommendations
Data Con LA
 
Data Con LA 2022 - AI Ethics
Data Con LA
 
Data Con LA 2022 - Improving disaster response with machine learning
Data Con LA
 
Data Con LA 2022 - What's new with MongoDB 6.0 and Atlas
Data Con LA
 
Data Con LA 2022 - Real world consumer segmentation
Data Con LA
 
Data Con LA 2022 - Modernizing Analytics & AI for today's needs: Intuit Turbo...
Data Con LA
 
Data Con LA 2022 - Moving Data at Scale to AWS
Data Con LA
 
Data Con LA 2022 - Collaborative Data Exploration using Conversational AI
Data Con LA
 
Data Con LA 2022 - Why Database Modernization Makes Your Data Decisions More ...
Data Con LA
 
Data Con LA 2022 - Intro to Data Science
Data Con LA
 
Data Con LA 2022 - How are NFTs and DeFi Changing Entertainment
Data Con LA
 
Data Con LA 2022 - Why Data Quality vigilance requires an End-to-End, Automat...
Data Con LA
 
Data Con LA 2022-Perfect Viral Ad prediction of Superbowl 2022 using Tease, T...
Data Con LA
 
Data Con LA 2022- Embedding medical journeys with machine learning to improve...
Data Con LA
 
Data Con LA 2022 - Data Streaming with Kafka
Data Con LA
 

Recently uploaded (20)

PDF
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
Akar16
 
PPTX
KOM of Painting work and Equipment Insulation REV00 update 25-dec.pptx
muhammadmuneebnaeem7
 
PDF
How UI/UX Design Impacts User Retention in Mobile Apps.pdf
SynapseIndia
 
PDF
Approach and Philosophy of On baking technology
30anthuong17
 
PDF
Reach Out and Touch Someone: Haptics and Empathic Computing
Mark Billinghurst
 
PPTX
Understanding_Digital_Forensics_Presentation.pptx
ImranKhan423233
 
PPTX
MYSQL Presentation for SQL database connectivity
Swati270511
 
PDF
Mobile App Security Testing_ A Comprehensive Guide.pdf
flufftailshop
 
PPTX
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
ReZa AdineH
 
PPTX
Big Data Technologies - Introduction.pptx
kuthubussaman1
 
PPTX
PA Analog/Digital System: The Backbone of Modern Surveillance and Communication
AVTRON Technologies LLC
 
PPTX
VMware vSphere Foundation How to Sell Presentation-Ver1.4-2-14-2024.pptx
blackmambaettijean
 
PDF
TokAI - TikTok AI Agent : The First AI Application That Analyzes 10,000+ Vira...
SOFTTECHHUB
 
PPTX
Digital-Transformation-Roadmap-for-Companies.pptx
David Malick Dieng
 
PDF
Review of recent advances in non-invasive hemoglobin estimation
IAESIJAI
 
PDF
Architecting across the Boundaries of two Complex Domains - Healthcare & Tech...
Peter Tan
 
PDF
Unlocking AI with Model Context Protocol (MCP)
Brian McKeiver
 
PDF
KodekX | Application Modernization Development
KodekX
 
PDF
Modernizing your data center with Dell and AMD
Principled Technologies
 
PDF
Dropbox Q2 2025 Financial Results & Investor Presentation
Dropbox
 
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
Akar16
 
KOM of Painting work and Equipment Insulation REV00 update 25-dec.pptx
muhammadmuneebnaeem7
 
How UI/UX Design Impacts User Retention in Mobile Apps.pdf
SynapseIndia
 
Approach and Philosophy of On baking technology
30anthuong17
 
Reach Out and Touch Someone: Haptics and Empathic Computing
Mark Billinghurst
 
Understanding_Digital_Forensics_Presentation.pptx
ImranKhan423233
 
MYSQL Presentation for SQL database connectivity
Swati270511
 
Mobile App Security Testing_ A Comprehensive Guide.pdf
flufftailshop
 
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
ReZa AdineH
 
Big Data Technologies - Introduction.pptx
kuthubussaman1
 
PA Analog/Digital System: The Backbone of Modern Surveillance and Communication
AVTRON Technologies LLC
 
VMware vSphere Foundation How to Sell Presentation-Ver1.4-2-14-2024.pptx
blackmambaettijean
 
TokAI - TikTok AI Agent : The First AI Application That Analyzes 10,000+ Vira...
SOFTTECHHUB
 
Digital-Transformation-Roadmap-for-Companies.pptx
David Malick Dieng
 
Review of recent advances in non-invasive hemoglobin estimation
IAESIJAI
 
Architecting across the Boundaries of two Complex Domains - Healthcare & Tech...
Peter Tan
 
Unlocking AI with Model Context Protocol (MCP)
Brian McKeiver
 
KodekX | Application Modernization Development
KodekX
 
Modernizing your data center with Dell and AMD
Principled Technologies
 
Dropbox Q2 2025 Financial Results & Investor Presentation
Dropbox
 

Spark after Dark by Chris Fregly of Databricks

  • 1. after! Dark! ! Spark Streaming, Machine Learning, Graph Processing, Lambda Architecture, Probabilistic Data Structs/Algos, Approximations! ! Chris Fregly!  
  • 2. Who am I?   Former Netflix’er,! NetflixOSS Committer! (netflix.github.io)   ! ! Spark Contributor! (github.com/apache/spark)!     ! Author! (sparkinaction.com)! (effectivespark.com)! ! ! ! ! Field Engineering! (databricks.com)!  
  • 4. Not-so-Quick Poll   •  Execution Engines! –  Spark, Hadoop, Hive, Pig! •  File Formats! –  Parquet, Avro, RCFile, ORCFile! •  Streaming! –  Spark Streaming, Storm, Flume, Kafka, Kinesis ! •  AWS Big Data! –  EMR, DynamoDB, Redshift, Kinesis! •  Lambda Architecture! –  Combined Batch and Real-time Processing! •  Machine Learning! –  Recommendations, Clustering, Classification! •  Graph Processing! –  PageRank, Nearest Neighbor, Triangle Count! •  Approximations! –  Bloom Filters, HyperLogLog, Count-min Sketch!
  • 5. Agenda   •  Spark After Dark Overview! •  Spark Overview! •  Use Cases! •  API and Libraries! •  Machine Learning! •  Graph Processing! •  Execution Model! •  Fault Tolerance! •  Cluster Deployment! •  Monitoring! •  Scaling and Tuning! •  Lambda Architecture! •  Probabilistic Data Structs/Algos! •  Approximations! DEMOS!!! DEMOS!!! DEMOS!!! DEMOS!!! DEMOS!!! DEMOS!!! DEMOS!!! DEMOS!!! DEMOS!!! DEMOS!!! DEMOS!!! DEMOS!!! DEMOS!!! DEMOS!!! DEMOS!!!
  • 6. Spark After Dark (SAD) Overview! sparkafterdark.com   “Better Dating through Algorithms”!
  • 7. Spark After Dark (SAD) ~ Tinder*   Swipe Left (Reject) Swipe Right (Accept) *not affiliated with Tinder in any way  
  • 8. Spark After Dark Sample Dataset  
  • 10. Matches: Spark SQL, BlinkDB   •  Traditional rating aggregations! – Most-active users (SUM)! – Most-rated users (SUM)! – Top-rated users (AVG)! – Spark SQL! •  Approximate online rating aggregations! – Error-bound, time-bound! – When exact results aren’t needed! – Near-real-time trend analysis! – BlinkDB!
  • 11. Matches: MLlib, GraphX   •  Find similar based on! interests, religion, etc! –  MLlib: K-Means Cluster, ! DIMSUM Similarity, ! DecisionTree & ! RandomForest Classifier! ! •  Recommendations! –  Fill in missing ratings! –  MLlib: Alternating Least ! Squares (ALS)!
  • 12. Matches: MLlib + GraphX   •  Profile Text Analysis! –  MLlib: TF/IDF,! N-gram, LDA Topics! -  GraphX: TextRank! •  Top Influencers! –  Most desirable people! –  GraphX: PageRank! ! •  People You May Know! –  More intimate matches! –  GraphX: Shortest Path,! Connected Components!
  • 13. Matches: Spark Streaming   •  Capture real-time, online, incremental trends!! •  Spark Streaming + MLlib + GraphX! – Clustering: StreamingKMeans.scala! – Logistics Regression: StreamingLogR.scala! – Linear Regression: StreamingLinR.scala! – Naïve-Bayes Classification: TODO! – DecisionTree/RandomForest: TODO! – PageRank: TODO! •  Spark Streaming + Spark SQL! – Lambda Architecture: real-time + batch  
  • 15. What is Spark?   •  Based on 2007 Microsoft Dryad paper! •  Written in Scala! •  Supports Java, Python, SQL, and R! •  Data fits in memory when possible, but not required! •  Improved efficiency over MapReduce! – 100x in-memory, 2-10x on-disk! •  Compatible with Hadoop! – File formats, SerDes, UDFs, Hive, and Pig!
  • 16. Timeline of Spark Evolution  
  • 17. Spark and Berkeley AMP Lab! Berkeley Data Analytics Stack (BDAS)!
  • 18. Advantages of Unified Libraries   •  Advancements in higher-level libraries are pushed down into core and vice-versa! •  Spark Core! –  Highly-optimized, low overhead, network-saturating shuffle! •  Spark Streaming! –  GC, memory management, and cleanup improvements ! •  Spark GraphX! –  IndexedRDD for random access within a partition vs scanning entire partition! •  MLlib ! –  Statistics (Correlations, sampling, heuristics, etc)!
  • 19. Hadoop Support! •  Shark! –  Hive on Spark! –  Used Hive execution engine – didn’t allow for optimizations! –  *Deprecated in favor of Spark SQL + Thrift RPC Server for JDBC/ODBC connectivity! •  Hive on Spark! –  Cloudera project! •  Spork! –  Pig on Spark! –  Collabortion: Sigmoid Analytics and Twitter! •  Parquet! –  Columnar File! –  Collaboration: Twitter and Cloudera (Impala)!
  • 20. Spark Use Cases   Spark SQL! •  Ad hoc, exploratory, interactive analytics! •  Traditional ETL, analytics, and reporting! Spark Streaming + MLlib! •  Real-time traffic shifting based on CDN performance (video or music streaming)! •  Proactively notify customers when errors are elevated (SaaS model)! MLlib + GraphX! •  Recommendations, anomaly/spam detection, related news! •  Social analytics: “You May Know”, top influencers! Spark Streaming + Spark SQL! •  Lambda Architecture: batch + real-time!
  • 23. RDD Overview   •  Core Spark abstraction! •  Represents partitions ! across the cluster nodes! •  Enables parallel processing ! on data sets! •  Partitions can be in-memory! or on-disk! •  Immutable, recomputable, ! fault tolerant! •  Contains transformation history (“lineage”) for whole data set!
  • 24. RDD Interface   Lineage (Required)! 1.  Set of partitions! 2.  List of dependencies (0, 1, many)! 3.  Compute function per partition! Execution optimizations (Optional)! 4.  Partitioner (ie. Hash, Range)! 5.  Preferred locations per partition!
  • 25. Types of RDDs   •  Spark Out-of-the-Box! –  HadoopRDD   –  FilteredRDD! –  MappedRDD! –  PairRDD! –  ShuffledRDD! –  UnionRDD! –  DoubleRDD! –  JdbcRDD! –  JsonRDD! –  SchemaRDD! –  VertexRDD! –  EdgeRDD! •  External! –  CassandraRDD (DataStax)! –  GeoRDD (Esri)! –  EsSpark (ElasticSearch)!
  • 26. Hadoop RDD   Lineage (Required)! 1.  Set of partitions! 1 per HDFS block! 2.  List of dependencies! None! 3.  Compute function ! per partition! Read HDFS block ! Execution optimizations ! (Optional)! 4.  Partitioner! (ie. Hash, Range)! None! 5.  Preferred locations ! per partition! HDFS block location ! from NameNode!
  • 28. Filtered RDD   Lineage (Required)! 1.  Set of partitions! Same as parent RDD! 2.  List of dependencies! Parent RDD! 3.  Compute function! Compute parent and filter! Execution optimizations ! (Optional)! 4.  Partitioner ! (ie. Hash, Range)! None! 5.  Preferred locations ! per partition! Same as parent RDD!
  • 29. Mapped RDD   Lineage (Required)! 1.  Set of partitions! Same as parent RDD! 2.  List of dependencies! Parent RDD! 3.  Compute function! Compute parent, ! apply map function! Execution optimizations ! (Optional)! 4.  Partitioner ! (ie. Hash, Range)! None! 5.  Preferred locations ! per partition! Same as parent RDD!
  • 30. Joined RDD   Lineage (Required)! 1.  Set of partitions! One per reduce task ! (PART- file)! 2.  List of dependencies! Every parent ! (many to many shuffle)! 3.  Compute function! Read and join ! shuffled data! Execution optimizations ! (Optional)! 4.  Partitioner ! (ie. Hash, Range)! HashPartitioner! 5.  Preferred locations! per partition! None!
  • 33. Demo: Joining Users and Gender   •  Load user data! •  Load gender data! •  Join user data ! with gender data! •  Analyze lineage  
  • 34. Join Optimizations   •  When joining large dataset with small dataset (reference data)! •  Broadcast small dataset to each node/partition of large dataset (one broadcast per worker node)!
  • 36. Spark Core API Overview   •  Richer, more expressive than MapReduce! •  “Distributed Scala collections”! •  Native support for Java, Scala, Python, SQL, and R* (coming soon)! •  Unified API across all libraries! •  Operations ! – Transformations (lazy evaluation DAG)! – Actions (execute transformations)!
  • 37. Why Scala?   •  Java/JVM is used by all popular big data frameworks! –  Lucene, SOLR, Hadoop, Mahout, Flume, Cassandra, HBase, Giraph, Storm, Avro, Protocol Buffers, Thrift, Parquet, etc! •  Supports closures! –  Allows shipping code to data! •  Functional, but not obscure! •  Typing allows performance optimizations! •  Leverage Spark REPL! •  It’s cool!!
  • 41. Job Scheduler   •  Job! –  Contains many stages! •  Stages! –  Contains many tasks! •  Job resource allocation is aggressive! –  FIFO! •  Long-running jobs may starve resources for other jobs! –  Fair! •  Round-robin to prevent resource starvation! •  Fair Scheduler Pools! –  High-priority pool for important jobs! –  Separate user pools! –  FIFO within the pool! –  Modeled after Hadoop Fair Scheduler!
  • 42. Spark Execution Model Overview   •  Parallel, distributed! •  DAG-based, lazy evaluation! •  Allows optimizations! –  Reduce disk I/O! –  Reduce shuffle I/O! –  Single pass through dataset! –  Parallel execution! –  Task pipelining! •  Data locality and rack awareness! •  PROCESS, NODE, RACK, ANY! •  Fault tolerance: RDD lineage graphs!
  • 45. Daytona GraySort Contest   250,000 partitions, on-disk (No caching)!
  • 46. Improved Shuffle & Sort   •  aka “Sort-based Shuffle”! •  Released across both Spark 1.1 and 1.2! •  Improves performance for large number of partitions/reducers (>10,000)! •  Reduces memory pressure, GC, and OOMs! •  Reduces OS open files! •  Sequential versus random disk I/O! •  Saturate the network link! –  ~125 MB/s (1 GB ethernet)! –  1.25 GB/s (10 GB ethernet)!
  • 47. Improved Shuffle   •  Sort-based Shuffle (SPARK-2045)! –  1 file for all partitions (grouped by partition key)! •  Versus many files (1 per partition key)! –  Sorted in-memory, sequential file write I/O! •  Versus random file write I/O (1 per partition key)! •  Java-Netty Network Module (SPARK-2468)! –  Replaces old nio socket-level code! –  epoll, zero-copy, kernel-space! –  Self-managed memory pool, less GC! •  External Shuffle Service (SPARK-3796)! –  Decoupled from the Executor JVMs! –  Uses Network Module! –  Can serve shuffle files after an Executor crash or during GC!
  • 48. Improved Sort   •  TimSort! –  Combo of merge and insertion sort! –  Optimized for partially-ordered datasets! –  Used in both map and reduce phases! •  Reduced Data Movement, Cache Misses! –  Sort using (key, pointer) pairs! –  GraySort keys are 10 bytes (2^3.16227766??)! –  Compressed Oops = 4 bytes! –  10 key bytes + 2 pad bytes + 4 pointer bytes = 16 bytes (2^4!!)! –  AlphaSort Paper ~1995 (Chris Nyberg, Jim Gray)!
  • 49. Elastic Spark   •  Scaling up/down based on resource idle threshold! •  Common problem! –  Long-running apps hold resources even when idle! –  Spark Shell is a long-running Spark app! –  Spark Streaming is a long running Spark app! •  Best practice! –  Scale up quickly (exponentionall)! –  Scale down slowly!
  • 51. Cluster Resource Managers   •  Spark Standalone (AMPLab, Spark default)! –  Suitable for a lot of production workloads! –  Only Spark workloads! •  YARN (Hadoop)! –  Allows hierarchies of resources! –  Kerberos integration! –  Multiple workloads from different execution frameworks! •  Hive, Pig, Spark, MapReduce, Cascading, etc! •  Mesos (AMPLab)! –  Similar to YARN, but allows elastic allocation to disparate execution frameworks! –  Coarse-grained ! •  Single, long-running Mesos tasks runs Spark mini tasks! –  Fine-grained! •  New Mesos task for each Spark task! •  Higher overhead, not good for long-running Spark jobs (ie. Streaming)!
  • 54. Spark Master High Availability   •  Multiple Master Nodes! •  ZooKeeper maintains current Master! •  Existing applications and workers will be notified of new Master election   •  New applications and workers need to explicitly specify current Master! •  Alternatives (Not recommended)! – Local filesystem ! – NFS Mount!
  • 58. Hive In-Memory Caching   •  In-memory columnar storage format! •  Physically stores columns together (instead of rows)! •  Optimized for fast column-based aggregations! –  SELECT AVG(age) FROM users! –  Not “SELECT *” which is row-based! •  Predicate pushdowns (WHERE age >= 21)! •  Column filtering (SELECT age FROM users)! •  Partition pruning (WHERE date=“2015-01”)!
  • 59. Parquet File Format   •  On-disk columnar storage format! •  Heavy compression:! –  Run Length Encoding! –  Dictionary Encoding! –  Bit packing! •  Nested structures! •  Self-describing, ! evolving schema! •  Optimized for:! –  Column-based ! aggregations! –  Data at rest! •  Not optimized for:! –  SELECT * (row-based)! –  INSERTS/UPDATES (entire dataset needs re-encoding)!
  • 60. Demo: Rating Aggregations (SparkSQL, Parquet)   •  Load CSV! •  Convert to JSON! •  Query JSON using SQL (slow)! •  Convert to Parquet file! •  Save Parquet file to disk! •  Query Parquet using SQL (fast!)  
  • 61. MLlib and GraphX! (Machine Learning and! Graph Processing)  
  • 62. Machine Learning Algos   •  Supervised! – Prediction: Train a model with existing data + label, predict label for new data! •  Classification (categorical or numeric)! •  Regression (continuous numeric)! – Recommendation: recommend to similars! •  User-user, item-item, user-item similarity! •  Unsupervised! – Clustering: Find natural clusters in data based on similarities (No training needed)! •  Must be parallelizable!!
  • 63. ML Model Training   •  Input Data (Instances)! –  Training data (~80%)! –  Hold-out data: model validation (~10%) & testing (~10%)! –  Shape/density: Tall & skinny, short & wide, dense vs. sparse?! •  Features! –  Columns/attributes of instances! –  Types: Categorical, numeric (categorical & continuous)! –  Recommender: user id, product id! –  Clustering: height, religion, interest! –  Combine features to derive new ones! –  Requires domain knowledge! •  Labels! –  Supervised: predictions, classifications! •  Model Hyperparameters! –  Convergence threshold, number of iterations, etc!
  • 64. ML Model Validation & Testing   •  Use hold-out data (20%)! –  Compare known validation label to model-predicted label! •  Measure the difference with evaluation criteria:! –  Root Mean Squared Error (RMSE)! –  Mean Absolute Error (MAE)! •  Optimize evaluation criteria! –  Choose the features and hyperparameters with the best evaluation criteria! –  ML Optimizer/ML Pipelines (AMPLab)! •  K-folds validation! –  Split input data k-ways! –  Alternate the role of each split: training, validation, testing! –  Especially useful for smaller datasets! •  Avoid overfitting to a particular dataset!!
  • 65. Shape and Density of Input   •  Convert data to matrix! –  Rows (instances) x Columns (features)! •  SparseVector vs. DenseVector! –  Sparse: O(12nnz+4); ! nnz = number of non-zeros ! –  Dense requires O(8n); ! n = size of vector! •  SparseVector: wide rows! –  Lots of 0’s (most large distributed datasets)! –  LIBSVM Format! •  Some ML algos optimized for tall and skinny, some for short and wide! –  Depends on parallelism/shuffle characteristics!
  • 66. Categorical Features! •  ML algos require data to be numeric (Double)! •  Categorical is usually not numeric! –  ie. NFL Teams: Bears, 49’ers, Steelers! •  Index? 1->Bears 2->49’ers 3-> Steelers! –  No! ML algo will assign irrelevant meaning to number! •  Instead, use “one-hot” or “1 of n” encoding! –  Convert categories to unique combos of 0’s and 1’s! –  ie. Bears -> 1,0,0! 49’ers -> 0,1,0! Steelers-> 0,0,1! •  Lots of open Spark PR’s for one-hot impls!
  • 67. Similarity Algos (1/2)   •  DIMSUM! –  “Dimension Independent Matrix Square Using MapReduce”! –  From Twitter! –  Merged into MLlib as RowMatrix.columnSimilarities()! –  Similar to Cosine Similarity, but overcomes O(m*n^2) shuffle by sampling! –  Becomes O(n^2/Epsilon^2); n = columns, Epsilon is threshold! –  Sampling requires ! shuffling! –  40% improvement in ! Twitter Ad similarity ! computation   h#ps://blog.twi#er.com/2014/all-­‐pairs-­‐similarity-­‐via-­‐dimsum  
  • 68. Similarity Algos (2/2)   •  LSH ! – Locality-Sensitive Hash! – Hashes values and creates buckets of similar values! – Hash function takes into account similarity using euclidean distance! – Requires pre-processing to create buckets! – Could suffer from skew! •  Reduce the space and return approximation! – O(n^2) brute force w/o LSH! – O(n/m * m^2) w/ LSH!
  • 69. Common LSH Use Case   •  ”I have a 500k x 500k matrix and I want to calculate pair-wise similarities”! –  O(n^2), cartesian, lots of shuffle!! •  Use LSH!! –  Divide similar data into ! m buckets! –  Compare between buckets! –  Reduces down to ! O(n/m * m^2); ! where m = # of buckets! •  O(250,000,000,000) vs.! O(25,000,000); m=50!
  • 70. Serving Model Results   •  Cold Start Problem! –  New users suffer cold start! –  Need suitable starting points! •  Top-rated users from Spark SQL! •  Most-desirable users from PageRank! •  Randomization! –  Don’t underestimate the power of randomizing! –  Need to surface results that wouldn’t otherwise surface! •  A/B test diff models to determine actual effectiveness! •  Combine model results! –  Track which model the result came from! –  Remember that position in results matters!
  • 71. Demo: Generating Matches ! (PageRank & ALS)   GraphX: PageRank (TopK)   MLlib: ALS (Recommend)  
  • 73. “Streaming”   Kinesis   Streaming   Video ! Streaming   Piping   Big Data! Streaming  
  • 74. Spark Streaming Overview! •  Low latency, high throughput, fault- tolerance (mostly)! •  Long-running Spark application! •  Supports Flume, Kafka, Twitter, Kinesis, Socket, File, etc.! •  Graceful shutdown, in-flight message draining! •  Uses Spark Core, DAG Execution Model, and Fault Tolerance! •  Submits micro-batch jobs to the cluster!
  • 76. Spark Streaming Use Cases   •  ETL and enrichment ! of streaming data! on injest! –  Spark Streaming,! Spark SQL, MLlib! •  Lambda Architecture! –  Spark Streaming, ! Spark SQL, MLlib, GraphX! –  Everything!!! ! •  Operational dashboards! –  Lambda Architecture! –  Spark Streaming, ! Spark SQL!
  • 77. Discretized Stream (DStream)   •  Core Spark Streaming abstraction! •  Micro-batches of RDDs! •  Operations similar to RDD! •  Operates on all underlying RDDs of DStream!
  • 78. Spark Streaming API Overview   •  Rich, expressive API similar to core! •  Operations! –  Transformations (lazy)! –  Actions (execute transformations)! •  Window and Stateful Operations! –  Current state depends on previous state! •  Requires 2 types of durable checkpointing (HDFS) to snip long-running lineages! –  Metadata: DStream DAG, queue of jobs, ! –  Data: input data, window and stateful data! •  Register DStream as a Spark SQL table for querying?! Wow.  
  • 81. Window and State DStream Operations  
  • 82. DStream Window and State Example  
  • 83. Spark Streaming ! Cluster Deployment  
  • 88. Spark Streaming + Kinesis Architecture  
  • 90. Demo: Real-time ALS Matches   Kinesis   Streaming   Scala:    …/scala/org/apache/spark/examples/streaming/KinesisWordCountASL.scala   Java:    …/java/org/apache/spark/examples/streaming/JavaKinesisWordCountASL.java     h#ps://github.com/apache/spark/blob/master/extras/kinesis-­‐asl/src/main/…  
  • 91. Kinesis Best Practices   •  Avoid resharding! –  Over-provision shards to avoid splitting/merging & increase throughput! –  Remember to include egress rate when provisioning – not just ingest! –  Choose high-cardinality partition hash key to avoid hot spots/shards (ie. CustomerID, etc)! •  Avoid excess PUTs! –  Batch vs. single records! –  Kinesis Mobile Client supports batch to reduce battery consumption! –  Increases throughput! –  Filter on client-side! –  Async producer like FluentD, Log4J, Flume! •  Avoid data loss! –  Use an archiver consumer app to move the data into S3 for longer-term durability past 24 hours! –  Write to durable storage (Spark 1.2 WAL) before checkpointing! –  De-dupe as deep into the pipeline as possible – allows retry! –  Use deterministic checkpoint strategy to get repeatable failover results ! (ie. number of records since last checkpoint)! –  Idempotent processing where possible!
  • 93. Types of Streaming Sources   •  Batched! –  Flume, Kafka, Kinesis! –  Improves throughput at expense of possible duplication during failure replay! •  Partitioned! –  Kafka, Kinesis! –  Different receivers dedicated to different partitions! –  Improves throughput through parallelism! –  Order is only guaranteed within a single partition! •  Buffered! –  Flume, Kafka, Kinesis! –  Allows back pressure, rate limiting, etc! •  Reliable (Checkpointed)! –  Kafka, Kinesis! –  Allows replay from specific checkpoint! –  Requires ACK after each batch is durably stored within Spark Streaming! •  Transactional! –  Flume! –  Start, Commit semantics!
  • 94. Message Delivery Guarantees   •  Exactly once [1]! –  No loss! –  No redeliver! –  Perfect delivery! –  Incurs higher latency for transactional semantics! –  Spark default per batch using DStream lineage! –  Degrades to less guarantees depending on source! •  At least once [1..n]! –  No loss! –  Possible redeliver! •  At most once [0,1]! –  Possible loss! –  No redeliver! –  *Best configuration if some data loss is acceptable! •  Ordered! –  Per partition: Kafka, Kinesis! –  Global across all partitions: Hard to scale!
  • 95. Types of Checkpoints   Spark! 1.  Metadata: Spark checkpointing of StreamingContext DStreams and metadata ! 2.  Data: Lineage of input data, state, and window DStream operations! Kinesis! 3.  Kinesis Client Library (KCL) checkpoints current position within shard! –  Checkpoint info is stored in DynamoDB per Kinesis application keyed by shard!
  • 96. Fault Tolerance   •  Points of Failure! –  Receiver! •  Should restart, replace automatically! •  Possible data loss if not using Write Ahead Log (WAL)! –  Driver! •  Application is torn down including worker allocations! –  Worker/Executor! •  Normal Spark partition-level recovery occurs! •  Possible Solutions! –  Use HDFS File Source for durability   –  Data Replication! –  Secondary/Backup Nodes! –  Checkpoints! •  Stream, Window, and State info!
  • 97. Streaming Receiver Failure   •  Use multiple receivers pulling from multiple shards! •  Use checkpoint-enabled, reliable, sharded source (ie. Kafka, Kinesis) for replay! •  Data is replicated to 1 other node in-memory immediately upon ingestion! –  2nd node is chosen at random! –  Spills to disk if data doesn’t fit in memory! •  Write Ahead Log (WAL)! –  Prevents losing in-flight metadata and data (received, but not processed)! –  Stored in HDFS! –  ACK back to source when safely stored in WAL! –  Spark processes the stored-but-not-processed data upon failover!
  • 98. Streaming Driver Failure   •  Use a backup Driver ! –  Use DStream metadata checkpoint info to recover! –  StreamingContext.getOrCreate()! •  Single point of failure! –  Interrupts stream processing! –  State and Window data is lost! –  Use checkpointing to avoid data loss! •  Streaming Driver is a long-running Spark application! –  Schedules long-running receivers! –  Checkpointing prevents infinite DStream lineage for long-running streaming app!
  • 99. Stream Worker/Processor Failure   •  No problem!! •  DStream RDD partitions will be recalculated from lineage! •  Causes temporary processing blip during failover!
  • 101. Monitoring   •  Monitor driver, receiver, worker nodes, and streams! •  Alert upon failure or unusually high latency! •  Spark Web UI! – Streaming tab! •  Ganglia, CloudWatch (AWS)! •  Custom StreamingListener callback hooks during DStream processing!
  • 103. Tuning (1/2)   •  DStream batch time interval! –  Higher: reduces overhead by submitting less tasks overall! –  Lower: keeps latencies low! –  Sweet spot: DStream job time (scheduling + processing) is steady and less than batch interval! ! •  Checkpoint time interval! –  Higher: reduces load on checkpoint overhead! –  Lower: reduces amount of data loss on failure! –  Recommendation: 5-10x sliding window interval!
  • 104. Tuning (2/2)   •  Use DStream.repartition() ! –  Increase parallelism of processing DStream jobs across more cluster nodes! •  Use spark.streaming.unpersist=true! –  Let’s the Streaming Framework figure out when to unpersist out-of-scope resources! •  Use Concurrent Mark and Sweep (CMS) GC Strategy! –  For consistent processing times! –  Note: Default stores data as serialized byte arrays to minimize GC! •  Consistency is key!!!
  • 105. Spark and the Lambda Architecture  
  • 106. Lambda Architecture Overview   •  Batch Layer! –  Immutable, ! Batch read,! Append-only write! –  Source of truth! –  ie. HDFS! •  Speed Layer! –  Mutable,! Random read/write! –  Most complex! –  Recent data only! –  ie. Cassandra! •  Serving Layer! –  Immutable,! Random read, ! Batch write! –  ie. ElephantDB!
  • 107. Spark + AWS + Lambda  
  • 108. Spark + Lambda + GraphX + MLlib  
  • 110. Approximation Overview   •  Required for scaling efficiently! •  Speed up analysis of large datasets! •  Reduce size of working dataset! •  Data is messy! •  Collection of data is messy! •  Exact isn’t always necessary! •  “Approximate is the new Exact”! !
  • 111. Probabilistic Data Structs/Algos   •  HyperLogLog! –  Count distinct elements for a date range! –  16K for 99% accuracy! •  Count-min Sketch! –  Heavy hitters, top-k, most-frequent elements! –  Supports streaming data! –  Fixed amount of space for all counts! •  Bloom Filter! –  Search/filter elements! –  Bitset representation of hash! –  “Maybe yes, definitely no”!
  • 112. Benefits of Approximations                 Figure:    Memory  Savings  with  ApproximaLon  Techniques   (h#p://highlyscalable.wordpress.com/2012/05/01/probabilisLc-­‐structures-­‐web-­‐analyLcs-­‐data-­‐mining/)  
  • 113. Spark’s Approximation Methods   •  BlinkDB! –  Approximate error-bound, time-bound “online aggregation” queries! •  RDD: sample(), takeSample()! –  Bernouilli and Poisson Sampling! •  PairRDD: countApproxDistinctByKey()! –  HyperLogLog   •  Spark Streaming and Twitter Algebird! •  Count-min Sketch! •  MLlib RowMatrix.columnSimilarities()! –  DIMSUM approximation! •  Bloom Filters! –  Everywhere!!
  • 114. Spark’s Statistics Library   •  Correlations! –  Dependence between 2 random variables! –  Pearson, Spearman! •  Hypothesis Testing! –  Measure of statistical significance! –  Chi-squared test! •  Stratified Sampling! –  Sample separately from different sub-populations! –  Bernoulli and Poisson sampling! –  With and without replacement! •  Random data generator! –  Uniform, standard normal, and Poisson distribution  
  • 116. Summary   •  Spark, Spark Streaming Overview ! •  Use Cases! •  API and Libraries! •  Machine Learning! •  Graph Processing! •  Execution Model! •  Fault Tolerance! •  Cluster Deployment! •  Monitoring! •  Scaling and Tuning! •  Lambda Architecture! •  Probabilistic Data Structs/Algos! •  Approximations! http://sparkinaction.com! http://effectivespark.com   Thanks!!! ! Chris Fregly! @cfregly!