Tiny Batches, in the wine: Shiny New Bits in Spark Streaming
36 likes7,572 views
The document discusses the evolution and capabilities of Apache Spark, a powerful engine for big data processing that supports both batch and real-time analytics. It highlights Spark's advantages, such as in-memory computing, ease of development, and its integration with various data sources, which is particularly useful for streaming applications. The presentation includes insights on Spark Streaming, its requirements, use cases, and projected advancements in stability and support for additional environments.
![Demo: PySpark Streaming Network Word Count
import sys!
from pyspark import SparkContext!
from pyspark.streaming import StreamingContext!
!
sc = SparkContext(appName="PyStreamNWC", master="local[*]")!
ssc = StreamingContext(sc, Seconds(5))!
!
lines = ssc.socketTextStream(sys.argv[1], int(sys.argv[2]))!
!
counts = lines.flatMap(lambda line: line.split(" ")) !
.map(lambda word: (word, 1)) !
.reduceByKey(lambda a, b: a+b)!
!
counts.pprint()!
!
ssc.start()!
ssc.awaitTermination()
47](https://image.slidesharecdn.com/ukstreaming-141109114751-conversion-gate02/85/Tiny-Batches-in-the-wine-Shiny-New-Bits-in-Spark-Streaming-47-320.jpg)
![Demo: PySpark Streaming Network Word Count - Stateful
import sys!
from pyspark import SparkContext!
from pyspark.streaming import StreamingContext!
!
def updateFunc (new_values, last_sum):!
return sum(new_values) + (last_sum or 0)!
!
sc = SparkContext(appName="PyStreamNWC", master="local[*]")!
ssc = StreamingContext(sc, Seconds(5))!
ssc.checkpoint("checkpoint")!
!
lines = ssc.socketTextStream(sys.argv[1], int(sys.argv[2]))!
!
counts = lines.flatMap(lambda line: line.split(" ")) !
.map(lambda word: (word, 1)) !
.updateStateByKey(updateFunc) !
.transform(lambda x: x.sortByKey())!
!
counts.pprint()!
!
ssc.start()!
ssc.awaitTermination()
48](https://image.slidesharecdn.com/ukstreaming-141109114751-conversion-gate02/85/Tiny-Batches-in-the-wine-Shiny-New-Bits-in-Spark-Streaming-48-320.jpg)
Tiny Batches, in the wine: Shiny New Bits in Spark Streaming
- 1. Tiny Batches, in the wine: Shiny New Bits in Spark Streaming London Spark Meetup 2014-11-11 meetup.com/Spark-London/events/217362972/ Paco Nathan @pacoid 1
- 2. BTW, in case anyone didn’t get the pun… http://youtu.be/mlCiDEXuxxA 2
- 3. Spark, the elevator pitch 3
- 4. Spark, the elevator pitch Developed in 2009 at UC Berkeley AMPLab, open sourced in 2010, Spark has since become one of the largest OSS communities in big data, with over 200 contributors in 50+ organizations spark.apache.org “Organizations that are looking at big data challenges – including collection, ETL, storage, exploration and analytics – should consider Spark for its in-memory performance and the breadth of its model. It supports advanced analytics solutions on Hadoop clusters, including the iterative model required for machine learning and graph analysis.” Gartner, Advanced Analytics and Data Science (2014) 4
- 5. Spark, the elevator pitch 5
- 6. Spark, the elevator pitch Spark Core is the general execution engine for the Spark platform that other functionality is built atop: ! • in-memory computing capabilities deliver speed • general execution model supports wide variety of use cases • ease of development – native APIs in Java, Scala, Python (+ SQL, Clojure, R) 6
- 7. WordCount in 3 lines of Spark Spark, the elevator pitch WordCount in 50+ lines of Java MR 7
- 8. Spark, the elevator pitch Sustained exponential growth, as one of the most active Apache projects ohloh.net/orgs/apache 8
- 9. A very brief history 9
- 10. A Brief History: Functional Programming for Big Data Theory, eight decades ago: what can be computed? Haskell Curry haskell.org Alonso Church wikipedia.org Praxis, four decades ago: algebra for applicative systems John Backus acm.org David Turner wikipedia.org Reality, two decades ago: machine data from web apps Pattie Maes MIT Media Lab 10
- 11. A Brief History: Functional Programming for Big Data circa 2002: mitigate risk of large distributed workloads lost due to disk failures on commodity hardware… Google File System Sanjay Ghemawat, Howard Gobioff, Shun-Tak Leung research.google.com/archive/gfs.html ! MapReduce: Simplified Data Processing on Large Clusters Jeffrey Dean, Sanjay Ghemawat research.google.com/archive/mapreduce.html 11
- 12. A Brief History: Functional Programming for Big Data 2002 2004 MapReduce paper 2002 MapReduce @ Google 2004 2006 2008 2010 2012 2014 2006 Hadoop @ Yahoo! 2014 Apache Spark top-level 2010 Spark paper 2008 Hadoop Summit 12
- 13. A Brief History: Functional Programming for Big Data MapReduce Pregel Giraph Dremel Drill S4 Storm F1 MillWheel General Batch Processing Specialized Systems: Impala GraphLab iterative, interactive, streaming, graph, etc. Tez MR doesn’t compose well for large applications, and so specialized systems emerged as workarounds 13
- 14. A Brief History: Functional Programming for Big Data circa 2010: a unified engine for enterprise data workflows, based on commodity hardware a decade later… Spark: Cluster Computing with Working Sets Matei Zaharia, Mosharaf Chowdhury, Michael Franklin, Scott Shenker, Ion Stoica people.csail.mit.edu/matei/papers/2010/hotcloud_spark.pdf ! Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Justin Ma, Murphy McCauley, Michael Franklin, Scott Shenker, Ion Stoica usenix.org/system/files/conference/nsdi12/nsdi12-final138.pdf 14
- 15. TL;DR: Applicative Systems and Functional Programming – RDDs action value RDD RDD RDD transformations RDD // transformed RDDs! val errors = lines.filter(_.startsWith("ERROR"))! val messages = errors.map(_.split("t")).map(r => r(1))! messages.cache() // action 1! messages.filter(_.contains("mysql")).count() 15
- 16. TL;DR: Generational trade-offs for distributing compute tasks Cheap Memory Cheap Storage Cheap Network recompute replicate reference (RDD) (DFS) (URI) 16
- 17. TL;DR: Smashing The Previous Petabyte Sort Record databricks.com/blog/2014/11/05/spark-officially-sets- a-new-record-in-large-scale-sorting.html 17
- 19. Why Streaming? Because Machine Data! I <3 Logs Jay Kreps O’Reilly (2014) shop.oreilly.com/product/ 0636920034339.do 19
- 20. Why Streaming? Because Google! MillWheel: Fault-Tolerant Stream Processing at Internet Scale Tyler Akidau, Alex Balikov, Kaya Bekiroglu, Slava Chernyak, Josh Haberman, Reuven Lax, Sam McVeety, Daniel Mills, Paul Nordstrom, Sam Whittle Very Large Data Bases (2013) research.google.com/pubs/ pub41378.html 20
- 21. Why Streaming? Because IoT! kickstarter.com/projects/1614456084/b4rm4n-be- a-cocktail-hero 21
- 22. Why Streaming? Because IoT! (exabytes/day per sensor) bits.blogs.nytimes.com/2013/06/19/g-e-makes-the-machine- and-then-uses-sensors-to-listen-to-it/ 22
- 24. Spark Streaming: Requirements Let’s consider the top-level requirements for a streaming framework: • clusters scalable to 100’s of nodes • low-latency, in the range of seconds (meets 90% of use case needs) • efficient recovery from failures (which is a hard problem in CS) • integrates with batch: many co’s run the same business logic both online+offline 24
- 25. Spark Streaming: Requirements Therefore, run a streaming computation as: a series of very small, deterministic batch jobs ! • Chop up the live stream into batches of X seconds • Spark treats each batch of data as RDDs and processes them using RDD operations • Finally, the processed results of the RDD operations are returned in batches 25
- 26. Spark Streaming: Requirements Therefore, run a streaming computation as: a series of very small, deterministic batch jobs ! • Batch sizes as low as ½ sec, latency of about 1 sec • Potential for combining batch processing and streaming processing in the same system 26
- 27. Spark Streaming: Integration Data can be ingested from many sources: Kafka, Flume, Twitter, ZeroMQ, TCP sockets, etc. Results can be pushed out to filesystems, databases, live dashboards, etc. Spark’s built-in machine learning algorithms and graph processing algorithms can be applied to data streams 27
- 28. Spark Streaming: Timeline 2012 project started 2013 alpha release (Spark 0.7) 2014 graduated (Spark 0.9) Discretized Streams: A Fault-Tolerant Model for Scalable Stream Processing Matei Zaharia, Tathagata Das, Haoyuan Li, Timothy Hunter, Scott Shenker, Ion Stoica Berkeley EECS (2012-12-14) www.eecs.berkeley.edu/Pubs/TechRpts/2012/EECS-2012-259.pdf project lead: Tathagata Das @tathadas 28
- 29. Spark Streaming: Requirements Typical kinds of applications: • datacenter operations • web app funnel metrics • ad optimization • anti-fraud • various telematics and much much more! 29
- 30. Spark Streaming: Some Excellent Resources Programming Guide spark.apache.org/docs/latest/streaming-programming- guide.html TD @ Spark Summit 2014 youtu.be/o-NXwFrNAWQ?list=PLTPXxbhUt- YWGNTaDj6HSjnHMxiTD1HCR “Deep Dive into Spark Streaming” slideshare.net/spark-project/deep-divewithsparkstreaming-tathagatadassparkmeetup20130617 Spark Reference Applications databricks.gitbooks.io/databricks-spark-reference- applications/ 30
- 31. Quiz: name the bits and pieces… import org.apache.spark.streaming._! import org.apache.spark.streaming.StreamingContext._! ! // create a StreamingContext with a SparkConf configuration! val ssc = new StreamingContext(sparkConf, Seconds(10))! ! // create a DStream that will connect to serverIP:serverPort! val lines = ssc.socketTextStream(serverIP, serverPort)! ! // split each line into words! val words = lines.flatMap(_.split(" "))! ! // count each word in each batch! val pairs = words.map(word => (word, 1))! val wordCounts = pairs.reduceByKey(_ + _)! ! // print a few of the counts to the console! wordCounts.print()! ! ssc.start()! ssc.awaitTermination() 31
- 32. Because Use Cases 32
- 33. Because Use Cases: +40 known production use cases 33
- 34. Because Use Cases: Analysis Reasons for adopting/transitioning to Spark Streaming… the unified programming model is particularly relevant for real-time analytics that combine historical data: • Making data science accessible to non-scientists • Higher productivity for data workers • Exactly-once semantics • No compromises on scalability and throughput • Ease of operations 34
- 35. Because Use Cases: Analysis Reasons for adopting Spark Streaming: • Making data science accessible to non-scientists Spark’s declarative APIs enable users who have domain expertise but lack data science expertise. In other words, express a business problem and its associated processing algorithm and data pipeline by using simple, high-level operators. 35
- 36. Because Use Cases: Analysis Reasons for adopting Spark Streaming: • Higher productivity for data workers Spark’s write-once-run-anywhere approach unifies batch and stream processing. That ties together the different components of an analytics pipeline in the same tool – discovery, ETL, data engineering, machine learning model training and execution – across all types of structured and unstructured data. 36
- 37. Because Use Cases: Analysis Reasons for adopting Spark Streaming: • Exactly-once semantics Many crucial use cases in business need exactly-once stateful processing Not at-most-once (which includes zero) or at-least-once (which includes duplicates) Exactly-once provides users certainty on questions such as the exact number of fraud cases, emergencies, or outages occurring within a time period 37
- 38. Because Use Cases: Analysis Reasons for adopting Spark Streaming: • No compromises on scalability and throughput Spark Streaming is designed for hyper-scale environments and combines statefulness and persistence with high throughput. 38
- 39. Because Use Cases: Analysis Reasons for adopting Spark Streaming: • Ease of operations Spark provides a unified run time across different processing engines. One physical cluster and one set of operational processes covers the full spectrum of use cases. 39
- 40. Because Use Cases: Sharethrough Sharethrough Uses Spark Streaming to Optimize Bidding in Real Time Russell Cardullo, Michael Ruggier 2014-03-25 databricks.com/blog/2014/03/25/ sharethrough-and-spark-streaming.html • the profile of a 24 x 7 streaming app is different than an hourly batch job… • take time to validate output against the input… • confirm that supporting objects are being serialized… • the output of your Spark Streaming job is only as reliable as the queue that feeds Spark… • monoids… 40
- 41. Because Use Cases: Viadeo Spark Streaming As Near Realtime ETL Djamel Zouaoui 2014-09-18 slideshare.net/DjamelZouaoui/spark-streaming • Spark Streaming is topology-free • workers and receivers are autonomous and independent • paired with Kafka, RabbitMQ • 8 machines / 120 cores • use case for recommender system • issues: how to handle lost data, serialization 41
- 42. Because Use Cases: Stratio Stratio Streaming: a new approach to Spark Streaming David Morales, Oscar Mendez 2014-06-30 spark-summit.org/2014/talk/stratio-streaming- a-new-approach-to-spark-streaming • Stratio Streaming is the union of a real-time messaging bus with a complex event processing engine using Spark Streaming • allows the creation of streams and queries on the fly • paired with Siddhi CEP engine and Apache Kafka • added global features to the engine such as auditing 42 and statistics
- 43. Because Use Cases: Ooyala Productionizing a 24/7 Spark Streaming service on YARN Issac Buenrostro, Arup Malakar 2014-06-30 spark-summit.org/2014/talk/ productionizing-a-247-spark-streaming-service- on-yarn • state-of-the-art ingestion pipeline, processing over two billion video events a day • how do you ensure 24/7 availability and fault tolerance? • what are the best practices for Spark Streaming and its integration with Kafka and YARN? • how do you monitor and instrument the various 43 stages of the pipeline?
- 44. Because Use Cases: Guavus Guavus Embeds Apache Spark into its Operational Intelligence Platform Deployed at the World’s Largest Telcos Eric Carr 2014-09-25 databricks.com/blog/2014/09/25/guavus-embeds-apache-spark-into- its-operational-intelligence-platform-deployed-at-the-worlds- largest-telcos.html • 4 of 5 top mobile network operators, 3 of 5 top Internet backbone providers, 80% MSOs in NorAm • analyzing 50% of US mobile data traffic, +2.5 PB/day • latency is critical for resolving operational issues before they cascade: 2.5 MM transactions per second • “analyze first” not “store first ask questions later” 44
- 45. Demos 45
- 46. Demos, as time permits: brand new Python support for Streaming in 1.2 github.com/apache/spark/tree/master/examples/src/main/ python/streaming Twitter Streaming Language Classifier databricks.gitbooks.io/databricks-spark-reference-applications/ content/twitter_classifier/README.html ! ! For more Spark learning resources online: databricks.com/spark-training-resources 46
- 47. Demo: PySpark Streaming Network Word Count import sys! from pyspark import SparkContext! from pyspark.streaming import StreamingContext! ! sc = SparkContext(appName="PyStreamNWC", master="local[*]")! ssc = StreamingContext(sc, Seconds(5))! ! lines = ssc.socketTextStream(sys.argv[1], int(sys.argv[2]))! ! counts = lines.flatMap(lambda line: line.split(" ")) ! .map(lambda word: (word, 1)) ! .reduceByKey(lambda a, b: a+b)! ! counts.pprint()! ! ssc.start()! ssc.awaitTermination() 47
- 48. Demo: PySpark Streaming Network Word Count - Stateful import sys! from pyspark import SparkContext! from pyspark.streaming import StreamingContext! ! def updateFunc (new_values, last_sum):! return sum(new_values) + (last_sum or 0)! ! sc = SparkContext(appName="PyStreamNWC", master="local[*]")! ssc = StreamingContext(sc, Seconds(5))! ssc.checkpoint("checkpoint")! ! lines = ssc.socketTextStream(sys.argv[1], int(sys.argv[2]))! ! counts = lines.flatMap(lambda line: line.split(" ")) ! .map(lambda word: (word, 1)) ! .updateStateByKey(updateFunc) ! .transform(lambda x: x.sortByKey())! ! counts.pprint()! ! ssc.start()! ssc.awaitTermination() 48
- 50. Spark Integrations: Discover Insights Clean Up Your Data Run Sophisticated Analytics Integrate With Many Other Systems Use Lots of Different Data Sources cloud-based notebooks… ETL… the Hadoop ecosystem… widespread use of PyData… advanced analytics in streaming… rich custom search… web apps for data APIs… low-latency + multi-tenancy… 50
- 51. Spark Integrations: Advanced analytics for streaming use cases Kafka + Spark + Cassandra datastax.com/documentation/datastax_enterprise/4.5/ datastax_enterprise/spark/sparkIntro.html http://helenaedelson.com/?p=991 github.com/datastax/spark-cassandra-connector github.com/dibbhatt/kafka-spark-consumer unified compute data streams columnar key-value 51
- 52. Spark Integrations: Rich search, immediate insights Spark + ElasticSearch databricks.com/blog/2014/06/27/application-spotlight-elasticsearch. html elasticsearch.org/guide/en/elasticsearch/hadoop/current/ spark.html spark-summit.org/2014/talk/streamlining-search-indexing- using-elastic-search-and-spark unified compute document search 52
- 53. Spark Integrations: General Guidelines • use Tachyon as a best practice for sharing between two streaming apps • or write to Cassandra or HBase / then read back • design patterns for integration: spark.apache.org/docs/latest/streaming-programming- guide.html#output-operations- on-dstreams 53
- 54. A Look Ahead… 54
- 55. A Look Ahead… 1. Greater Stability and Robustness • improved high availability via write-ahead logs • enabled as an optional feature for Spark 1.2 • NB: Spark Standalone can already restart driver • excellent discussion of fault-tolerance (2012): cs.duke.edu/~kmoses/cps516/dstream.html • stay tuned: meetup.com/spark-users/events/218108702/ 55
- 56. A Look Ahead… 2. Support for more environments, i.e., beyond Hadoop • three use cases currently depend on HDFS • those are being abstracted out • could then use Cassandra, etc. 56
- 57. A Look Ahead… 3. Improved support for Python • e.g., Kafka is not exposed through Python yet (next release goal) 57
- 58. A Look Ahead… 4. Better flow control • a somewhat longer-term goal, plus it is a hard problem in general • poses interesting challenges beyond what other streaming systems have faced 58
- 59. A Big Picture 59
- 60. A Big Picture… 19-20c. statistics emphasized defensibility in lieu of predictability, based on analytic variance and goodness-of-fit tests ! That approach inherently led toward a manner of computational thinking based on batch windows ! They missed a subtle point… 60
- 61. A Big Picture… The view in the lens has changed 21c. shift towards modeling based on probabilistic approximations: trade bounded errors for greatly reduced resource costs highlyscalable.wordpress.com/2012/05/01/ probabilistic-structures-web-analytics-data- mining/ 61
- 62. A Big Picture… The view in the lens has changed 21c. shift towards modeling based on probabil approximations: trade bounded errors for greatly reduced resource costs Twitter catch-phrase: “Hash, don’t sample” highlyscalable.wordpress.com/2012/05/01/ probabilistic-structures-web-analytics-data- mining/ 62
- 63. Probabilistic Data Structures: a fascinating and relatively new area, pioneered by relatively few people – e.g., Philippe Flajolet provides approximation, with error bounds – in general uses significantly less resources (RAM, CPU, etc.) many algorithms can be constructed from combinations of read and write monoids aggregate different ranges by composing hashes, instead of repeating full-queries 63
- 64. Probabilistic Data Structures: Some Examples algorithm use case example Count-Min Sketch frequency summaries code HyperLogLog set cardinality code Bloom Filter set membership MinHash set similarity DSQ streaming quantiles SkipList ordered sequence search 64
- 65. Probabilistic Data Structures: Some Examples algorithm use case example Count-Min Sketch frequency summaries code HyperLogLog set cardinality code suggestion: consider these as your most quintessential collections data types at scale Bloom Filter set membership MinHash set similarity DSQ streaming quantiles SkipList ordered sequence search 65
- 66. Add ALL the Things: Abstract Algebra Meets Analytics infoq.com/presentations/abstract-algebra-analytics Avi Bryant, Strange Loop (2013) • grouping doesn’t matter (associativity) • ordering doesn’t matter (commutativity) • zeros get ignored In other words, while partitioning data at scale is quite difficult, you can let the math allow your code to be flexible at scale Avi Bryant @avibryant Probabilistic Data Structures: Performance Bottlenecks 66
- 67. Probabilistic Data Structures: Industry Drivers • sketch algorithms: trade bounded errors for orders of magnitude less required resources, e.g., fit more complex apps in memory • multicore + large memory spaces (off heap) are increasing the resources per node in a cluster • containers allow for finer-grain allocation of cluster resources and multi-tenancy • monoids, etc.: guarantees of associativity within the code allow for more effective distributed computing, e.g., partial aggregates • less resources must be spent sorting/windowing data prior to working with a data set • real-time apps, which don’t have the luxury of anticipating data partitions, can respond quickly 67
- 68. Probabilistic Data Structures: Recommended Reading Probabilistic Data Structures for Web Analytics and Data Mining Ilya Katsov (2012-05-01) A collection of links for streaming algorithms and data structures Debasish Ghosh Aggregate Knowledge blog (now Neustar) Timon Karnezos, Matt Curcio, et al. Probabilistic Data Structures and Breaking Down Big Sequence Data C. Titus Brown, O'Reilly (2010-11-10) Algebird Avi Bryant, Oscar Boykin, et al. Twitter (2012) Mining of Massive Datasets Jure Leskovec, Anand Rajaraman, Jeff Ullman, Cambridge (2011) 68
- 69. Resources 69
- 70. cloud-based notebooks: databricks.com/blog/2014/07/14/databricks-cloud-making- big-data-easy.html youtube.com/watch?v=dJQ5lV5Tldw#t=883 70
- 71. certification: Apache Spark developer certificate program • http://oreilly.com/go/sparkcert • defined by Spark experts @Databricks • assessed by O’Reilly Media • establishes the bar for Spark expertise 71
- 72. Big Data Partnership • Postcode Anywhere: • Spark, Cassandra, Elasticsearch • And machine learning • Dynamically optimizing website UX • In house: • Spark, HBase, Elasticsearch • In a Lambda architecture • And data science • Social data mining for ad optimization • Training: • Introduction to Spark • Regular courses • Next: Dec 5th, Feb 6th • Databricks materials & certified trainers http://www.bigdatapartnership.com 72
- 73. community: spark.apache.org/community.html video+slide archives: spark-summit.org events worldwide: goo.gl/2YqJZK resources: databricks.com/spark-training-resources workshops: databricks.com/spark-training Intro to Spark Spark AppDev Spark DevOps Spark DataSci Distributed ML on Spark Streaming Apps on Spark Spark + Cassandra 73
- 74. books: Fast Data Processing with Spark Holden Karau Packt (2013) shop.oreilly.com/product/ 9781782167068.do Spark in Action Chris Fregly Manning (2015*) sparkinaction.com/ Learning Spark Holden Karau, Andy Konwinski, Matei Zaharia O’Reilly (2015*) shop.oreilly.com/product/ 0636920028512.do 74
- 75. events: Big Data Spain Madrid, Nov 17-18 bigdataspain.org Strata EU Barcelona, Nov 19-21 strataconf.com/strataeu2014 Data Day Texas Austin, Jan 10 datadaytexas.com Strata CA San Jose, Feb 18-20 strataconf.com/strata2015 Spark Summit East NYC, Mar 18-19 spark-summit.org/east Strata EU London, May 5-7 strataconf.com/big-data-conference-uk-2015 Spark Summit 2015 SF, Jun 15-17 spark-summit.org 75
- 76. presenter: monthly newsletter for updates, events, conf summaries, etc.: liber118.com/pxn/ Just Enough Math O’Reilly, 2014 justenoughmath.com preview: youtu.be/TQ58cWgdCpA Enterprise Data Workflows with Cascading O’Reilly, 2013 shop.oreilly.com/product/ 0636920028536.do 76