Scio
1 like908 views
Scio is a Scala API for Google Cloud Dataflow that provides a simplified wrapper compared to native Dataflow APIs. It allows Spotify to process large datasets for tasks like personalized music recommendations using a functional programming style. Scio handles tasks like computing word counts and PageRank on Dataflow and is used by Spotify to generate weekly recommendations from 100GB of data and analyze user conversion patterns from 150GB datasets. The goal of Scio is to make Dataflow more usable and scalable for data processing while maintaining simplicity over optimization.
![Scio
Ecclesiastical Latin IPA: /ˈʃi.o/, [ˈʃiː.o], [ˈʃi.i̯o]
Verb: I can, know, understand, have knowledge.](https://image.slidesharecdn.com/scio-160727144212/85/Scio-16-320.jpg)
![WordCount
Almost identical to Spark version
val sc = ScioContext()
sc.textFile("shakespeare.txt")
.flatMap(_.split("[^a-zA-Z']+").filter(_.nonEmpty))
.countByValue()
.saveAsTextFile("wordcount.txt")](https://image.slidesharecdn.com/scio-160727144212/85/Scio-18-320.jpg)
![PageRank
def pageRank(in: SCollection[(String, String)]) = {
val links = in.groupByKey()
var ranks = links.mapValues(_ => 1.0)
for (i <- 1 to 10) {
val contribs = links.join(ranks).values
.flatMap { case (urls, rank) =>
val size = urls.size
urls.map((_, rank / size))
}
ranks = contribs.sumByKey.mapValues((1 - 0.85) + 0.85 * _)
}
ranks
}](https://image.slidesharecdn.com/scio-160727144212/85/Scio-19-320.jpg)
![Design and Implementation
• Simplicity over premature optimization
• Usability over Python/Java inter-op
• Ser/de: ☑kryo/chill ☒Coder[T]
• Closure cleaner](https://image.slidesharecdn.com/scio-160727144212/85/Scio-28-320.jpg)
Scio
- 1. Scio A Scala API for Google Cloud Dataflow Neville Li @sinisa_lyh
- 2. Who am I?
- 3. Origin Story Scalding and Spark ML, recommendations, analytics 50+ users, 400+ unique jobs
- 4. Moving to Google Cloud Early 2015 - Dataflow Scala hack project
- 6. Data model Spark • RDD for batch, DStream for streaming • Explicit caching semantics • Two sets ofAPIs Dataflow • PCollection for both batch and streaming • Windowed and timestamped values • One unifiedAPI
- 7. Execution Spark • Driver and executors • Dynamic execution from driver • Transforms and actions Dataflow • No master • Static execution planning • Transforms only, no actions
- 9. Why not Scalding on GCE Pros • Community Twitter, eBay, Etsy, Stripe, LinkedIn, … • Stable and proven
- 10. Why not Scalding on GCE Cons • Hadoop cluster operations • Multi-tenancy resource contention and utilization • No streaming mode (Summingbird?)
- 11. Why not Spark on GCE Pros • Batch, streaming, interactive and SQL • MLlib, GraphX • Scala, Python, and R support • Zeppelin, spark-notebook, Hue
- 12. Why not Spark on GCE Cons • Hard to tune and scale • Cluster lifecycle management
- 13. Why Dataflow with Scala Dataflow • Hosted solution, no operations • Ecosystem GCS, BigQuery, PubSub, Bigtable, … • Unified batch and streaming model
- 14. Why Dataflow with Scala Scala • High level DSL easytransition for developers • Reusable and composable code via FP • Numerical libraries: Breeze,Algebird
- 16. Scio Ecclesiastical Latin IPA: /ˈʃi.o/, [ˈʃiː.o], [ˈʃi.i̯o] Verb: I can, know, understand, have knowledge.
- 18. WordCount Almost identical to Spark version val sc = ScioContext() sc.textFile("shakespeare.txt") .flatMap(_.split("[^a-zA-Z']+").filter(_.nonEmpty)) .countByValue() .saveAsTextFile("wordcount.txt")
- 19. PageRank def pageRank(in: SCollection[(String, String)]) = { val links = in.groupByKey() var ranks = links.mapValues(_ => 1.0) for (i <- 1 to 10) { val contribs = links.join(ranks).values .flatMap { case (urls, rank) => val size = urls.size urls.map((_, rank / size)) } ranks = contribs.sumByKey.mapValues((1 - 0.85) + 0.85 * _) } ranks }
- 20. Spotify Running 60 million tracks 30m users * 10 tempo buckets * 25 tracks Audio: tempo, energy, time signature ... Metadata: genres, categories, … Latent vectors from collaborative filtering
- 25. Personalized new releases • Pre-computed weekly on Hadoop (on-premise cluster) • 100GB recommendations from HDFS to Bigtable in US+EU • 250GB Bloom filters from Bigtable to HDFS • 200 LOC
- 26. User conversion analysis • For marketing and campaigning strategies • Track usertransitions through products • Aggregated for simulation and projection • 150GB BigQuery in and out
- 27. Demo Time!
- 28. Design and Implementation • Simplicity over premature optimization • Usability over Python/Java inter-op • Ser/de: ☑kryo/chill ☒Coder[T] • Closure cleaner
- 29. What’s next? • Apache Beam donation • Migrating internal teams • BigQuery SQL-2011 dialect • Better streaming support • PRs and issues welcome!