Custom Applications with Spark's RDD: Spark Summit East talk by Tejas Patil
5 likes1,627 views
The document discusses the development of custom applications using Spark's RDD, focusing on n-gram language model training for real-world applications such as auto-subtitling and content filtering. It highlights performance issues, scalability challenges, and lessons learned while transitioning from SQL to Spark solutions, emphasizing improvements in modularity and maintainability. The performance evaluation shows a significant increase in efficiency with Spark compared to the previous Hive solution.
![def findLargeShardIds(sc: SparkContext, threshold: Long, …..): Set[Int] = {
val shardSizesRDD = sc.textFile(shardCountsFile)
.map {
case line =>
val Array(indexStr, countStr) = line.split('t')
(indexStr.toInt, countStr.toLong)
}
val largeShardIds = shardSizesRDD.filter {
case (index, count) => count > threshold
}.map(_._1)
.collect().toSet
return largeShardIds
}](https://image.slidesharecdn.com/6ctejaspatil-170214194733/85/Custom-Applications-with-Spark-s-RDD-Spark-Summit-East-talk-by-Tejas-Patil-29-320.jpg)
![var iterationId = 0
do {
val currentCounts: RDD[(String, Long)] = allCounts(iterationId - 1)
val partitioner = new PartitionerForNgram(numShards, iterationId)
val shardCountsFile = s"${shard_sizes}_$iterationId"
currentCounts
.map(ngram => (partitioner.getPartition(ngram._1), 1L))
.reduceByKey(_ + _)
.saveAsTextFile(shardCountsFile)
largeShardIds = findLargeShardIds(sc, config.largeShardThreshold, shardCountsFile)
trainer.trainedModel (currentCounts, component, largeShardIds)
.saveAsObjectFile(s"${component.order}_$iterationId")
iterationId + 1
} while (largeShards.nonEmpty)](https://image.slidesharecdn.com/6ctejaspatil-170214194733/85/Custom-Applications-with-Spark-s-RDD-Spark-Summit-East-talk-by-Tejas-Patil-35-320.jpg)
![Upstream contributions to pipe()
• [SPARK-13793] PipedRDD doesn't propagate exceptions while
reading parent RDD
• [SPARK-15826] PipedRDD to allow configurable char encoding
• [SPARK-14542] PipeRDD should allow configurable buffer size for
the stdin writer
• [SPARK-14110] PipedRDD to print the command ran on non zero
exit](https://image.slidesharecdn.com/6ctejaspatil-170214194733/85/Custom-Applications-with-Spark-s-RDD-Spark-Summit-East-talk-by-Tejas-Patil-38-320.jpg)
Custom Applications with Spark's RDD: Spark Summit East talk by Tejas Patil
- 2. Agenda • Use case • Real world applications • Previous solution • Spark version • Data skew • Performance evaluation
- 7. Detecting low quality places • Non-public places • My home • Home sweet home • Non-real places • Apt #00, Fake lane, Foo City, CA • Mordor, Westeros !! • Non-suitable for watch • Anything containing nudity, intense sexuality, profanity or disturbing content
- 11. Lessons learned • SQL not good choice for building such applications • Duplication • Poor readability • Brittle, no testing • Alternatives • Map-reduce • Query templating • Latency while training with large data
- 12. Spark solution
- 13. Spark solution • Same high level architecture • Hive tables as final inputs and outputs • Same binaries used in Hive TRANSFORM • RDD not Datasets • `pipe()` operator • Modular, readable, maintainable
- 14. Configuration PipelineConfiguration - where is the input data ? - where to store final output ? - spark specific configs: "spark.dynamicAllocation.maxExecutors” "spark.executor.memory” "spark.memory.storageFraction” ………… - list of ComponentConfiguration ……
- 15. Scalability challenges • Executors lost as unable to heartbeat • Shuffle service OOM • Frequent executor GC • Executor OOM • 2GB limit in Spark for blocks • Exceptions while reading output stream of pipe process
- 16. Scalability challenges • Executors lost as unable to heartbeat • Shuffle service OOM • Frequent executor GC • Executor OOM • 2GB limit in Spark for blocks • Exceptions while reading output stream of pipe process
- 17. Data skew
- 29. def findLargeShardIds(sc: SparkContext, threshold: Long, …..): Set[Int] = { val shardSizesRDD = sc.textFile(shardCountsFile) .map { case line => val Array(indexStr, countStr) = line.split('t') (indexStr.toInt, countStr.toLong) } val largeShardIds = shardSizesRDD.filter { case (index, count) => count > threshold }.map(_._1) .collect().toSet return largeShardIds }
- 35. var iterationId = 0 do { val currentCounts: RDD[(String, Long)] = allCounts(iterationId - 1) val partitioner = new PartitionerForNgram(numShards, iterationId) val shardCountsFile = s"${shard_sizes}_$iterationId" currentCounts .map(ngram => (partitioner.getPartition(ngram._1), 1L)) .reduceByKey(_ + _) .saveAsTextFile(shardCountsFile) largeShardIds = findLargeShardIds(sc, config.largeShardThreshold, shardCountsFile) trainer.trainedModel (currentCounts, component, largeShardIds) .saveAsObjectFile(s"${component.order}_$iterationId") iterationId + 1 } while (largeShards.nonEmpty)
- 38. Upstream contributions to pipe() • [SPARK-13793] PipedRDD doesn't propagate exceptions while reading parent RDD • [SPARK-15826] PipedRDD to allow configurable char encoding • [SPARK-14542] PipeRDD should allow configurable buffer size for the stdin writer • [SPARK-14110] PipedRDD to print the command ran on non zero exit
- 39. Questions ?