Big Data Beyond the JVM - Strata San Jose 2018
3 likes705 views
The document discusses accelerating big data processing beyond just the Java Virtual Machine (JVM). It introduces Rachel Warren and Holden Karau, the presenters. It then covers the current state of PySpark and its performance limitations due to serialization between Python and the JVM. Future improvements discussed include using Apache Arrow to accelerate UDFs, Dask for pure Python processing, and Apache Beam for additional languages. The presenters promote their new book on high performance Spark and take questions at the end.
dataIn.readFully(result)
// Assume UTF8, what could go wrong? :p
new String(result)
}
From https://github.com/holdenk/kafka-streams-python-cthulhu](https://image.slidesharecdn.com/bigdatabeyondthejvm-stratasj2018-180309221042/85/Big-Data-Beyond-the-JVM-Strata-San-Jose-2018-43-320.jpg)
Big Data Beyond the JVM - Strata San Jose 2018
- 1. Accelerating Big Data Beyond the JVM Engine noises go here @holdenkarau & @warre_n_peace
- 2. Rachel - Rachel Warren โ She/ Her - Data Scientist / Software engineer at Salesforce Einstein - Formerly at Alpine Data (with Holden) - Lots of experience scaling Spark in different production environments - The other half of the High Performance Spark team :) - @warre_n_peace - Linked in: https://www.linkedin.com/in/rachelbwarren/ - Slideshare: https://www.slideshare.net/RachelWarren4/ - Github: https://github.com/rachelwarren
- 3. Holden: โ My name is Holden Karau โ Prefered pronouns are she/her โ Developer Advocate at Google โ Apache Spark PMC :) โ previously IBM, Alpine, Databricks, Google, Foursquare & Amazon โ co-author of Learning Spark & High Performance Spark โ @holdenkarau โ Slide share http://www.slideshare.net/hkarau โ Linkedin https://www.linkedin.com/in/holdenkarau โ Github https://github.com/holdenk โ Spark Videos http://bit.ly/holdenSparkVideos โ Talk feedback: http://bit.ly/holdenTalkFeedback http://bit.ly/holdenTalkFeedback
- 5. Who I think you wonderful humans are? โ Nice enough people โ Iโm sure you love pictures of cats โ Might know some the different distributed systems talked about โ Possibly know some Python or R โ Trying to learn Python to become a deep learning startup โ Or are just tired of scala/ java Lori Erickson
- 6. What will be covered? โ A more detailed look at the current state of PySpark โ Why it isnโt good enough and plans for the future โ Using Arrow for fast Python UDFS with Spark โ Dask โ Beam Outside the JVM โ Our even less subtle attempts to get you to buy our new book โ Pictures of cats & stuffed animals โ tl;dr - Weโve* made some bad** choices historically, and projects like Arrow & friends can save us from some of these (yay!)
- 7. Whatโs the state of non-JVM big data? Most of the tools are built in the JVM, so how do we play together? โ Pickling, Strings, JSON, XML, oh my! โ Unix pipes โ Sockets What about if we donโt want to copy the data all the time? Dataframe Api + Arrow โ Or standalone โpureโ* re-implementations of everything โ Reasonable option for things like Kafka where you would have the I/O regardless. โ Also cool projects like dask (pure python) -- but hard to talk to existing ecosystem David Brown
- 8. PySpark: โ The Python interface to Spark โ Fairly mature, integrates well-ish into the ecosystem, less a Pythonrific API โ Has some serious performance hurdles from the design โ Same general technique used as the bases for the other non JVM implementations in Spark โ C# โ R โ Julia โ Javascript - surprisingly different
- 9. Yes, we have wordcount! :p lines = sc.textFile(src) words = lines.flatMap(lambda x: x.split(" ")) word_count = (words.map(lambda x: (x, 1)) .reduceByKey(lambda x, y: x+y)) word_count.saveAsTextFile(output) No data is read or processed until after this line This is an โactionโ which forces spark to evaluate the RDD These are still combined and executed in one python executor Trish Hamme
- 10. A quick detour into PySparkโs internals + + JSON
- 11. Spark in Scala, how does PySpark work? โ Py4J + pickling + JSON and magic โ Py4j in the driver โ Pipes to start python process from java exec โ cloudPickle to serialize data between JVM and python executors (transmitted via sockets) โ Json for dataframe schema โ Data from Spark worker serialized and piped to Python worker --> then piped back to jvm โ Multiple iterator-to-iterator transformations are still pipelined :) โ So serialization happens only once per stage โ Spark SQL (and DataFrames) avoid some of this kristin klein
- 12. So what does that look like? Driver py4j Worker 1 Worker K pipe pipe
- 13. So how does that impact PySpark? โ Double serialization cost makes everything more expensive โ Python worker startup takes a bit of extra time โ Python memory isnโt controlled by the JVM - easy to go over container limits if deploying on YARN or similar โ Error messages make ~0 sense โ Spark Features arenโt automatically exposed, but exposing them is normally simple
- 14. Our saviour from serialization: DataFrames โ For the most part keeps data in the JVM โ Notable exception is UDFs written in Python โ Takes our python calls and turns it into a query plan โ if we need more than the native operations in Sparkโs DataFrames we end up in a pickle ;) โ be wary of Distributed Systems bringing claims of usabilityโฆ. Andy Blackledge
- 15. So what are Spark DataFrames? โ More than SQL tables โ Not Pandas or R DataFrames โ Semi-structured (have schema information) โ tabular โ work on expression as well as lambdas โ e.g. df.filter(df.col(โhappyโ) == true) instead of rdd.filter(lambda x: x.happy == true)) โ Not a subset of Spark โDatasetsโ - since Dataset API isnโt exposed in Python yet :( Quinn Dombrowski
- 16. Word count w/Dataframes df = sqlCtx.read.load(src) # Returns an RDD words = df.select("text").flatMap(lambda x: x.text.split(" ")) words_df = words.map( lambda x: Row(word=x, cnt=1)).toDF() word_count = words_df.groupBy("word").sum() word_count.write.format("parquet").save("wc.parquet") Still have the double serialization here :(
- 17. We can see the difference easily: Andrew Skudder * *Vendor benchmark. Trust but verify.
- 18. *For a small price of your fun libraries. Bad idea.
- 19. That was a bad idea, buuutโฆ.. โ Work going on in Scala land to translate simple Scala into SQL expressions - need the Dataset API โ Maybe we can try similar approaches with Python? โ POC use Jython for simple UDFs (e.g. 2.7 compat & no native libraries) - SPARK-15369 โ Early benchmarking w/word count 5% slower than native Scala UDF, close to 2x faster than regular Python โ Willing to share your Python UDFs for benchmarking? - http://bit.ly/pySparkUDF *The future may or may not have better performance than today. But bun-bun the bunny has some lettuce so its ok!
- 21. The โfutureโ*: faster interchange โ By future I mean availability starting in the next 3-6 months (with more improvements after). โ Yes much of this code exists, it just isnโt released yet so Iโm sure weโll find all sorts of bugs and ways to improve. โ Relatedly you can help us test in Spark 2.3 when we start the RC process to catch bug early! โ Unifying our cross-language experience โ And not just โnormalโ languages, CUDA counts yo Tambako The Jaguar
- 22. The present!!!!!!!!! โ Arrow support for UDF acceleration is in the latest Spark release โ Itโs not finished (e.g. UDAFs still make sadness) Ren Kuo
- 23. Andrew Skudder *Arrow: likely the future. I really hope so. Spark 2.3 and beyond! * *
- 24. What does the future look like?* *Source: https://databricks.com/blog/2017/10/30/introducing-vectorized-udfs-for-pyspark.html. *Vendor benchmark. Trust but verify.
- 25. What does the future look like - in code @pandas_udf("integer", PandasUDFType.SCALAR) def add_one(x): return x + 1
- 26. What does the future look like - in code @pandas_udf("id long, v double", PandasUDFType.GROUPED_MAP) def normalize(pdf): v = pdf.v return pdf.assign(v=(v - v.mean()) / v.std())
- 27. What does the future look like - in code @pandas_udf("word string", PandasUDFType.GROUPED_MAP) def special_tokenize(s): if s.strings is not None: return pandas.DataFrame(reduce(list.__add__, map(lambda x: x.split(' '), s.strings))) # This is a little ugly because currently the scalar transform # doesn't allow flat map like behaviour only map like. grouped = df.groupby("strings") tokenized = grouped.apply(special_tokenize) tokenized.show() Word Count #3 $$$$$
- 28. Beyond wordcount: depencies? โ Your machines probably already have pandas โ But maybe an old version โ But they might not have โspecial_business_logicโ โ Very special business logic, no one wants change fortran code*. โ Option 1: Talk to your vendor** โ Option 2: Try some sketchy open source software from a hack day โ Weโre going to focus on option 2! *Because itโs perfect, it is fortran after all. ** I donโt like this option because the vendor I work for doesnโt have an answer.
- 29. coffee_boat to the rescue* # You can tell it's alpha cause were installing from github !pip install --upgrade git+https://github.com/nteract/coffee_boat.git # Use the coffee boat from coffee_boat import Captain captain = Captain(accept_conda_license=True) captain.add_pip_packages("pyarrow", "edtf") captain.launch_ship() sc = SparkContext(master="yarn") # You can now use pyarrow & edtf captain.add_pip_packages("yourmagic") # You can now use your magic in transformations!
- 30. Hadoop โstreamingโ (Python/R) โ Unix pipes! โ Involves a data copy, formats get sad โ But the overhead of a Map/Reduce task is pretty high anyways... Lisa Larsson
- 31. Kafka: re-implement all the things โ Multiple options for connecting to Kafka from outside of the JVM (yay!) โ They implement the protocol to talk to Kafka (yay!) โ This involves duplicated client work, and sometimes the clients can be slow (solution, FFI bindings to C instead of Java) โ Buuuut -- we canโt access all of the cool Kafka business (like Kafka Streams) and features depend on client libraries implementing them (easy to slip below parity) Smokey Combs
- 32. Dask: a new beginning? โ Pure* python implementation โ Provides real enough DataFrame interface for distributed data โ Also your standard-ish distributed collections โ Multiple backends โ Primary challenge: interacting with the rest of the big data ecosystem โ Arrow & friends might make this better with time too, buuutโฆ. โ See https://dask.pydata.org/en/latest/ & http://dask.pydata.org/en/latest/spark.html Lisa Zins
- 33. BEAM Beyond the JVM โ Non JVM BEAM doesnโt work outside of Googleโs environment yet, so Iโm going to skip the details. โ tl;dr : uses grpc / protobuf โ But exciting new plans to unify the runners and ease the support of different languages (called SDKS) โ See https://beam.apache.org/contribute/portability/ โ If this is exciting, you can come join me on making BEAM work in Python3 โ Yes we still donโt have that :( โ But we're getting closer!
- 34. Why now? โ Thereโs been better formats/options for a long time โ JVM devs want to use libraries in other languages with lots of data โ e.g. startup + Deep Learning + ? => profit โ Arrow has solved the chicken-egg problem by building not just the chicken & the egg, but also a hen house Andrew Mager
- 35. References โ Apache Arrow: https://arrow.apache.org/ โ Brian (IBM) on initial Spark + Arrow https://arrow.apache.org/blog/2017/07/26/spark-arrow/ โ Li Jin (two sigma) https://databricks.com/blog/2017/10/30/introducing-vectorized-udfs-for-pyspar k.html โ Bill Maimone https://blogs.nvidia.com/blog/2017/06/27/gpu-computation-visualization/
- 36. Learning Spark Fast Data Processing with Spark (Out of Date) Fast Data Processing with Spark (2nd edition) Advanced Analytics with Spark Spark in Action High Performance SparkLearning PySpark
- 37. High Performance Spark! You can buy it today! Or come to our book signing at 3:20 and maybe get a free copy. Only one chapter on non-JVM stuff, Iโm sorry. Cats love it* *Or at least the box it comes in. If buying for a cat, get print rather than e-book.
- 38. And some upcoming talks: โ March โ PyData Ann Arbor March 13 - I just booked my flight 4 hours ago! โ April โ Flink Forward โ Dataworks Summit Berlin โ Kafka Summit London โ PyData London โ May โ Stata London: https://conferences.oreilly.com/strata/strata-eu/public/schedule/detail/6 4759
- 39. k thnx bye :) If you care about Spark testing and donโt hate surveys: http://bit.ly/holdenTestingSpark I need to give a testing talk in a few months, help a โfriendโ out. Will tweet results โeventuallyโ @holdenkarau Do you want more realistic benchmarks? Share your UDFs! http://bit.ly/pySparkUDF Itโs performance review season, so help a friend out and fill out this survey with your talk feedback http://bit.ly/holdenTalkFeedback
- 40. Bonus Slides Maybe you ask a question and we go here :)
- 41. We can do that w/Kafka streams.. โ Why bother learning from our mistakes? โ Or more seriously, the mistakes werenโt that bad...
- 42. Our โspecialโ business logic def transform(input): """ Transforms the supplied input. """ return str(len(input)) Pargon
- 43. Letโs pretend all the world is a string: override def transform(value: String): String = { // WARNING: This may summon cuthuluhu dataOut.writeInt(value.getBytes.size) dataOut.write(value.getBytes) dataOut.flush() val resultSize = dataIn.readInt() val result = new Array[Byte](resultSize) dataIn.readFully(result) // Assume UTF8, what could go wrong? :p new String(result) } From https://github.com/holdenk/kafka-streams-python-cthulhu
- 44. Then make an instance to use it... val testFuncFile = "kafka_streams_python_cthulhu/strlen.py" stream.transformValues( PythonStringValueTransformerSupplier(testFuncFile)) // Or we could wrap this in the bridge but thats effort. From https://github.com/holdenk/kafka-streams-python-cthulhu
- 45. Letโs pretend all the world is a string: def main(socket): while (True): input_length = _read_int(socket) data = socket.read(input_length) result = transform(data) resultBytes = result.encode() _write_int(len(resultBytes), socket) socket.write(resultBytes) socket.flush() From https://github.com/holdenk/kafka-streams-python-cthulhu
- 46. What does that let us do? โ You can add a map stage with your data scientists Python code in the middle โ Youโre limited to strings* โ Still missing the โdriver sideโ integration (e.g. the interface requires someone to make a Scala class at some point)
- 47. What about things other than strings? Use another system โ Like Spark! (oh wait) or BEAM* or FLINK*? Write it in a format Python can understand: โ Pickling (from Java) โ JSON โ XML Purely Python solutions โ Currently roll-your-own (but not that bad) *These are also JVM based solutions calling into Python. Iโm not saying they will also summon Cuthulhu, Iโm just saying hang onto your souls.