New Directions in pySpark for Time Series Analysis: Spark Summit East talk by David Palaitis
8 likes3,963 views
The document provides an overview of using PySpark for time series analysis. It discusses that time series data can come from sources like IOT feeds, sensor data, and economic indicators. Time series analysis in PySpark allows for windowed aggregations and temporal joins on massive time series datasets that can be both wide and narrow. While basic analytics are possible in PySpark, libraries like Flint provide additional functions specialized for time series analysis on large datasets in a distributed environment. The document encourages attendees to speak with the author after the talk to see a time series analysis library in PySpark demonstrated.
![w is a window specification e.g. 500ms, 5s, 3 business days
RDD[(K,V)] -> RDD[(K,Seq[V])]](https://image.slidesharecdn.com/1adavidpalaitis-170214185024/85/New-Directions-in-pySpark-for-Time-Series-Analysis-Spark-Summit-East-talk-by-David-Palaitis-40-320.jpg)
![reduceByWindow(f: (V, V) => V, w): RDD[(K, W)] => RDD[(K, V)]](https://image.slidesharecdn.com/1adavidpalaitis-170214185024/85/New-Directions-in-pySpark-for-Time-Series-Analysis-Spark-Summit-East-talk-by-David-Palaitis-41-320.jpg)
![reduceByWindow(f: (V, V) => V, w): RDD[(K, V)] => RDD[(K, V)]](https://image.slidesharecdn.com/1adavidpalaitis-170214185024/85/New-Directions-in-pySpark-for-Time-Series-Analysis-Spark-Summit-East-talk-by-David-Palaitis-42-320.jpg)
New Directions in pySpark for Time Series Analysis: Spark Summit East talk by David Palaitis
- 1. PySpark for Time Series Analysis David Palaitis Two Sigma Investments
- 2. About Me
- 3. Important Legal Information The information presented here is offered for recruiting purposes only and should not be used for any other purpose (including, without limitation, the making of investment decisions). Examples provided herein are for illustrative purposes only and are not necessarily based on actual data. Nothing herein constitutes an offer to sell or the solicitation of any offer to buy any security or other interest. We consider this information to be confidential and not for redistribution or dissemination. Some of the images, logos or other material used herein may be protected by copyright and/or trademark. If so, such copyrights and/or trademarks are most likely owned by the entity that created the material and are used purely for identification and comment as fair use under international copyright and/or trademark laws. Use of such image, copyright or trademark does not imply any association with such organization (or endorsement of such organization) by Two Sigma, nor vice versa.
- 5. Time Series IOT feeds sensor data economic data An ordered sequence of values of a variable
- 9. Time Series at Two Sigma Millions of Time Series Big and Small (1GB – 1PB) Narrow (10 columns) and Wide (1MM Columns) Evenly and Unevenly Spaced Observations
- 11. Let’s start from the beginning …
- 21. Examples!
- 22. What’s Missing?
- 23. You can’t even do “Word Count”
- 32. “Word Count” !
- 40. w is a window specification e.g. 500ms, 5s, 3 business days RDD[(K,V)] -> RDD[(K,Seq[V])]
- 41. reduceByWindow(f: (V, V) => V, w): RDD[(K, W)] => RDD[(K, V)]
- 42. reduceByWindow(f: (V, V) => V, w): RDD[(K, V)] => RDD[(K, V)]
- 52. Looking ahead.
- 53. Thank You. Find me after the talk to see Flint in action.