Using Pluggable Apache Spark SQL Filters to Help GridPocket Users Keep Up with the Jones' (and save the planet) Paula Ta-Shma and Guy Gerson
2 likes794 views
The document discusses the use of pluggable Apache Spark SQL filters by GridPocket to enhance energy management and consumer engagement in smart grid applications. It emphasizes the importance of peer comparison in influencing energy-saving behaviors and details a dataset management solution involving object storage and metadata indexing. The proposed methods aim to improve query performance and reduce data transfer costs in Spark SQL operations, specifically for geospatial and other types of data.
![© 2017 IBM Corporation28
The Interface
Filters should extend this trait and implement application specific filtering logic.
trait ExecutionFileFilter {
/**
* @param dataFilters query predicates for data columns
* @param f represents an object which exists in the file catalog
* @return true if the object needs to be scanned during execution
*/
def isRequired(dataFilters: Seq[Filter], f: FileStatus) : Boolean
}
Turn filter on by specifying Filter class:
sqlContext.setConf("spark.sql.execution.fileFilter", “TestFileFilter”)
< 20 lines of code to integrate into Spark
#EUres2](https://image.slidesharecdn.com/l22tashmagerson-171031222334/85/Using-Pluggable-Apache-Spark-SQL-Filters-to-Help-GridPocket-Users-Keep-Up-with-the-Jones-and-save-the-planet-Paula-Ta-Shma-and-Guy-Gerson-28-320.jpg)
![© 2017 IBM Corporation42
Example Filter
class LiveSensorFilter extends ExecutionFileFilter {
//get a list of live sensors from an external Service
val activeSensors = SensorService.getLiveSensors
//returns true if object represents a live sensor
@Override
def isRequired(dataFilters:Seq[org.apache.spark.sql.sources.Filter],
fileStatus: FileStatus) : Boolean = {
activeSensors.contains(Utils.getSensorIdFromPath(fileStatus))
}
Turn filter on:
sqlContext.setConf("spark.sql.execution.fileFilter", “LiveSensorFilter ”)
#EUres2](https://image.slidesharecdn.com/l22tashmagerson-171031222334/85/Using-Pluggable-Apache-Spark-SQL-Filters-to-Help-GridPocket-Users-Keep-Up-with-the-Jones-and-save-the-planet-Paula-Ta-Shma-and-Guy-Gerson-42-320.jpg)
Using Pluggable Apache Spark SQL Filters to Help GridPocket Users Keep Up with the Jones' (and save the planet) Paula Ta-Shma and Guy Gerson
- 1. © 2017 IBM Corporation Using Pluggable Apache Spark SQL Filters to help GridPocket users keep up with the Jones’ (and save the planet) Paula Ta-Shma, Guy Gerson IBM Research Contact: paula@il.ibm.com Joint work with: Tal Ariel, IBM Filip Gluszak, GridPocket Gal Lushi, IBM Yosef Moatti, IBM Papa Niamadio, GridPocket Nathaël Noguès, GridPocket
- 2. © 2017 IBM Corporation2 You have 5 minutes to pack your bags…
- 3. © 2017 IBM Corporation3 ?
- 4. © 2017 IBM Corporation4 ?
- 5. © 2017 IBM Corporation Using Pluggable Apache Spark SQL Filters to help GridPocket users keep up with the Jones’ (and save the planet) Paula Ta-Shma, Guy Gerson IBM Research Contact: paula@il.ibm.com Joint work with: Tal Ariel, IBM Filip Gluszak, GridPocket Gal Lushi, IBM Yosef Moatti, IBM Papa Niamadio, GridPocket Nathaël Noguès, GridPocket
- 6. © 2017 IBM Corporation6 What would convince you to reduce your energy usage ? #EUres2
- 7. © 2017 IBM Corporation7 What would convince you to reduce your energy usage ? § Save $ ? #EUres2
- 8. © 2017 IBM Corporation8 What would convince you to reduce your energy usage ? § Save $ ? § Save the planet ? #EUres2
- 9. © 2017 IBM Corporation9 Competing to Save Energy § Research shows that utility customers are most influenced by peer pressure to save energy § => Help utilities compare their customers’ energy consumption with that of their neighbours – Queries are anonymized #EUres2
- 10. © 2017 IBM Corporation10 GridPocket § A smart grid company developing energy management applications and cloud services for electricity, water and gas utilities § HQ based in France § http://www.gridpocket.com/ § Developed open source data generator § Provided industry specific use cases and algorithms The GridPocket Dataset § Our target: 1 million meters reporting every 15 minutes § Records are ~100 bytes § Generated ~1TB in 3 months § Allowing for 1 order of magnitude growth in each dimension gives 1PB in 3 months § => Use object storage § Typically at least order of magnitude lower storage cost than NoSQL databases § NoSQL dataset – one large denormalized table containing meter reading information – SQL query for nearest neighbours #EUres2
- 11. © 2017 IBM Corporation11 What is Object Storage ? § Objects contain data and metadata § Written once and not modified – Cannot append or update data - Can overwrite completely - Can update metadata – No rename operation § Accessed through RESTful HTTP – PUT/GET/POST/DELETE object/bucket – Flat namespace § High capacity, low cost § Storage of choice for Big datasets – Analytics works best on equally sized objects § Examples – Amazon S3, IBM COS, Google Cloud Storage, OpenStack Swift Swift
- 12. © 2017 IBM Corporation12 Object Storage and Spark are separately managed microservices
- 13. © 2017 IBM Corporation13 Object Storage and Spark are separately managed microservices Want to query data on object storage directly
- 14. © 2017 IBM Corporation14 Object Storage and Spark are separately managed microservices Want to query data on object storage directly
- 15. © 2017 IBM Corporation15 Object Storage and Spark are separately managed microservices Want to query data on object storage directly Our goal:Minimize 1. Number of bytes shipped 2. Number of REST requests THE key factors affecting cost (and performance)
- 16. © 2017 IBM Corporation16 How is this done today ? 1. Use specialized column based formats such as Parquet, ORC – Column wise compression – Column pruning – Specialized metadata
- 17. © 2017 IBM Corporation17 How is this done today ? 1. Use specialized column based formats such as Parquet, ORC – Column wise compression – Column pruning – Specialized metadata We want one solution for all data formats
- 18. © 2017 IBM Corporation18 How is this done today ? 1. Use specialized column based formats such as Parquet, ORC – Column wise compression – Column pruning – Specialized metadata We want one solution for all data formats 2. Use Hive Style Partitioning to layout Object Storage data – Partition pruning
- 19. © 2017 IBM Corporation19 Hive Style Partitioning and Partition Pruning § Relies on a dataset naming convention § Object storage has flat namespace, but can create a virtual folder hierarchy – Use ‘/’ in object name § Data can be partitioned according to a column e.g. dt – Information about object contents is encoded in object name e.g. dt=2015-09-14 § Spark SQL can query fields in object name as well as in data e.g. “dt” – Filters the objects which need to be read from Object Storage and sent to Spark
- 20. © 2017 IBM Corporation20 Limitations of Today’s Hive Style Partitioning in Spark § Only one hierarchy is possible – Like database primary key, no secondary keys § Changing partitioning scheme requires rewriting entire dataset ! – Hierarchy cannot be changed without renaming all objects § No range partitioning – Only supports partitioning with discrete types e.g. Gender:M/F, date, age etc. – doesn't work well for timestamps, arbitrary floating point numbers etc. § A deep hierarchy may result in small and non uniform object sizes – May reduce performance
- 21. © 2017 IBM Corporation21 Can we do more ? NY
- 22. © 2017 IBM Corporation22 Can we do more ? § Generate metadata per object column and index it § Various index types – Min/max, bounding boxes, value lists § Filter objects according to this metadata § Applies to all formats e.g. json, csv NY
- 23. © 2017 IBM Corporation23 Can we do more ? § Generate metadata per object column and index it § Various index types – Min/max, bounding boxes, value lists § Filter objects according to this metadata § Applies to all formats e.g. json, csv NY Min/max values
- 24. © 2017 IBM Corporation24 Can we do more ? § Generate metadata per object column and index it § Various index types – Min/max, bounding boxes, value lists § Filter objects according to this metadata § Applies to all formats e.g. json, csv NY One or more bounding boxes
- 25. © 2017 IBM Corporation25 Can we do more ? § Generate metadata per object column and index it § Various index types – Min/max, bounding boxes, value lists § Filter objects according to this metadata § Applies to all formats e.g. json, csv NY A set of values
- 26. © 2017 IBM Corporation26 Filter According to Metadata § Store one metadata record per object – Unlike database fully inverted index § Various index types – Min and max value for ordered columns – Bounding boxes for geospatial data – Bloom filters as space efficient value lists § Users can choose which columns to index and index type per column – Can index additional columns later on § Main requirement: no false negatives § Avoids touching irrelevant objects altogether § Handles updates (PUT/DELETE object) – Filtering out irrelevant objects always works #EUres2 NY
- 27. © 2017 IBM Corporation27 Spark SQL query execution flow Query Prune partitions Read data Query Prune partitions Optional file filter Read data Metadata Filter
- 28. © 2017 IBM Corporation28 The Interface Filters should extend this trait and implement application specific filtering logic. trait ExecutionFileFilter { /** * @param dataFilters query predicates for data columns * @param f represents an object which exists in the file catalog * @return true if the object needs to be scanned during execution */ def isRequired(dataFilters: Seq[Filter], f: FileStatus) : Boolean } Turn filter on by specifying Filter class: sqlContext.setConf("spark.sql.execution.fileFilter", “TestFileFilter”) < 20 lines of code to integrate into Spark #EUres2
- 29. © 2017 IBM Corporation29 Data Ingest: How to best Partition the Data § Organizing data in objects in a smart way generates more effective metadata § Partition during data ingestion § Example: Kafka allows user to define partitioning function § Needs to scale horizontally – want stateless partitioner #EUres2 Client 1 Partition 0 Client 2 Client 3 Partition 1 Partition 2 Partition 3
- 30. © 2017 IBM Corporation30 Geospatial Partitioning: Grid Partitioner § Divide the world map into a grid – Precision depends on use case § Each data point belongs to a cell § Each cell is hashed to a partition § A partition can contain multiple cells § Partitions periodically generate objects § Each object is described using a list of bounding boxes – 1 per participating cell #EUres2 1 2 3 4 5 6 7 8 Partition 0 Partition 1 Partition 2 8 2 6 4 5 1 7 3 6 6 5 1 2 7 Objects
- 31. © 2017 IBM Corporation31 Experimental Results The GridPocket Dataset § 1 million meters reporting every 15 minutes § Records are ~100 bytes § Generated ~1TB in 3 months § Partitioned using Grid Partitioner using precision 1 (cells are roughly 10 km2) § Compared using 50 and 100 Kakfa partitions #EUres2 Get my neighbours’ average usage SELECT AVG(usage) FROM ( SELECT vid as meter_id, (MAX(index)-MIN(index)) as usage FROM dataset WHERE (lat BETWEEN 43.300 AND 44.100) AND (lng BETWEEN 6.800 AND 7.600) GROUP BY vid )
- 32. © 2017 IBM Corporation32 Experimental Results The GridPocket Dataset § 1 million meters reporting every 15 minutes § Records are ~100 bytes § Generated ~1TB in 3 months § Partitioned using Grid Partitioner using precision 1 (cells are roughly 10 km2) § Compared using 50 and 100 Kakfa partitions #EUres2 Get my neighbours’ average usage SELECT AVG(usage) FROM ( SELECT vid as meter_id, (MAX(index)-MIN(index)) as usage FROM dataset WHERE (lat BETWEEN 43.300 AND 44.100) AND (lng BETWEEN 6.800 AND 7.600) GROUP BY vid )
- 33. © 2017 IBM Corporation33 #GB Transferred for bounding box queries #EUres2 0 200 400 600 800 1000 1200 20x20 km 10x10 km 5x5 km no filter filter #GB transferred 50 partitions 100 partitions 1 TB dataset, grid partitioner, precision 1, average of 10 randomly located queries
- 34. © 2017 IBM Corporation34 #GET Requests for bounding box queries #EUres2 1 TB dataset, grid partitioner, precision 1, took average of 10 randomly located queries 0 2000 4000 6000 8000 10000 12000 20x20 km 10x10 km 5x5 km no filter filter #GET requests 50 partitions 100 partitions
- 35. © 2017 IBM Corporation35 Time (sec) for bounding box queries #EUres2 1 TB dataset, precision 1, took average of 10 randomly located queries 0 500 1000 1500 2000 2500 20x20 km 10x10 km 5x5 km no filter filter Time (sec) 50 partitions 100 partitions
- 36. © 2017 IBM Corporation36 Demo § Runs on IOSTACK testbed – 3 Spark and 3 Object Storage nodes § Demo dataset – 1 million meters – Report every 15 mins – 1 day’s worth of data – 11 GB (csv) § Grid Partitioner Config – Cells have precision 1 - ~10 km2 – 100 partitions Get my neighbours’ average usage SELECT AVG(usage) FROM ( SELECT vid as meter_id, (MAX(index)-MIN(index)) as usage FROM dataset WHERE (lat BETWEEN 43.300 AND 44.100) AND (lng BETWEEN 6.800 AND 7.600) GROUP BY vid )
- 37. © 2017 IBM Corporation37 Conclusions: Don’t let your vacation turn into a relocation § Running SQL queries directly on Big Datasets in Object Storage is viable using metadata § A small change to Spark enables dramatic performance improvements § We focused here on geospatial data and the GridPocket use case, although it also applies to other data types and use cases § IBM is investing in Spark SQL as a backbone for SQL processing services in the cloud #EUres2
- 38. © 2017 IBM Corporation38 Thanks! Contact: paula@il.ibm.com https://www.linkedin.com/in/paulatashma/ guyger@il.ibm.com https://www.linkedin.com/in/guy-gerson-82619164/
- 39. © 2017 IBM Corporation39 Backup #EUres2
- 40. © 2017 IBM Corporation40 GridPocket Use Case and Dataset § NoSQL dataset – one large denormalized table – date, index, sumHC, sumHP, type, vid, size, temp, city, region, lat, lng – Index = meter reading – sumHC = total of energy consumed since midnight in off-hours – sumHP = total of energy consumed since midnight in rush-hours – Type = elec/gas/water – Vid = meter id – Size = apartment size in square meters
- 41. © 2017 IBM Corporation41 Example Filter Scenario § Want to analyze data from active sensors only § External DB contains sensor activity info Data Layout Archives/dt=01-01-2014 Archives/dt=01-01-2014/sensor1.json (500MB) Archives/dt=01-01-2014/sensor2.json (500MB) Archives/dt=01-01-2014/sensor3.json (500MB) Archives/dt=02-01-2014 Archives/dt=02-01-2014/sensor1.json (500MB) Archives/dt=02-01-2014/sensor2.json (500MB) Archives/dt=02-01-2014/sensor3.json (500MB) more... #EUres2 sensor active sensor1 FALSE sensor2 TRUE sensor3 FALSE
- 42. © 2017 IBM Corporation42 Example Filter class LiveSensorFilter extends ExecutionFileFilter { //get a list of live sensors from an external Service val activeSensors = SensorService.getLiveSensors //returns true if object represents a live sensor @Override def isRequired(dataFilters:Seq[org.apache.spark.sql.sources.Filter], fileStatus: FileStatus) : Boolean = { activeSensors.contains(Utils.getSensorIdFromPath(fileStatus)) } Turn filter on: sqlContext.setConf("spark.sql.execution.fileFilter", “LiveSensorFilter ”) #EUres2