Smartnets2018
- 1. 16th November, 2018 1st IEEE SmartNets @ Hammamet, Tunisia 1 The Quest for Scalable and Intelligent Trajectory Data Analytics Systems: Status Report and Future Directions Rim Moussa LaTICE Lab. Univ. of Tunis and University of Carthage Ahmed Haddad LaTICE Lab. Univ. of Tunis and University of Carthage Tarek Bejaoui MEDIATRON Lab. University of Carthage 1st IEEE SmartNets @ Hammamet 16th of November, 2018
- 2. 16th November, 2018 1st IEEE SmartNets @ Hammamet, Tunisia 2 Scalable Trips' Records Analysis ●Characteristics of trips' records »Big volume »Big velocity ●NYC Cabs' dataset »http://www.nyc.gov/html/tlc/html/about/trip_record_data.shtml »More than 200GB »Yellow and Green taxi trips' records from 2009 to now »csv format »capturing pick-up and drop-off dates/times, pick-up and drop-off locations, fares, rate types, payment types, and driver-reported passenger counts
- 3. 16th November, 2018 1st IEEE SmartNets @ Hammamet, Tunisia 3 Goals of Scalable Trips' Records Analysis ●Turn trajectory data into knowledge »Multi-dimensional analysis of trajectory data ●e.g. Average fare, Average trip duration... for a given pick-up location and a given drop-off location between 9pm and 10pm »Mining of Trajectory Patterns ●Hotspots and cold areas ●Frequent/Infrequent trajectory patterns ●Turn knowledge into decisions ●Intelligent urban computing
- 4. 16th November, 2018 1st IEEE SmartNets @ Hammamet, Tunisia 4 Outline ●Key Functional Requirements of Intelligent and Scalable Trajectory Data Analysis ●Overview of state-of-the-art open -source Technologies »Elastic stack -data shippers + search engine + visualization »Geomondrian -spatial relational OLAP engine + Relational DBMS »Leaflet -JavaScript library for mobile-friendly interactive maps + relational data store »Neo4j -graph database ●Neo4j Extension ●Conclusions ●Future Directions
- 5. 16th November, 2018 1st IEEE SmartNets @ Hammamet, Tunisia 5 Key Functional Requirements ↬ Spatial On-Line Analytical Processing (SOLAP) ●OLAP tools enable users to analyze multidimensional data interactively from multiple perspectives ●Multi-dimensional data analysis »Spatial dimension: GPS data, area, ... »Time dimension: time range, day/night, »Standard dimensions such as #passengers, payment type... »Measures: count trips, sum trips' fares …. ●OLAP operations »Drill-down: show trips' details for a combination of dimensions »Roll-up: aggregate trips' data for a combination of dimensions »Slice: filter along dimension »Dice: filter along more than one dimension
- 6. 16th November, 2018 1st IEEE SmartNets @ Hammamet, Tunisia 6 Key Functional Requirements ↬ Spatial Data Mining ●Algorithms for learning trajectory patterns from historical data »Path patterns (infrequent/frequent path patterns, triangle patterns) »Hotspots/cold areas »Co-location patterns, e.g. weather conditions and trips patterns »Stay Points, trips’ trajectory patterns, driving and speed patterns. ●Algorithms for predicting future events such as a car destination, future traffic congestion, trip’s cost, et cetera
- 7. 16th November, 2018 1st IEEE SmartNets @ Hammamet, Tunisia 7 Key Functional Requirements ↬ Geo-visualization ●Geo-visualization »Interactive Maps »change the visual appearance of the map (e.g. colors, day/night theme)
- 8. 16th November, 2018 1st IEEE SmartNets @ Hammamet, Tunisia 8 Key Functional Requirements ↬ Geo-visualization (ctnd. 2) ●Zoom-in and zoom-out
- 9. 16th November, 2018 1st IEEE SmartNets @ Hammamet, Tunisia 9 Key Functional Requirements ↬ processing mode ●Batch processing »Capacity to process historical data ●Real-time processing »Capacity to process Real-time data »Stream systems ●Lambda architecture »Combine both systems
- 10. 16th November, 2018 1st IEEE SmartNets @ Hammamet, Tunisia 10 Overview of state-of-the-art Technologies ↬ elastic stack ●Elasticsearch »Distributed search engine and document store system »Distributed inverted indices for querying free text ●Logstash and Beats for data ingest ●Kibana for visualization ●Real world users: NASA, Uber, Lyft, Tinder, CISCO, New York Times, eBay, Groupon, Wikipedia, Stackoverflow, GitHub...
- 11. 16th November, 2018 1st IEEE SmartNets @ Hammamet, Tunisia 11 Overview of state-of-the-art Technologies ↬ NYC cabs' records exploration with elastic stack
- 12. 16th November, 2018 1st IEEE SmartNets @ Hammamet, Tunisia 12 Overview of state-of-the-art Technologies ↬ NYC cabs' records exploration with elastic stack
- 13. 16th November, 2018 1st IEEE SmartNets @ Hammamet, Tunisia 13 Overview of state-of-the-art Technologies ↬ NYC cabs' records exploration with elastic stack
- 14. 16th November, 2018 1st IEEE SmartNets @ Hammamet, Tunisia 14 Overview of state-of-the-art Technologies ↬ elastic stack discussion ●Geo-visualization »Interactive maps ●Spatial OLAP »Use Domain Specific Language (DSL) to query elasticsearch cluster ●Spatial data mining and trajectory patterns »Not supported ●Processing mode »Both batch and real-time
- 15. 16th November, 2018 1st IEEE SmartNets @ Hammamet, Tunisia 15 Overview of state-of-the-art Technologies ↬ Jpivot + Geomondrian + PostgreSQL
- 16. 16th November, 2018 1st IEEE SmartNets @ Hammamet, Tunisia 16 Overview of state-of-the-art Technologies ↬ Geomondrian discussion ●Geo-visualization »No Interactive maps »Need SOLAP client ●Spatial OLAP »Use SQL to query the Relational store ●Spatial data mining and trajectory patterns »pgpRouting -spatial extension of PostgreSQL ●Processing mode »batch processing
- 17. 16th November, 2018 1st IEEE SmartNets @ Hammamet, Tunisia 17 Overview of state-of-the-art Technologies ↬ Leaflet + relational data store (MySQL) Visualizing Millions of NYC taxi pick-ups locations for year 2014 (spatial points are clustered using superclustreing)
- 18. 16th November, 2018 1st IEEE SmartNets @ Hammamet, Tunisia 18 Overview of state-of-the-art Technologies ↬ Leaflet discussion ●Geo-visualization »Interactive maps ●Spatial OLAP »Use SQL to query the Relational store ●Spatial data mining and trajectory patterns »Not supported ●Processing mode »batch processing
- 19. 16th November, 2018 1st IEEE SmartNets @ Hammamet, Tunisia 19 Graph oriented data store solution ↬ Graph oriented databases: Neo4j, Graph Frames/Spark ●Directed Graph design »Vertices: aggregated spatial locations »Relationships: Bags of trips data ●Extend Neo4j CYPHER query language to support OLAP operations »Roll-up graph »Drill down graph ●Use of Apache Spark for data preprocessing »Processing Spatial data ●Map each GPS pick-up/drop-off into a geohash ●Geohash =12 → Area width x height: 3.7cm x 1.9cm ●Geohash =7 → Area width x height: 152.9m x 152.4m ●Geohash =4→ Area width x height: 39.1km x 19.5km .. »Processing Time data ●Map each date-time pick-up/drop-off into a timehash
- 20. 16th November, 2018 1st IEEE SmartNets @ Hammamet, Tunisia 20 Graph oriented data store solution ↬ Graph oriented databases: Neo4j, Graph Frames/Spark ●Scalable data processing with CAPS -Cypher for Apache Spark ●Most Trajectory patterns are provided or implemented using either Neo4j-Cypher , Apache Spark/Graph Frames or MLib »Graph traversal Algorithms -Breadth First Search is provided »Depth First Search is implemented »Frequent/infrequent trajectory patterns »Hotspots and cold areas »Page rank »Connected components »Clustering ● Work-in-progress »Visualization on a world Map
- 21. 16th November, 2018 1st IEEE SmartNets @ Hammamet, Tunisia 21 Conclusion & Future work ●Conduct experiments on a HPC platform »Benchmark NoSQL graph databases: Neo4j vs. Graph Frames for each defined business query ●Extend Graph capabilities in Elastic ●Combine multiple datasets » e.g. Trajectory data with open datasets such as weather data, crime data in NYC
- 22. 16th November, 2018 1st IEEE SmartNets @ Hammamet, Tunisia 22 Thank you for your Attention Q & A The Quest for Scalable and Intelligent Trajectory Data Analytics Systems: Status Report and Future Directions Rim Moussa, Ahmed Haddad and Tarek Bejaoui 16th of November, 2018 1st IEEE SmartNets @ Hammamet, Tunisia