Paper@Soict2015: GPSInsights: towards a scalable framework for mining massive transportation data
1 like517 views
This document presents GPSInsights, a framework for efficiently storing and mining massive real-time vehicle location data. The framework uses an architecture based on scalable open-source components like Apache Spark, Storm, MongoDB and GeoMesa to store GPS data in real-time and perform mining tasks like scalable map-matching. The document describes experiments on 12 million GPS records which showed the framework can complete map-matching within seconds and scale to support millions of vehicles.
Paper@Soict2015: GPSInsights: towards a scalable framework for mining massive transportation data
- 1. GPSInsights: Towards an efficient framework for storing and mining massive real-time vehicle location data Linh-Truong Hoang, Duy-Khanh Bui, Viet- Trung Tran Hanoi University of Science and Technology 1
- 2. Agenda • Motivation • System architecture • Scalable map-matching • Experimentation • Conclusion 2
- 3. Global Navigation Satellite System (GNSS) • Autonomous geo-spatial positioning – position – velocity – time • "Great" points about GNSS – Free – Real-time – No required local infrastructures 3
- 4. GNSS as part of Intelligent transport system (ITS) • "precious" data for real-time traffic managements – traffic dashboard – speed control – traffic jams monitoring 4 Need for collec-ng and mining massive GNSS data in REAL-‐TIME
- 5. GNSS data characteristics • Real-time – reported every second • Massive in volume – from millions cars • "bad" data • Need to be processed within digital map topology 5
- 6. GNSS data is Bigdata's 5V 6
- 7. SYSTEM ARCHITECTURE Store massive GNSS data Real-time mining 7
- 8. 8 Elas(city High-‐throughput Fault-‐tolerance Scalable First-‐class spa(o-‐temporal API High-‐thoughput Fault-‐tolerance Online processing Scalable Fault-‐tolerence Leverage opensource components
- 9. 9
- 10. Apache spark processing • Resilient Distributed dataset (RDD) – In-memory, backed by persistent storage (HDFS) – fault-tolerance by lineage – Support interactive – iterative analysis 10
- 12. Apache storm 12
- 13. MongoDb with geo-indexing 13
- 14. Geomesa: Accumulo + geo-indexing 14
- 16. Map-matching • Online vs. Offline • OSM map 16
- 17. Algorithm • OSM map format • Filling intermediate points – Millions more points – Massive data – but simple calculations • real-time, scalable 17
- 18. K-d tree for closest neighbours • Run by apache spark/storm 18
- 20. Experiment setup • 12 millions GPS records collected by vehicles equipped with the GPS receiver in March 2014 • 4 nodes cluster – 8-cores Intel Xeon 2.6GHz CPU, 32GB memory 20
- 22. Latency 22
- 23. "Scalability" 23
- 24. Demonstration 24
- 27. Conclusion • GPSInsights: Scalable framework for storing and mining massive location data – built on open-source scalable components – scalable storage + real-time mining – Plug-able components – Demonstration with scalable map-matching algorithm • Future work – Advance map-matching algorithms – Traffic jam prediction 27
- 28. Current state-of-the-arts • PostGIS – Spatial objects management over Postgres – Small size – No mining supported 28