Traffic Data Analysis and Prediction using Big Data
Download as PPTX, PDF1 like1,640 views
The document presents a study on traffic data analysis and prediction using big data and machine learning at California State University, Los Angeles. It details the methodology, hardware specifications, implementation steps, and analysis techniques employed to predict traffic congestion, highlighting the decision forest model achieving 83% accuracy. The study identifies major traffic areas and peak hours, emphasizing the potential for improved predictions with larger datasets.
Traffic Data Analysis and Prediction using Big Data
- 1. Jongwook Woo HiPIC CalStateLA KSII The 14th Asia Pacific International Conference on Information Science and Technology(APIC-IST), Beijing June 24 2019 Dalya (Dalyapraz) Dauletbak, dmanato@calstatela.edu Jongwook Woo, PhD Big Data AI Center (BigDAI) California State University Los Angeles Traffic Data Analysis and Prediction using Big Data
- 2. Big Data Artificial Intelligence Center (BigDAI) Jongwook Woo CalStateLA Contents Introduction H/W Specification Architecture Chart Implementation steps Data structure Analysis Prediction Summary
- 3. Big Data Artificial Intelligence Center (BigDAI) Jongwook Woo CalStateLA Introduction About me: Graduate Computer Information Systems Student at California State University, Los Angeles – BS (2015): Mathematics at Nazarbayev University – Previously: Senior Consultant/Data Analyst @ Management consulting at KPMG Central Asia – Current: Community Manager @ International Data Engineering and Science Association (IDEAS) Data source: A GPS navigation mobile application Provide real-time directions and up-to-date information Traffic Accidents Road closure Weather hazards Lurking police vehicles and etc.
- 4. Big Data Artificial Intelligence Center (BigDAI) Jongwook Woo CalStateLA Introduction Data source: Navigation app traffic data set from LA City Department* Information reported by users - Alerts information captured by user’s device - Jams We are going to find out: Areas with high volume of traffic (geography) Peak-hours Density of Alerts and Incidents Traffic volume by road types Prediction of traffic jam *Limited authorization to access the full datasets 100 GB + original; we used limited dataset to 9 days (Dec 31– Jan 8, 2018) ~2GB
- 5. Big Data Artificial Intelligence Center (BigDAI) Jongwook Woo CalStateLA Contents Introduction H/W Specification Architecture Chart Implementation steps Data structure Analysis Prediction Summary
- 6. Big Data Artificial Intelligence Center (BigDAI) Jongwook Woo CalStateLA H/W Specification Number of nodes 6 OCPUs 12 CPU speed 2195.196MHz Memory 180 GB Storage 682 GB
- 7. Big Data Artificial Intelligence Center (BigDAI) Jongwook Woo CalStateLA Architecture Chart Source: Hadoop Masterclass Part 4 of 4: Analyzing Big Data Lars George | EMEA Chief Architect Cloudera
- 8. Big Data Artificial Intelligence Center (BigDAI) Jongwook Woo CalStateLA Implementation steps Local Computer Raw data files (JSON) Geo-Spatial Visualization (3D map) Dashboard for Analytics Hadoop/Hive Upload dataset to HDFS Parse JSON files using Pandas Create tables’ schema Clean data Create sample/summary dataset for prediction and visualization Microsoft Azure ML Studio Upload sample dataset Apply data transformation Split dataset for training and scoring Train model(s) Evaluate model(s)
- 9. Big Data Artificial Intelligence Center (BigDAI) Jongwook Woo CalStateLA Data structure
- 10. Big Data Artificial Intelligence Center (BigDAI) Jongwook Woo CalStateLA Contents Introduction H/W Specification Architecture Chart Implementation steps Data structure Analysis Prediction Summary
- 11. Big Data Artificial Intelligence Center (BigDAI) Jongwook Woo CalStateLA Analysis Information we are using: Location/Time Level of traffic intensity X and Y coordinates (Longitude & Latitude) Counts of jams/alerts Tools we are using: Excel - 3D map Power BI - Flow map, pie charts, bar charts What we are predicting: Level of traffic (1 to 3 – light, medium, heavy) Based on date, time, location
- 12. Big Data Artificial Intelligence Center (BigDAI) Jongwook Woo CalStateLA Traffic in LA (captured from users' devices)
- 13. Big Data Artificial Intelligence Center (BigDAI) Jongwook Woo CalStateLA Traffic in LA (reported by app users)
- 14. Big Data Artificial Intelligence Center (BigDAI) Jongwook Woo CalStateLA Video-Simulation of Traffic in LA (captured from users' devices)
- 15. Big Data Artificial Intelligence Center (BigDAI) Jongwook Woo CalStateLA Video-Simulation of Traffic in LA (reported by app users)
- 16. Big Data Artificial Intelligence Center (BigDAI) Jongwook Woo CalStateLA Traffic Analysis Dashboard Peak Peak
- 17. Big Data Artificial Intelligence Center (BigDAI) Jongwook Woo CalStateLA Traffic Analysis Dashboard Major areas of traffic are: Downtown Los Angeles, Santa Monica, Hollywood, and highways.
- 18. Big Data Artificial Intelligence Center (BigDAI) Jongwook Woo CalStateLA Contents Introduction H/W Specification Architecture Chart Implementation steps Data structure Analysis Prediction Summary
- 19. Big Data Artificial Intelligence Center (BigDAI) Jongwook Woo CalStateLA Prediction of traffic congestion with Machine Learning Data preparation Group label values Join additional dataset Apply data transformation Normalize data Model building Model(s) selection Cross Validation Train model(s) Model evaluation Score model Evaluate model (Accuracy, Recall)
- 20. Big Data Artificial Intelligence Center (BigDAI) Jongwook Woo CalStateLA Features/columns in a dataset location x, location y X and Y -coordinate of location date_pst Pacific Time of the publication of traffic report level jam level, where 1 – almost no jam and 5 – standstill jam speed driver’s captured speed in mph length length of the traffic ahead in the route of user in meters *date_pst *date splits into month, day, hour, min, sec, weekday
- 21. Big Data Artificial Intelligence Center (BigDAI) Jongwook Woo CalStateLA Data transformation Randomly selected data ~ 100MB Select relevant features Group level into 2 classes (label: 0 & 1) Join holidays dataset Add attribute is_holiday (0 or 1) Change cyclical attributes from Polar coordinates to Cartesian Add is_rush, is_weekend (0 or 1) Normalize features Make categorical: is_rush, is_holiday, is_weekend, label
- 22. Big Data Artificial Intelligence Center (BigDAI) Jongwook Woo CalStateLA SELECT location_x, location_y, SIN((weekday)*(2*PI()/7)) as sin_weekday, COS((weekday)*(2*PI()/7)) as cos_weekday, SIN((month-1)*(2*PI()/12)) as sin_month, COS((month-1)*(2*PI()/12)) as cos_month, SIN((day-1)*(2*PI()/31)) as sin_day, COS((day- 1)*(2*PI()/31)) as cos_day, SIN(hour*(2*PI()/24)) as sin_hour, COS(hour*(2*PI()/24)) as cos_hour, SIN(min*(2*PI()/60)) as sin_min, COS(min*(2*PI()/60)) as cos_min , SIN(sec*(2*PI()/60)) as sin_sec, COS(sec*(2*PI()/60)) as cos_sec, …
- 23. Big Data Artificial Intelligence Center (BigDAI) Jongwook Woo CalStateLA MODEL Evaluation Model Accuracy Precision Recall AUC ROC LR 0.662 0.662 1.0 0.571 BDT 0.805 0.832 0.884 0.868 DF 0.832 0.868 0.880 0.885
- 24. Big Data Artificial Intelligence Center (BigDAI) Jongwook Woo CalStateLA Summary of Traffic Prediction with Machine Learning Model is based on sampled dataset ~ 1M rows (100 MB) Best model - Decision Forest Accuracy – 0.832 Precision - 0.868 Recall - 0.880 Area under the Curve – 0.885 Confusion Matrix
- 25. Big Data Artificial Intelligence Center (BigDAI) Jongwook Woo CalStateLA Contents Introduction H/W Specification Architecture Chart Implementation steps Data structure Analysis Prediction Summary
- 26. Big Data Artificial Intelligence Center (BigDAI) Jongwook Woo CalStateLA Summary Denser traffic on Freeways 101, 405, 10 Rush hours from 7 am to 9 am produce a lot of traffic, the heaviest traffic time start from 3pm and gets better after 6pm. Major areas of traffic in DTLA, Santa Monica, Hollywood More insights can be found with bigger dataset using this framework for analysis of traffic Using such data and platform can also give an opportunity to predict traffic congestions. Prediction can be performed using machine learning algorithm – Decision Forest with the accuracy of 83% for predicting the heaviest traffic jam.
- 27. Big Data Artificial Intelligence Center (BigDAI) Jongwook Woo CalStateLA Questions?
- 28. Big Data Artificial Intelligence Center (BigDAI) Jongwook Woo CalStateLA References 1. J. Barbaresso, G. Cordahi, D. Garcia et al., “USDOT’s Intelligent Transportation Systems (ITS) ITS Strategic Plan 2015- 2019,” 2014. 2. “Integrated Corridor Management,” Intelligent Transportation Systems - Integrated Corridor Management, www.its.dot.gov/research_archives/icms/. Accessed April 14, 2019. 3. J. Kestelyn, “Real-Time Data Visualization and Machine Learning for London Traffic Analysis,” Google Cloud, 2016, cloud.google.com/blog/products/gcp/real-time-data-visualization-and-machine-learning-for-london- traffic-analysis. Accessed April 14, 2019. 4. “Connected Citizens by Waze,” Waze, www.waze.com/ccp. Accessed April 14, 2019. 5. M. Schnuerle, “Louisville and Waze: Applying Mobility Data in Cities,” Harvard Civic Analytics Network Summit on Data-Smart Government, 2017. 6. Louisville Metro. “Thunder Jams, 2017 Traffic Delays.” CARTO, louisvillemetro- ms.carto.com/builder/d98732d0-1f6a-4db2-9f8a-e58026bf0d39/embed. Accessed April 14, 2019. 7. Louisville Metro. “Pothole Animation.” CARTO, cdolabs-admin.carto.com/builder/a80f62bf-98e1-4591-8354- acfa8e51a8de/embed. Accessed April 14, 2019. 8. E. Necula, “Analyzing Traffic Patterns on Street Segments Based on GPS Data Using R,” Transportation Research Procedia, Vol. 10, pp. 276–285, 2015.
- 29. Big Data Artificial Intelligence Center (BigDAI) Jongwook Woo CalStateLA References 9. J. Woo and Y. Xu, “Market Basket Analysis Algorithm with Map/Reduce of Cloud Computing,” in Proc. of International Conference on Parallel and Distributed Processing Techniques and Applications (PDPTA), Las Vegas. 2011. 10. “Pandas.io.json.json_normalize.” Pandas.io.json.json_normalize - Pandas 0.24.2 Documentation, pandas.pydata.org/pandas-docs/stable/reference/api/pandas.io.json.json_normalize.html. Accessed April 14, 2019. 11. United States, Chief Executive Office County of Los Angeles. “Cities within the County of Los Angeles.” lacounty.gov. Accessed April 14, 2019. 12. Garyericson. “What Is - Azure Machine Learning Studio.” Microsoft Docs, docs.microsoft.com/en- us/azure/machine-learning/studio/what-is-ml-studio. Accessed April 14, 2019. 13. A. Tharwat, “Classification Assessment Methods.” Applied Computing and Informatics, 2018. 14. M. Sokolova and L. Guy, “A Systematic Analysis of Performance Measures for Classification Tasks,” Information Processing & Management, Vol. 45. No. 4, pp. 427–437, 2009.