Python time series analysis and visualization for self-driving cars
1 like687 views
This document discusses using Python and Spark for time series analysis and visualization of data from self-driving cars. It introduces DaSense, a language and framework for analyzing large sets of sensor data from autonomous vehicles on Spark. DaSense allows users to write Python code for time series analysis without knowledge of parallel programming. It also includes features like data extractors, lazy evaluation, optimized data structures, and the ability to integrate custom algorithms. The document concludes with an example analysis pipeline using DaSense to calibrate the automatic distance control system on a vehicle by analyzing histograms of distances to preceding vehicles.
![4NorCom Information Technology AG
Data Tsunami is Coming
Source: Stephan Heinrich (Lucid Motors) presented at “Flash Memory Summit“ [2017]](https://image.slidesharecdn.com/pyconwebv8-180715202702/85/Python-time-series-analysis-and-visualization-for-self-driving-cars-4-320.jpg)
Python time series analysis and visualization for self-driving cars
- 1. 1NorCom Information Technology AG Python time series analysis and visualization for self-driving cars Pyconweb, Munich, July 1, 2018 Andreas Pawlik
- 2. 2NorCom Information Technology AG Outline ‚Autonomous Driving and the Data Tsunami ‚Big Data Technologies: Hadoop, Spark ‚DaSense & DaSense Language for Time Series ‚From Development to Deployment ‚Application: Calibration of Automatic Distance Control
- 3. 3NorCom Information Technology AG Autonomous Driving Image: Newshawktime.com
- 4. 4NorCom Information Technology AG Data Tsunami is Coming Source: Stephan Heinrich (Lucid Motors) presented at “Flash Memory Summit“ [2017]
- 5. 5NorCom Information Technology AG Big Data = Hadoop 2003 Google Distributed File System Paper MapReduce Paper 2006 Hadoop is born from Nutch 2008 Facebook launches Hive 2004 Doug Cutting adds DFS and MapReduce to Nutch 2009 Yahoo! used Hadoop to sort one terabyte in 62 seconds 2018 -Innovation -Operation -Stability 2010 Spark Paper 2000 2005 2010 2015
- 6. 6NorCom Information Technology AG Leverage Big Data Technology for Automotive - DaSense Technology - Automotive Formats - Geo-Distributed Analyses - Engineer Self Service - Enterprise Level Implementation - Tiered Storage - Security & Access Control - IT Process Integration - Open-Source HADOOP Technology: - Internet-native measurement data-analysis framework - Scalable, Cost Effective, Flexible, Fast Access, Resilient 6
- 7. 7NorCom Information Technology AG Ready-To-Go Development Environment Based on Jupyter Bring your own libraries!
- 8. 8NorCom Information Technology AG DaSense Language - Time Series Analysis in Spark Data Logger ‚Python/Spark framework for time series ‚Reduces complexity – program as usual, no knowledge of parallelization required ‚Preserves lazy evaluation – optimize computation graphs ‚Open architecture – combine it with your Python libraries of choice, go Spark native ‚Data Structures optimized for Time Series Analysis ‚Plugin-Structure for existing algorithms
- 9. 9NorCom Information Technology AG DaSense Language Concepts – Lazy Evaluation Build the Expression Tree locally on the driver Evaluate it on the data distributed in the cluster histogram where rpm < speed 50 Data Extractor Function Math Data Extractor Float Function Inspect Result locally on the driver DATA
- 10. 10NorCom Information Technology AG DaSense Language Concepts – Data Extractors ‚Interface to the actual data uUnits (km/h, mph) uChannel name aliases (e.g., Velocity vs VehSpd) uResampling/Fusion ‚Assumes data is stored in Apache Parquet, Big Data conversion routines for many sensor data formats ‚Data Schema optimized for Time Series Analysis
- 11. 11NorCom Information Technology AG No knowledge of parallel programming needed DATA histogram where rpm < speed 50 Data Extractor Function Math Data Extractor Float Function
- 12. 12NorCom Information Technology AG Open architecture – combine it with your libraries of choice Pandas Dataframe – proceed as usual Spark Dataframe – for experts
- 13. 13NorCom Information Technology AG Data Structures optimized for Time Series Analysis Time Interval List Time Series Time Interval Time Series List Identical API for local and Spark computing!
- 14. 14NorCom Information Technology AG Plugin-structure for parallelizing existing algorithms Register your custom function Select it for execution in the expression tree
- 15. 15NorCom Information Technology AG From Development to Deployment Interactive Development Build DaSense App PROD INT DEV
- 16. 16NorCom Information Technology AG Time Interval Lists (TILs) chain Apps into Big Data Workflows Input: TILs Output: Modifizierte TILs Input: TILs Output: Snippets Input: TILs Output: Report (HTML/PDF)
- 17. 17NorCom Information Technology AG Application: calibration of automatic distance control Should the automatic distance control (ADC) system be more flexible? Does the vehicle ahead matter to the driver? When and why will he take over? Lets have a look! Compare sensor data to video data:1 2 BRAKE Get to know the driver!4 Analyze data3 The ADC system works just great – I feel totally safe and comfortable! I`d rather turn ADC off. I will brake and respect the distance myself.
- 18. 18NorCom Information Technology AG Analysis pipeline Search for transitions in ADC Check if a next vehicle exists Determine class of vehicle Determine histogram: Distance to preceding vehicle Determine histogram: Distance to preceding vehicle truck car Visualize result
- 19. 19NorCom Information Technology AG Analysis pipeline Search for transitions in ADC Check if a next vehicle exists Determine class of vehicle Determine histogram: Distance to preceding vehicle Determine histogram: Distance to preceding vehicle truck car Visualize result State transitions: ADC on -> off Event Search Driver node Spark Driver Worker node Task Executor Task Worker node Task Executor Task Task i Taskj
- 20. 20NorCom Information Technology AG Analysis pipeline Search for transitions in ADC Check if a next vehicle exists Determine class of vehicle Determine histogram: Distance to preceding vehicle Determine histogram: Distance to preceding vehicle truck car Visualize result Event Search State transitions: ADC on -> off
- 21. 21NorCom Information Technology AG Search for transitions in ADC Check if a next vehicle exists Determine class of vehicle Determine histogram: Distance to preceding vehicle Determine histogram: Distance to preceding vehicle car Visualize result Video data: CNN classification Car Truck State transitions: ADC on -> off truck Analysis pipeline
- 22. 22NorCom Information Technology AG Analysis pipeline Search for transitions in ADC Check if a next vehicle exists Determine class of vehicle Determine histogram: Distance to preceding vehicle Determine histogram: Distance to preceding vehicle truck car Visualize result Car Truck Data analysis Car Truck State transitions: ADC on -> off
- 23. 23NorCom Information Technology AG Result Search for transitions in ADC Check if a next vehicle exists Determine class of vehicle Determine histogram: Distance to preceding vehicle Determine histogram: Distance to preceding vehicle truck car Visualize result ‚Python/Spark based approach for flexible analysis ‚Makes use of DaSense Language for Time Series u Easy to write u Develop and run ‚Big Data Workflow Distance to next car truck car
- 24. 24NorCom Information Technology AG Summary ‚DaSense Language u Python/Spark based approach for flexible analysis of large sets of sensor data https://github.com/Dasense
- 25. 25NorCom Information Technology AG Thank you for your attention! Andreas Pawlik apw@norcom.de NorCom Information Technology AG Gabelsbergerstraße 4 80333 München