The Car of the Future - Autonomous, Connected, and Data Centric
2 likes344 views
The document discusses the future of autonomous vehicles, highlighting predictions of widespread adoption in the coming years and the corresponding explosion of data from connected vehicles. It outlines the data management challenges faced in the current environment and proposes solutions focused on in-place data storage and deep learning to enhance autonomous driving capabilities. Emphasizing the importance of data governance and security, the presentation concludes that successful data management innovation is crucial for the advancement of autonomous vehicle technology.
The Car of the Future - Autonomous, Connected, and Data Centric
- 1. The Car of the Future Autonomous, Connected and Data-Centric DWS Melbourne, Australia 2019 Robert Hryniewicz
- 2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda 1. Autonomous Vehicles 2. Autonomous Vehicle Learning Lifecycle 3. Data Management Challenges 4. Next-Generation Data Management 5. Architecture for Success 6. Solution in Action 7. Q & A
- 3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Title Goes HereThe Autonomous Car – It’s Happening “Nearly all new vehicles will be capable of full autonomy within 10 years” Elon Musk, 2017 10 million self-driving cars will be on the road by 2020 BI Intelligence, 2016 12 million fully autonomous vehicles by 2035 BCG, 2017 Varying predictions, but the future is clear McKinsey, 2016 Up to 15 percent of new cars in 2030 could be fully autonomous
- 4. 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Title Goes Here DATA EXPLOSION Annual Connected Vehicle Data to Grow Almost 10X By 2020* * Source: Cowen and Company, Gartner And Autonomous Vehicles Will Increase this by a further 25X** ** Source: Cowen and Company, Datameer
- 5. 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Title Goes HereAutonomous Driving Learning Lifecycle Training Data for Autonomous Driving Model Development Deploy Autonomous Driving Model To Vehicle In-Vehicle Processing Unit • Camera • Radar • Lidar • GPS • IMU • Other Sensors AUTOMONOMOUS VEHICLE In-Vehicle Storage Device DATA AUTONOMOUS DRIVING LEARNING LIFECYCLE Data Storage 2 Petabyte Scale Data Pre- Processing 3 Labeling, etc. Machine Learning 4 Auto- Labeling, etc. Rules Definition 5 Manual or Automatic Testing & Simulation 6 Using Training Data Data Ingestion 1 From Vehicle Model Deployment 7 To Vehicle
- 6. 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Title Goes Here COUNTRY 1 Autonomous Test Cars INGEST1 NAS Storage STORE2 MOVE3 WorkstationsCOMPUTE4 Engineer Engineer Traditional Data Management Approach COUNTRY 2…N
- 7. 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Title Goes HereData Management Challenges New Deployment Models Difficulty managing both…. Excessive Data Movement Movement between… STORE COMPUTE Data Volume and Variety NAS not optimized for… VOLUME VARIETY Data Governance & Security Fragmented management across…. PROGRAMS PARK ASSIST LANE DEPARTURE ADAPTIVE CRUISE LEVEL 4, 5 GEOGRAPHIES Data Intensive Computing Workstations & CPUs not optimized for ML X Workstation
- 8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Title Goes HereA Better Way SECURED & GOVERNED Centralized Governance - Across Geographies and Teams DEEP LEARNING & GPU Deep Learning frameworks (TensorFlow, PyTorch) GPU Pooling/Isolation “In Place” Data Processing DON’T MOVE DATA HYBRID & MULTI CLOUD Unified Management of Clusters - whether on Cloud, On-Premise or Hybrid UNIVERSAL DATA STORAGE Autonomous Vehicle Research Data Lake Infinitely Scalable (Billions of files, Exabytes) Low TCO
- 9. 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Title Goes Here“In-Place” Data Storage TO ü CREATE ALGORITHM ü STORE + COMPUTE AUTONOMOUS CARS • Data stored and processed on Hadoop • Elimination of data movement • Massively parallel computing • Dramatically faster computing times FROM ü CREATE ALGORITHM ü STORE ü COMPUTE AUTONOMOUS CARS • Data moved to workstation • Data processed on workstation
- 10. 10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Title Goes HereSmarter Decisions Made Based on Support for Deep Learning Workloads Why GPU support? à Enhances the performance of computations needed for enterprise ML/DL apps à DL requires intense computational algorithms à Containerized software powered by GPUs helps data processing at scale Result: Data Scientists can run DL models in days vs months, hours vs days, minutes vs hours
- 11. 11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Title Goes HereSecurity and Governance Full chain of custody of data across the Hadoop ecosystem Auditing of events for fine grained and detailed info Tag propagation allows auditors to see where the data is going across the enterprise & retain context of sensitive data Time-based policies to allow temporary access to users
- 12. 12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Title Goes Here SOLUTION IN ACTION
- 13. 13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Title Goes Here lidars (front) radar GPS GPS lidar (side) lidar (rear) lidar stereo camera remote control antenna 2007 DARPA Urban Challenge
- 14. 14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Title Goes Here lidar stereo camera Nvidia Jetson Tx2 ”the brain” power supply (computer) power supply (car)
- 15. 15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Title Goes HereTraining workflow HDP Cluster for Training Model Data (30K images+ Steering angles) 5 Servers 6 GPU cards Dockerized TensorFlow GPU Pooling Zeppelin Inference
- 16. 16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Title Goes Here
- 17. 17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Title Goes HereIn Conclusion Autonomous Vehicles Coming Soon Data-Centric Learning Based on Data! Data Challenges Volume, Variety, Storage, Compute, Learning, Governance and Security Crossing the Chasm Data Management Innovation Foundational to Success
- 18. 18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Questions?
- 19. 19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Thanks! Robert Hryniewicz