Data pipeline and data lake for autonomous driving
0 likes551 views
This document outlines autonomous driving data pipelines and data lakes used by various companies. It discusses Tesla, Google Waymo, PlusAI, Alibaba Cloud, Nvidia, NetApp, Amazon AWS, Amazon TRI, Amazon Momenta, and data pipeline strategies from Eckerson DataOps and IBM. The document also provides a detailed overview of an autonomous driving data lake built on AWS that ingests vehicle telemetry data and processes drive data for labeling and search capabilities.
Data pipeline and data lake for autonomous driving
- 1. DATA PIPELINE AND DATA LAKE IN AUTONOMOUS DRIVING YU HUANG SUNNYVALE, CALIFORNIA YU.HUANG07@GMAIL.COM
- 2. OUTLINE • TESLA • GOOGLE WAYMO • PLUSAI • ALIBABA CLOUD • NVIDIA • NETAPP • AMAZON-AWS • AMAZON-TRI • AMAZON-MOMENTA • ECKERSON DATAOPS • IBM
- 3. Tesla
- 7. Nvidia
- 8. Nvidia
- 9. Nvidia
- 10. Nvidia
- 11. Nvidia
- 12. Nvidia
- 13. NetApp
- 14. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS Reference Architecture Autonomous Driving Data Lake Build an MDF4/Rosbag-based data ingestion and processing pipeline for Autonomous Driving and Advanced Driver Assistance Systems (ADAS). 10 9 8 7 6 5 4 3 2 Ingest data from autonomous fleet with AWS Outposts for local data processing. Ingest vehicle telemetry data in real time using AWS IoT Core and Amazon Kinesis Data Firehose. Remove and transform low quality data. Schedule the extract, transform, load (ETL) jobs using Apache Airflow. Enrich data with weather conditions based on GPS location and timestamp. Extract metadata using ASAM OpenSCENARIO and store data in Amazon DynamoDB and Amazon Elasticsearch Service. Store data lineage in Amazon Neptune and catalog data using AWS Glue Data Catalog. Process drive data and perform deep signal validation. Perform automated labeling using Amazon SageMaker Ground Truth. Provide a search function for particular scenarios using AWS AppSync. 1 1 2 3 4 5 6 7 8 9 10
- 15. Radar and video data processing in MDF4 format achieves the highest scalability by leveraging AWS Fargate for Amazon ECS.
- 18. Autonomous driving at AWS
- 19. TRI
- 20. Momenta
- 21. Next Generation Data Pipelines
- 22. Data Pipeline Complexity Eckerson DataOps
- 23. Analytic Models Failure to Launch Eckerson DataOps
- 24. Streaming Data Pipelines Eckerson DataOps
- 25. Requirements for Autonomous Navigation Eckerson DataOps
- 26. DataOps for Model and Pipeline Development and Operationalization Continuous Integration / Continuous Development（CI/CD） Eckerson DataOps
- 27. Components and Complexities of Analytics Ecosystems Eckerson DataOps
- 28. IBM Enterprise Data pipeline Marketecture
- 29. IBM Autonomous driving development use cases
- 30. Thanks