[DSC Europe 24] Aleksandar Cvejic - Rivian Autonomy and AI: Building a Scalable Platform for Data and ML Workloads

Editor's Notes

• #3: Hi everyone, my name is Aleksandar, I’m a lead of Data Applications and ML infra team here in Serbia. We are part of the Autonomy and AI organisation at Rivian and together with our colleagues in the US we manage and maintain Cloud infra and cloud data platform for Rivian’s Advanced Driver Assistance Systems—or ADAS.
• #4: Today, I want to share our journey in tackling ADAS challenges. I’ll walk you through how we manage data at scale, the role of Polaris—our centralized platform—and how it enables workflows like simulations and machine learning. Along the way, I’ll highlight some specific optimizations we’ve implemented to ensure scalability, efficiency, and innovation. Let’s start by exploring the challenges unique to this domain.
• #5: In the world of autonomy, data isn’t just critical—it’s massive, diverse, and constantly growing. Let’s look at the key challenges we face in ADAS development. Volume and Variety Data grows in an unprecedented manner and requirements to process and utilize that data change continuously. Processing and Orchestration Managing this data is no small task - anonymizing sensitive information, tagging metadata, and preparing it for machine learning or simulation workflows. Each step needs to be efficient, scalable, and accurate to meet the demands of our iteration cycles. Speed is crucial.
• #6: DATA Variety Rivian’s vehicles are equipped with: 11 cameras capturing 55MP real-time imaging at state-of-the-art resolution and dynamic range, 5 advanced radars providing multi-modal sensing and detecting objects up to 300 meters ahead, 360° sensing with overlapping sensors for redundancy and performance. The result of those sensors is vaste volume of data that grows exponentially.
• #7: On that scale each optimization translates in cost reduction of hundreds of thousands of dollars in cloud infra and in engineering working hours.
• #8: So how does the Data loop, or data lifeccyle in ADAS looks like? Bird eye view of data lifecycle in Autonmy is pretty much straightforward. Data is collected uploaded, processed, used for model training, deployed to our vehicles using OTA and then the cycle starts again.
• #9: Processing Before we built our Data platform, we had tools and pipelines built for the specific use cases, built by different teams, many times different teams solved same problems in a different ways. So we decided to create a comprehensive and centralised Data platform that we named Polaris
• #10: Polaris is the backbone of Rivian’s ADAS data operations, designed to unify, simplify, and scale our workflows. Think of it as the operating system for autonomy data. We had few of the guiding principles while we were building it: —-- By solving these challenges, Polaris empowers teams to focus on innovation rather than infrastructure.
• #11: If we go back to that Data loop that I was showing few slides ago I want to reiterate on it with the focus on our ML workload workflow. —------- Now, let’s take a closer look at how simulations and machine learning workflows work within Polaris.
• #12: Simulations are vital to ADAS development, enabling us to generate additional data and validate models under controlled conditions. Data Augmentation Collecting real-world data is time-consuming and costly. Simulations help us generate millions of miles of synthetic data, accelerating model training. Scenario Testing Simulations allow us to validate models against specific KPIs, such as obstacle detection or lane-keeping accuracy.
• #13: Fluentd, as well as Prometheus, for collecting logs from the applications and metrics. We also use EBS, EFS, S3, which are storage offerings or storage plugins that interact with Amazon services. GPU operator. It is used to essentially expose the GPUs that are available on the compute as a Kubernetes resource. So users can choose any rate GPUs when they setup their jobs. Kueue, open source plugin from Google, gang scheduling or preemption, prioritization, and different kinds of scheduling. We also use RayJob CRD, which is part of KueueRay operator. This RayJob CRD is plugin that is responsible for managing the lifecycle of a job. PyTorch for distributed data parallel processing inside the RayJob. MLflow for tracking so all the metrics that are emited from the job are collected and stored in MLflow for analysis later. We also store checkpoints in MLflow
• #14: Rivx is a job scheduling tool for automatically launching jobs on the cloud. Backends Docker image To use rivx-train, you need to create a .rivxconfig. Based on resource requests kueue schedules your job at one of the falvors that can fulfile requirements. Ray clusters are ephemeral- usage optimization and removing single point of failure making it harder to debug issues, get logs, track progress of runs …etc So we built observability and tracking system for users
• #15: So we use prometheus to collect metrics and logs and expose them in Grafana. As grafana is AWS managed and thus costs and not everyone outside the team are not well-versed we build our own app that will be entry point for all ML Workload insights, inside the Polaris platform, called Cosmo.
• #17: —------------------ That would be all I wanted to show you today regarding ADAS Cloud platform at Rivian
• #18: What’s Next? If there is one thing I want you to remember from today, it would be that chart I showed you at the begging. As Rivian expands its fleet and enters new markets, including Europe, so be ready to see one of these in your neighborhood. With this growth, the scale and complexity of data will continue to grow. It’s difficult to predict exactly what we’ll face a year from now, so the only constant in this field is change. Polaris is a scalable, efficient, and user-friendly platform that enables us to stay ahead of the curve. Thank you for your time—I’m happy to take your questions.