Vision-Language-Action Models for Embodied AI
- 1. Vision-Language-Action Models for Embodied AI A Deep Dive into the 2024 Survey Mohibkhan Pathan CMPE 258 - Deep Learning Professor Vijay Eranti
- 2. What is Embodied AI? 08 Why Do We Need VLA Models? 02 What Are VLA Models? 03 High-Level Task Planners 07 Training Data and Benchmarks 01 Challenges 09 Table of contents The Three Main Parts of VLA Models 11 Key Components 05 Low-Level Control 06 The Future of VLAs 10 References 04
- 3. ● AI that lives in the real world, not just on a screen ● Can see, understand, and take actions ● Example: A robot that follows the command “bring me a cup” ● It combines vision, language, and movement ● Not like ChatGPT or CLIP — they don’t interact with the physical world What is Embodied AI?
- 4. ● Real-world tasks are complex and multi-step ● Robots must understand what to do, what they see, and how to act ● One model for just vision or just language is not enough ● VLA models let robots follow natural commands like humans ● Helps in homes, hospitals, factories, and more Why Do We Need VLA Models?
- 5. What are VLA Models? ● Real-world tasks are complex and multi-step ● Robots must understand what to do, what they see, and how to act ● One model for just vision or just language is not enough ● VLA models let robots follow natural commands like humans ● Helps in homes, hospitals, factories, and more
- 6. The Three Main Parts of VLA Models ● VLA models have 3 main parts: 1. Components – visual/language encoders and world models 2. Low-Level Control – small step actions (e.g. move, pick) 3. High-Level Planners – break big tasks into small steps ● This structure helps models plan and act better ● Like a team: planner = brain, controller = hands
- 7. Key Components ● Vision Encoder – helps robot “see” (e.g. CLIP, R3M, MVP) ● Language Encoder – understands commands (e.g. BERT, GPT) ● Dynamics Model – learns how actions change the world ● World Model – predicts what will happen next (like a mini-simulator) ● These parts work together to help the robot think before it moves
- 8. Low-Level Control ● Low-level control = small actions like pick, move, turn ● Uses info from camera + instruction to act in real-time ● Common methods: ○ FiLM – adjusts vision using language ○ Cross-Attention – connects vision + language deeply ○ Concatenation – joins both inputs together ● Models: CLIPort, BC-Z, RT-1, UniPi ● Some use transformers or even learn from videos
- 9. High-Level Task Planners ● Planner breaks long tasks into smaller actions ● (e.g. “clean room” → pick up toy → wipe table) ● Two common types: ○ Language-based: LLM writes out steps in text ○ Code-based: LLM creates commands using functions like pick() or move() ● Helps robot know what to do next ● Famous examples: SayCan, InnerMonologue, ProgPrompt
- 10. Training Data and Benchmarks ● Real robot data is hard to collect and expensive ● Simulators are used for faster and safer training (e.g. Habitat, AI2-THOR) ● Some models learn from human videos or internet data ● Benchmarks help test models: ○ EmbodiedQA: ask + explore ○ RLBench: robot manipulation ○ EgoPlan / PlanBench: test planning skills ● Important to compare models fairly
- 11. ● Real data is hard to get – robot demos take time ● Models are slow – need to act faster in real life ● System is complex – many parts must work together ● Struggle with new tasks – not good at generalizing ● No standard tests – hard to compare different models ● Safety is important – robots must be trusted by people Challenges
- 12. ● Smarter planning with better world models ● Faster and smaller models for real-time use ● Use in homes, hospitals, factories, and more ● Safer and more human-friendly robot behavior ● Learn from the world just like humans do The Future of VLA Models
- 13. Ma, Y., Song, Z., Zhuang, Y., Hao, J., & King, I. (2024). A Survey on Vision-Language-Action Models for Embodied AI. arXiv preprint arXiv:2408.14496. https://arxiv.org/abs/2408.14496 Resources