How to Manage Multimodal Robotics Data Formats

Ever wondered what robots 🤖 could achieve if they could not just see – but also feel and hear? We introduce FuSe: a recipe for finetuning large vision-language-action (VLA) models with heterogeneous sensory data, such as vision, touch, sound, and more. We use language instructions to ground all sensing modalities by introducing two auxiliary losses. In fact, we find that naively finetuning on a small-scale multimodal dataset results in the VLA over-relying on vision, ignoring much sparser tactile and auditory signals. By using FuSe, pretrained generalist robot policies finetuned on multimodal data consistently outperform baselines finetuned only on vision data. This is particularly evident in tasks with partial visual observability, such as grabbing objects from a shopping bag. FuSe policies reason jointly over vision, touch, and sound, enabling tasks such as multimodal disambiguation, generation of object descriptions upon interaction, and compositional cross-modal prompting (e.g., “press the button with the same color as the soft object”). Moreover, we find that the same general recipe is applicable to generalist policies with diverse architectures, including a large 3B VLA with a PaliGemma vision-language-model backbone. We open source the code and the models, as well as the dataset, which comprises 27k (!) action-labeled robot trajectories with visual, inertial, tactile, and auditory observations. This work is the result of an amazing collaboration at Berkeley Artificial Intelligence Research with the other co-leads Joshua Jones and Oier Mees, as well as Kyle Stachowicz, Pieter Abbeel, and Sergey Levine! Paper: https://lnkd.in/dDU-HZz9 Website: https://lnkd.in/d7A76t8e Code: https://lnkd.in/d_96t3Du Models and dataset: https://lnkd.in/d9Er5Jsx 

Here's how I built a Multimodal RAG with DeepSeek Janus (100% local with open-source tools)👇 I used: - Colpali to understand and embed docs using vision capabilities. - Qdrant as the vector database. - DeepSeek AI's latest Janus-Pro multimodal LLM to generate a response. Regarding the data: - I used this complex multimodal PDF with several complex diagrams, text within visualizations, and tables—perfect for multimodal RAG. The steps are simple: 1) Embed data ↳ We extract each document page as an image and embed it using ColPali. ↳ ColPali uses vision capabilities to understand the context. It produces patches for every page, and each patch gets an embedding vector. 2) Create a vector database ↳ Next, we create a Qdrant vector database and store these embeddings in it. 3) Set up DeepSeek Janus-Pro ↳ We download DeepSeek's latest Janus-Pro from HuggingFace. 4) Query vector database and generate a response ↳ We query the vector database to get the most relevant pages. ↳ Pass the pages (as images) along with the query to DeepSeek Janus-Pro to generate the response. Done! This gives us a powerful Multimodal RAG system that's running 100% locally. Find the code here: https://lnkd.in/dCZPWHVU --- If you want to learn AI/ML engineering, I have put together a free PDF (530+ pages) with 150+ core DS/ML lessons. Get here: https://lnkd.in/gi6xKmDc

Databricks just donated "Declarative Pipelines" to "Apache Spark". I couldn't agree more that declarative approaches make data pipelines simpler and more maintainable. Multimodal AI needs its own declarative revolution. After working with dozens of AI/ML teams for the past decade, we saw the same pattern everywhere: → 80% of development time spent on data infrastructure (video frame extraction, audio transcription, image preprocessing, document management...) → Every team maintaining the same embedding and vector index patterns → Zero standardization for lineage, versioning, or caching across modalities This is exactly why we built Pixeltable as a declarative infrastructure specifically for multimodal AI, from data storage to model execution. There is no other Python library or all-in-one open source solution providing such developer experience. Pixeltable is the only Python library that provides incremental storage, transformation, indexing, and orchestration of your multimodal data. Period. Here's what makes it truly different: 🎯 Declarative AI Functions - Define multimodal tables and data transformations, let Pixeltable figure out how to do it 📹 Native Multimodal - Video, images, audio, documents in one unified system alongside with structured data ⚡ Incremental Update - Caching, retries, versioning, and lineage for AI model outputs, embeddings and more... 🧠 Built-in Models & Python UDFs - YOLOX, CLIP, Whisper, OpenAI… and bring anything with User-Define Functions 100% open-source under Apache 2.0. Instead of writing hundreds of lines of orchestration code that adds 0 value to your business, app, data teams, simply declare what you want. pip install pixeltable, today.