Today we’re announcing ❄️ Snowglobe - the simulation engine for AI chatbots! Snowglobe makes it easy to simulate realistic user conversations at scale so you can reveal the blind spots where your chatbots fail, and generate labeled datasets for finetuning them. We built Snowglobe to solve a problem that we ran into again and again through our journey building Guardrails for the last two years — evaluating AI agents is very challenging. If you spend days and weeks manually creating test scenarios for your chatbots, Snowglobe generates hundreds of realistic user conversations in minutes. How do you even formulate a test plan for evaluating something that can take infinite inputs? How do you deal with the many edge cases that break AI chatbots in prod all the time? Interestingly, self driving cars had the exact same problem. They built high fidelity simulation environments to systematically test cars under a wide range of scenarios. Waymo had 20+ million miles on real roads, but 20+ BILLION miles in sim so they had the confidence needed to ship. Today, we’re excited to bring that same tooling to AI agents with the general availability of Snowglobe!
Shreya Rajpal this is amazing! Can I use snowglobe to run such a simulation in the design phase itself so that I can design the system with much more knowledge of the edge cases ?
I hate having to manually test and break chatbots. This makes so much sense. Congrats on the launch Shreya Rajpal !!!
Such a great direction.. compelling, needed and not something most teams will build themselves (regardless of which of the 29 ai agent frameworks they are using, they need this and it’s hard to build!)
This is super cool congrats Shreya Rajpal I think the challenging part is also to figure out _why_ there was a failure It could come from pure LLM hallucination, wrong tool use, wrong context
Congrats Shreya! 🎉
Impressive Shreya. How do I test Snowglobe?
Congrats Shreya! 🎉
Addressing edge cases is important for chatbot success. Snowglobe helps uncover those tricky situations.
Congrats Shreya Rajpal and Guardrails AI team. Xuedong D. Huang fyi
AI/ML | NEXT AI, CDL Alumni
1moAmazing work Shreya Rajpal and the team! 🎉