Comet

Most eval methods stop at single responses. But real reliability comes from evaluating whole conversations. That is why "Building Conversational AI Agents with Thread Level Eval Metrics" was one of the hands-on workshops selected by our volunteer Steering Committee for the 6th Annual MLOps World | GenAI Summit (Oct 8-9, Austin) Co-hosted by Tony Kipkemboi, Head of Developer Relations, CrewAI, and Claire Longo, Lead AI Researcher, Comet, this session will show you how to: ✅ Use CrewAI to define multi agent workflows and tool integrations ✅ Apply Comet Opik to design thread level eval metrics that capture full conversations ✅ Combine orchestration and evaluation into a repeatable workflow that improves agent quality If you're a serious AI technologist this session will help you and your bot projects reach the next level. DYK? MLOps World is the only major AI tech event programmed by practitioners, for practitioners. You’ll find hands-on workshops, technical deep dives, and real-world case studies hosted by some of the world’s most esteemed AI teams, including JFrog, TikTok, DICK'S Sporting Goods, Google DeepMind, Outerbounds, and Fujitsu. Check out the full agenda and get tickets (few still available): https://lnkd.in/gVbbyAR2

In modern software development, we don't just guess if our code works. We write unit tests, run integration tests, and build CI/CD pipelines. We replaced manual guesswork with rigorous, automated validation. So why are many of us still in the "guesswork" phase with LLM prompts? The common workflow is a manual loop : tweak a prompt, test it, eyeball the result, and tweak it again. This is artisanal, slow, and doesn't scale. A prompt that works today might break tomorrow with a slight model update. It’s not an engineering discipline. The paradigm shift we need is Systematic Prompt Optimization. This is the move from "prompt art" to "prompt science." It’s about treating a prompt not as a magic incantation, but as a key component of a system that can be algorithmically tested, measured, and improved. The framework for this is surprisingly simple and powerful: 1./ Hypothesis (Your Base Prompt): Your initial, best-guess prompt. 2./ Ground Truth (An Evaluation Dataset): A set of inputs and ideal outputs that define success for your use case. 3./ Objective Function (An Evaluator): A measurable score for success (e.g., accuracy, semantic similarity, factuality). 4./ Optimizer: An algorithm that intelligently searches the vast space of possible prompt variations to find the one that maximizes your objective function. This approach is a repeatable, data-driven process. It allows you to prove why one prompt is better than another and ensures your system is robust. I've been exploring frameworks that enable this, and Comet's Opik is a fascinating, concrete example of this principle in action. It provides the optimizer and structure to automate this entire loop. Check here: https://lnkd.in/dZEfCW6S By adopting this mindset, we're not just writing better prompts. We're building more reliable, maintainable, and predictable AI systems. What steps is your team taking to bring more engineering discipline to your work with LLMs? #llm #ai #optimization #agents

This week, Ollie joined our product and engineering teams in Rome for their R&D offsite 🇮🇹 Nearly a year ago we launched Opik. Like Rome, great products aren’t built in a day -- they take vision, persistence, and teamwork. We're grateful for this group’s dedication. Here are a few moments from their well-deserved week in the Eternal City.

Can’t make it to one of our in-person events? We’re kicking off a new virtual workshop series 🥳 The first one is on Sept 24 with Claire Longo, who’s leading a session on creating evals and feedback loops for conversational AI agents. Come learn with other builders and see how to: ✔️ Log traces that actually show what your app is doing ✔️ Design LLM-as-a-Judge metrics that mimic human reasoning rsvp 🔗 https://luma.com/uupy7jxr

🦾 Interested in learning how to evaluate conversational Agents using Comet Opik?Comet is hosting a series of virtual workshops, and the first one is all about creating evals and feedback loops for conversational #AI Agents. When: September 24th, 2025 Where? Virtual and free! In this workshop, we will demonstrate how to log traces, annotate sessions with expert insights, and design LLM-as-a-Judge metrics that mimic human reasoning, turning domain expertise into a repeatable feedback loop. Join us for a great conversation. My favorite part of these workshops is the live Q&A where we can share ideas and learn from eachother 👯 . 👉 rsvp: https://luma.com/uupy7jxr

Excited for this year's MLOps World Summit! See you in Austin in a little more than a month 🤠 🚀 

Congratulations to Medhaswi Paturu on winning Comet’s raffle at Tuesday night’s AI Tinkerers Demo Day in NYC! 🎉 We love celebrating curiosity and creativity in our community. Join us at an upcoming event for your chance to win a Star Wars LEGO set and connect with other builders pushing the boundaries of #AI. 🚀

This summer we welcomed 5 new members to the Comet team across the US, UK, and Greece ☀️ Each one brings impressive experience to their role and unique passions outside of work, from board games to golf, hiking, and even professional soccer. We’re so glad to have Peter, Steven, James, Stephen, and Nasos on the team. Please join us in welcoming them! 👋 Nasos Lamprokostopoulos James Gonzalez Peter Gray Steven Miller Stephen McGhie

Exciting news — our video with the Google Partners team is now live on the Google Cloud YouTube Channel! We’re proud to be part of the ISV Startup Springboard program, officially available as a marketplace offering deployed on GCP. Big thanks to the Google Cloud team and our own team at Comet for making this happen 🙌