Assessing LLM Performance in Extended Projects

Many companies are diving into AI agents without a clear framework for when they are appropriate or how to assess their effectiveness. Several recent benchmarks offer a more structured view of where LLM agents are effective and where they are not. LLM agents consistently perform well in short, structured tasks involving tool use. A March 2025 survey on evaluation methods highlights their ability to decompose problems into tool calls, maintain state across multiple steps, and apply reflection to self-correct. Architectures like PLAN-and-ACT and AgentGen, which incorporate Monte Carlo Tree Search, improve task completion rates by 8 to 15 percent across domains such as information retrieval, scripting, and constrained planning. Structured hybrid pipelines are another area where agents perform reliably. Benchmarks like ThinkGeo and ToolQA show that when paired with stable interfaces and clearly defined tool actions, LLMs can handle classification, data extraction, and logic operations at production-grade accuracy. The performance drops sharply in more complex settings. In Vending-Bench, agents tasked with managing a vending operation over extended interactions failed after roughly 20 million tokens. They lost track of inventory, misordered events, or repeated actions indefinitely. These breakdowns occurred even when the full context was available, pointing to fundamental limitations in long-horizon planning and execution logic. SOP-Bench further illustrates this boundary. Across 1,800 real-world industrial procedures, Function-Calling agents completed only 27 percent of tasks. When exposed to larger tool registries, performance degraded significantly. Agents frequently selected incorrect tools, despite having structured metadata and step-by-step guidance. These findings suggest that LLM agents work best when the task is tightly scoped, repeatable, and structured around deterministic APIs. They consistently underperform when the workflow requires extended decision-making, coordination, or procedural nuance. To formalize this distinction, I use the SMART framework to assess agent fit: • 𝗦𝗰𝗼𝗽𝗲 & 𝗦𝘁𝗿𝘂𝗰𝘁𝘂𝗿𝗲 – Is the process linear and clearly defined? • 𝗠𝗲𝘁𝗿𝗶𝗰𝘀 & 𝗠𝗲𝗮𝘀𝘂𝗿𝗲𝗺𝗲𝗻𝘁 – Is there sufficient volume and quantifiable ROI? • 𝗔𝗰𝗰𝗲𝘀𝘀 & 𝗔𝗰𝘁𝗶𝗼𝗻𝗮𝗯𝗶𝗹𝗶𝘁𝘆 – Are tools and APIs integrated and callable? • 𝗥𝗶𝘀𝗸 & 𝗥𝗲𝗹𝗶𝗮𝗯𝗶𝗹𝗶𝘁𝘆 – Can failures be logged, audited, and contained? • 𝗧𝗲𝗺𝗽𝗼𝗿𝗮𝗹 𝗟𝗲𝗻𝗴𝘁𝗵 – Is the task short, self-contained, and episodic? When all five criteria are met, agentic automation is likely to succeed. When even one is missing, the use case may require redesign before introducing LLM agents. The strongest agent implementations I’ve seen start with ruthless scoping, not ambitious scale. What filters do you use before greenlighting an AI agent?

LLM-as-a-Judge is a dangerous way to do evaluation. In most cases, it just doesn't work. While it gives a false impression of having a grasp on your system's performance, it lures you with general metrics such as correctness, faithfulness, or completeness. They hide several complexities: - What does "completeness" mean for your application? In the case of a marketing AI assistant, what characterizes a complete post from an incomplete one? If the score goes higher, does it mean that the post is better? - Often, these metrics are scores between 1-5. But what is the difference between a 3 and a 4? Does a 5 mean the output is perfect? What is perfect, then? - If you "calibrate" the LLM-as-a-judge with scores given by users during a test session, how do you ensure the LLM scoring matches user expectations? If I arbitrarily set all scores to 4, will I perform better than your model? However, if LLM-as-a-judge is limited, it doesn't mean it's impossible to evaluate your AI system. Here are some strategies you can adopt: - Online evaluation is the new king in the GenAI era Log and trace LLM outputs, retrieved chunks, routing… Each step of the process. Link it to user feedback as binary classification: was the final output good or bad? Then take a look at the data. Yourself, no LLM. Take notes and try to find patterns among good and bad traces. At this stage, you can use an LLM to help you find clusters within your notes, not the data. After taking this time, you'll already have some clues about how to improve the system. - Evaluate deterministic steps that come before the final output Was the retrieval system performant? Did the router correctly categorize the initial query? Those steps in the agentic system are deterministic, meaning you can evaluate them precisely: Retriever: Hit Rate@k, Mean Reciprocal Rank, Precision, Recall Router: Precision, Recall, F1-Score Create a small benchmark, synthetically or not, to evaluate offline those steps. It enables you to improve them later on individually (hybrid search instead of vector search, fine-tune a small classifier instead of relying on LLMs…) - Don't use tools that promise to externalize evaluation Your evaluation system is your moat. If you externalize it, not only will you have no control over your AI application, but you also won't be able to improve it. This is your problem, your users, your revenue. Evaluate your system. Not a generic one. All problems are different. Yours is unique as well. ... Those are some unequivocal ideas proposed by the AI community. Yet, I still see AI projects relying on LLM-as-a-judge and generic metrics among companies. Being able to evaluate your system gives you the power to improve it. So take the time to create the perfect evaluation for your use case.

We've all shipped an LLM feature that "felt right" in dev, only to watch it break in production. Why? Because human "eyeballing" isn't a scalable evaluation strategy. The real challenge in building robust AI isn't just getting an LLM to generate an output. It’s ensuring the output is 𝐫𝐢𝐠𝐡𝐭, 𝐬𝐚𝐟𝐞, 𝐟𝐨𝐫𝐦𝐚𝐭𝐭𝐞𝐝, 𝐚𝐧𝐝 𝐮𝐬𝐞𝐟𝐮𝐥, consistently, across thousands of diverse user inputs. This is where 𝐄𝐯𝐚𝐥𝐮𝐚𝐭𝐢𝐨𝐧 𝐌𝐞𝐭𝐫𝐢𝐜𝐬 become non-negotiable. Think of them as the sophisticated unit tests and integration tests for your LLM's brain. You need to move beyond "does it work?" to "how well does it work, and why?" This is precisely what Comet's 𝐎𝐩𝐢𝐤 is designed for. It provides the framework to rigorously grade your LLM's performance, turning subjective feelings into objective data. Here's how we approach it, as shown in the cheat sheet below: 1./ Heuristic Metrics => the 'Linters' & 'Unit Tests' - These are your non-negotiable, deterministic sanity checks. - They are low-cost, fast, and catch objective failures. - Your pipeline should fail here first. ▫️Is it valid? → IsJson, RegexMatch ▫️Is it faithful? → Contains, Equals ▫️Is it close? → Levenshtein 2./ LLM-as-a-Judge => the 'Peer Review' - This is for everything that "looks right" but might be subtly wrong. - These metrics evaluate quality and nuance where statistical rules fail. - They answer the hard, subjective questions. ▫️Is it true? → Hallucination ▫️Is it relevant? → AnswerRelevance ▫️Is it helpful? → Usefulness 3./ G-Eval => the dynamic 'Judge-Builder' - G-Eval is a task-agnostic LLM-as-a-Judge. - You define custom evaluation criteria in plain English (e.g., "Is the tone professional but not robotic?"). - It then uses Chain-of-Thought reasoning internally to analyze the output and produce a human-aligned score for those criteria. - This allows you to test specific business logic without writing new code. 4./ Custom Metrics - For everything else. - This is where you write your own Python code to create a metric. - It’s for when you need to check an output against a live internal API, a proprietary database, or any other logic that only your system knows. Take a look at the cheat sheet for a quick breakdown. Which metric are you implementing first for your current LLM project? ♻️ Don't forget to repost.

🚨 Public Service Announcement: If you're building LLM-based applications for internal business use, especially for high-risk functions this is for you. Define Context Clearly ------------------------ 📋 Document the purpose, expected behavior, and users of the LLM system. 🚩 Note any undesirable or unacceptable behaviors upfront. Conduct a Risk Assessment ---------------------------- 🔍 Identify potential risks tied to the LLM (e.g., misinformation, bias, toxic outputs, etc), and be as specific as possible 📊 Categorize risks by impact on stakeholders or organizational goals. Implement a Test Suite ------------------------ 🧪 Ensure evaluations include relevant test cases for the expected use. ⚖️ Use benchmarks but complement them with tests tailored to your business needs. Monitor Risk Coverage ----------------------- 📈 Verify that test inputs reflect real-world usage and potential high-risk scenarios. 🚧 Address gaps in test coverage promptly. Test for Robustness --------------------- 🛡 Evaluate performance on varied inputs, ensuring consistent and accurate outputs. 🗣 Incorporate feedback from real users and subject matter experts. Document Everything ---------------------- 📑 Track risk assessments, test methods, thresholds, and results. ✅ Justify metrics and thresholds to enable accountability and traceability. #psa #llm #testingandevaluation #responsibleAI #AIGovernance Patrick Sullivan, Khoa Lam, Bryan Ilg, Jeffery Recker, Borhane Blili-Hamelin, PhD, Dr. Benjamin Lange, Dinah Rabe, Ali Hasan

LLM systems don’t fail silently. They fail invisibly. No trace, no metrics, no alerts - just wrong answers and confused users. That’s why we architected a complete observability pipeline in the Second Brain AI Assistant course. Powered by Opik from Comet, it covers two key layers: 𝟭. 𝗣𝗿𝗼𝗺𝗽𝘁 𝗠𝗼𝗻𝗶𝘁𝗼𝗿𝗶𝗻𝗴 → Tracks full prompt traces (inputs, outputs, system prompts, latencies) → Visualizes chain execution flows and step-level timing → Captures metadata like model IDs, retrieval config, prompt templates, token count, and costs Latency metrics like: Time to First Token (TTFT) Tokens per Second (TPS) Total response time ...are logged and analyzed across stages (pre-gen, gen, post-gen). So when your agent misbehaves, you can see exactly where and why. 𝟮. 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗼𝗻 𝗳𝗼𝗿 𝗔𝗴𝗲𝗻𝘁𝗶𝗰 𝗥𝗔𝗚 → Runs automated tests on the agent’s responses → Uses LLM judges + custom heuristics (hallucination, relevance, structure) → Works offline (during dev) and post-deployment (on real prod samples) → Fully CI/CD-ready with performance alerts and eval dashboards It’s like integration testing, but for your RAG + agent stack. The best part? → You can compare multiple versions side-by-side → Run scheduled eval jobs on live data → Catch quality regressions before your users do This is Lesson 6 of the course (and it might be the most important one). Because if your system can’t measure itself, it can’t improve. 🔗 Full breakdown here: https://lnkd.in/dA465E_J

I reviewed the literature on the LLM-as-a-judge technique. Here are the key findings. 📉 Model Performance Variability LLMs show inconsistent performance across datasets and tasks. No single model dominates all scenarios. GPT-4 generally leads, with open-source models like Llama-3-70B close behind. 👥 Alignment with Human Judgments LLMs correlate better with non-expert human judgments than expert annotations. Top models approach but don't match human-to-human alignment levels. Improved alignment comes mainly from increased recall, not precision. ⚖️ Evaluation Method Comparison Comparative assessment outperforms absolute scoring in robustness and accuracy. Reference-guided evaluation shows promise but has limitations. Simple methods sometimes unexpectedly outperform complex ones in specific tasks. ⚠️ Vulnerabilities and Biases LLMs struggle with toxicity, safety assessments, and basic perturbations like spelling errors. They show leniency bias and are susceptible to simple adversarial attacks, especially in absolute scoring scenarios. 🛑 Limitations of Fine-tuned Judges Fine-tuned models excel in-domain but lack generalizability and aspect-specific evaluation capabilities. They're prone to superficial biases and don't benefit from prompt engineering techniques (e.g., few-shot prompting). 👩⚖️ LLMs-as-a-jury Using LLMs-as-a-jury (PoLL) outperforms single large judges, reducing bias and cost while improving consistency across tasks. This approach mitigates intra-model favoritism. 💡 Practical Recommendations Use both quantitative metrics and qualitative analysis. Consider perplexity-based detection for adversarial inputs. Multiple judges are better than one. This is a technique used at scale to review models and filter data. It's imperfect, but a poor correlation with human judgment doesn't necessarily mean it's bad. Careful prompt engineering and multiple iterations can give you excellent results in most use cases.

🤔 As a generative AI practitioner, I spend a good chunk of time developing task-specific metrics for various tasks/domains and use-cases. Microsoft's AgentEval seems like a promising tool to assist with this! ❗ Traditional evaluation methods focus on generic and end-to-end success metrics, which don't always capture the nuanced performance needed for complex or domain specific tasks. This creates a gap in understanding how well these applications meet user needs and developer requirements. 💡 AgentEval provides a structured approach to evaluate the utility of LLM-powered applications through three key agents: 🤖 CriticAgent: Proposes a list of evaluation criteria based on the task description and pairs of successful and failed solutions. Example: For math problems, criteria might include efficiency and clarity of the solution. 🤖 QuantifierAgent: Quantifies how well a solution meets each criterion and returns a utility score. Example: For clarity in math problems, the quantification might range from "not clear" to "very clear." 🤖 VerifierAgent: Ensures the quality and robustness of the assessment criteria, verifying that they are essential, informative, and have high discriminative power. Turns out that AgentEval demonstrates robustness and effectiveness in two applications: math problem-solving and household tasks and it outperforms traditional methods by providing a comprehensive multi-dimensional assessment. I want to try this out soon, let me know if you've already used it and have some insights! #genai #llms

Evaluating long-memory agents is rarely discussed, but it is a crucial application of LLMs that ensures a personalized experience when interacting with them. Here are two of the most interesting papers on this topic ⭐️ 1️⃣ Evaluating Very Long-Term Conversational Memory of LLM Agents from Snapchat 2️⃣ Beyond Prompts: Dynamic Conversational Benchmarking of Large Language Models from GoodAI The first challenge in this area is the lack of open datasets available for evaluating long-memory agents. Both papers begin by proposing methods to create synthetic data for this purpose. The Snapchat team built LoCoMo [1] by first synthesizing personas, then generating temporal events in each persona’s life (ordered chronologically), and finally synthesizing conversations that included multimedia document sharing, connecting personas and events. GoodAI’s approach involves creating hardcoded scenarios, such as ordering food from a restaurant, and establishing a predefined correct path that the agent should follow if it has successfully memorized the events and instructions. Once the datasets are created, both works evaluate agent performance by defining different query types. Snapchat’s work focuses on traditional single-hop, multi-hop, and temporal reasoning queries, among others [1]. GoodAI’s work incorporates tests inspired by cognitive theory, such as episodic memory and conflicting information handling[2]. For metrics, both papers use a combination of traditional methods like ROUGE and newer LLM-based evaluation techniques. In both studies, the authors also observed instances where LLMs, acting as judges, failed due to alignment issues.