Why Most Agent Evals Fail Before They Start

There's a growing consensus in the AI engineering community that evaluating agents is fundamentally different from evaluating traditional LLM responses. Yet most guides on AI agent evaluation still treat it like a chatbot benchmark problem: send a prompt, get an output, score the output. That approach misses almost everything interesting about how agents actually fail.

Agents fail mid-trajectory. They call the wrong tool. They hallucinate a parameter value. They loop. They abandon a subtask silently and return a confident but incomplete answer. None of that shows up if you're only checking the final response.

This guide is for engineers and technical builders who are shipping agents into production, or who are about to. It covers how to design evaluation frameworks that reflect how agents actually behave, not how you hope they do.

What Makes Agent Evaluation Different

A standard LLM eval is straightforward: you have an input, an expected output or set of acceptable outputs, and a grading function. That model breaks down with agents for three reasons.

First, agents take multi-step actions. A research agent might run five tool calls before producing a final answer. The final answer could look correct even though three of those tool calls were wrong, redundant, or dangerous. Evaluating only the output gives you a false positive.

Second, agent behavior is non-deterministic at the trajectory level. Two runs with identical inputs might use completely different tool sequences and still produce equivalent final answers. Your eval framework needs to handle this gracefully, without penalizing valid alternative paths.

Third, the cost of errors compounds. In a chatbot, a bad response is a bad response. In an agent that's writing files, calling APIs, or sending emails, a bad decision at step 2 can corrupt everything downstream. Catching failures early in the trace matters much more than catching them at the output stage.

The Three Layers of Agent Evals

A solid evaluation framework for production agents operates across three layers simultaneously. Anthropic's engineering team describes these as code-based, model-based, and human graders, which is a useful framing. Here's how each layer works in practice.

Layer 1: Deterministic Checks

These are your cheapest, fastest, most reliable evals. They don't use an LLM to judge. They check for things that are objectively correct or incorrect.

Examples: Did the agent call the right tool? Did the function call include all required parameters? Was the response returned in the expected schema? Did the agent terminate within the step budget? Did it avoid calling a tool it was explicitly restricted from using?

Deterministic checks should run on every eval and every CI pass. They catch the obvious failures instantly. If your agent is supposed to query a database and return structured JSON, a code-based check that validates the schema catches 40% of your failures for essentially zero cost.

Layer 2: LLM-as-Judge

For things that can't be reduced to a binary check, you use a second model to grade the output or trajectory of your agent. This is what the community usually calls LLM-as-judge, and it's both powerful and easy to misuse.

The key mistake is using a weak or poorly-prompted judge. A judge that simply asks "is this response helpful?" produces noisy, unreliable scores. A better judge prompt is narrow and specific: "Given the user's task and the agent's tool calls, did the agent retrieve the correct information before formulating its answer? Answer yes or no, then explain."

Model-based grading works well for evaluating reasoning steps, checking whether tool use was appropriate given context, and assessing response quality on open-ended tasks. Use a stronger model than your agent for judging, or at minimum a model with different training, to reduce confirmation bias.

The paper "AI Agents That Matter" (Kapoor et al.) makes an important point here: many published evaluations optimize for benchmark performance while ignoring cost and reliability. Your internal evals should explicitly measure cost-per-run and error rate as first-class metrics, not afterthoughts.

Layer 3: Human Review

Human review doesn't mean manually checking every agent run. It means having a structured process for reviewing a sample of traces, especially when your automated evals report confidence scores rather than hard pass/fail results.

In practice, this looks like: randomly sample 5-10% of production traces per week. Flag any run where the LLM judge score fell below a threshold. Review flagged traces in a shared interface where the full tool call sequence is visible. Use those reviews to refine your judge prompts and add new deterministic checks.

Human review is how you catch the failure modes that your automated evals haven't seen yet. Think of it as the source of your next set of deterministic checks.

Trajectory Evaluation: The Part Most Teams Skip

Output evaluation tells you if the agent got the right answer. Trajectory evaluation tells you if the agent got there the right way. For production agents, the trajectory matters as much as the output.

Consider a customer support agent that's supposed to look up an order status, check for any recent issues in the system, and then respond to the customer. If it skips the second tool call and answers anyway, the output might look fine most of the time. But for customers with an active incident, it gives a completely wrong answer. Output-only evaluation won't catch this consistently.

Trajectory eval involves checking the sequence of tool calls and deciding whether it's valid given the task. Some useful trajectory-level checks:

• Required steps: Did the agent call tool A before tool B when the task requires that ordering?

• Unnecessary steps: Did the agent make redundant calls that inflate cost and latency?

• Forbidden actions: Did the agent attempt any tool call that was out of scope?

• Loop detection: Did the agent repeat the same tool call more than N times with no progress?

Arize AI's agent evaluation documentation covers trajectory-level grading in detail if you want to go deeper on tooling options for this layer.

Designing Test Cases That Actually Find Bugs

The quality of your eval suite depends almost entirely on the quality of your test cases. Most teams start with happy-path examples: inputs where the agent is expected to succeed. Those are necessary but not sufficient.

Good agent eval suites include at least four categories of test cases:

Happy path cases. Standard inputs where the agent should succeed. Baseline performance.

Edge cases. Unusual but valid inputs. What happens when the user's request is ambiguous? When required information is missing? When multiple tools could answer the question but only one is correct?

Adversarial cases. Inputs designed to make the agent fail. For customer-facing agents, this includes prompt injection attempts, requests to perform actions outside the agent's scope, and contradictory instructions. Prompt injection is a serious and growing attack surface: the OWASP LLM Top 10 lists it as the number one vulnerability for LLM applications.

Regression cases. Any failure that made it to production gets added to this suite. Once you fix a bug, you write a test for it. This suite only grows over time and is your primary defense against regressions.

When building these cases, start from your actual user data if you have it. Synthetic test cases are useful, but they consistently miss the weird, specific, real-world inputs that break your agent in production. Pull a sample of real traces from your first week of deployment and manually classify the ones that went wrong. Those become your first adversarial cases.

Metrics Worth Tracking

Once your eval framework is running, you need to know what to measure. A few metrics that actually matter for production agents:

Task completion rate. Did the agent complete the assigned task? This sounds obvious but defining "complete" precisely is harder than it seems, and forces clarity about what your agent is actually supposed to do.

Tool call accuracy. Of all tool calls made, what percentage were correct (right tool, right parameters, right sequence)? Track this per tool if you have multiple tools. Degradation in a specific tool often signals a prompt change interacted badly with how that tool's description is written.

Step efficiency. Average number of tool calls per successful task completion. Creep in this number usually signals prompt degradation or a model update changed behavior quietly.

Failure mode distribution. Track categories of failure, not just a single pass/fail rate. Knowing that 60% of your failures are "wrong tool selected" versus 40% "correct tool, wrong parameters" tells you exactly where to focus engineering effort.

Latency and cost per eval. Your eval suite itself has a cost. If running evals on every PR takes 45 minutes and costs $50, developers will skip them. Optimize your eval pipeline the same way you'd optimize production code.

Common Mistakes in Agent Evaluation

A few patterns that consistently lead teams astray:

Using only your agent's own output to evaluate itself. Self-evaluation is convenient and cheap, but it inherits the same biases and blind spots as the agent. An agent that confidently gives wrong answers will also confidently grade those answers highly.

Evaluating a single snapshot of the agent without tracking eval scores over time. A pass rate of 87% is meaningless without knowing whether it was 92% last week. Time-series tracking of eval metrics is how you detect regressions from model updates, prompt changes, or data drift.

Treating evals as a one-time activity. Eval suites need to evolve with your agent. Every new feature, every new tool, every model upgrade should trigger a review of existing tests and addition of new ones. Teams that write evals once at launch and never update them consistently discover production failures their suite would never catch.

Over-relying on public benchmark scores. Standard benchmarks like GAIA or WebArena measure specific capabilities in controlled environments. They're useful for comparing models but they don't tell you whether your specific agent, with your specific tools and prompts, works for your specific users. Build internal evals that reflect your actual use case.

Fitting Evals Into a CI Pipeline

Evals that only run manually aren't really evals, they're spot checks. The goal is a suite that runs automatically on every significant change: new prompt versions, model upgrades, new tool additions, deployments to production.

A practical structure for an agent eval CI pipeline:

1. Run deterministic checks first. Fast, cheap, catches obvious regressions immediately. Fail the build if any deterministic check fails.

2. Run LLM-as-judge on a representative subset of test cases. 50-100 is often enough to detect major regressions. Report scores but don't necessarily block the build on them, because LLM judge scores carry variance.

3. Run the full eval suite, including adversarial cases, on a schedule rather than every PR. Weekly is common for stable agents, daily for ones under active development.

4. Log all eval results with timestamps, commit hash, and model version. This data is what lets you diagnose regressions weeks or months after the fact.

Evaluation connects directly to the broader reliability and observability question for agents. If you're thinking about how evals fit into a larger production monitoring strategy, the post on AI Observability and Reliability Engineering for Agentic Systems covers the runtime monitoring layer that complements your eval work.

What Good Actually Looks Like

A mature agent eval setup has a few defining characteristics. The eval suite runs in CI. Every production failure generates a new test case. Eval scores are tracked over time and someone reviews the trend weekly. The team can answer, without pulling logs manually, questions like: "Did the model update we deployed on Tuesday change our task completion rate?" and "Which tool is responsible for the most failures this month?"

Getting there from zero takes a few weeks of deliberate work. Most teams start with just deterministic checks and a handful of LLM-judge test cases, which is a perfectly reasonable starting point. The important thing is to begin with the right framework so you can add coverage incrementally without rebuilding everything later.

The agents that behave reliably in production aren't usually built on better models. They're built by teams who understand where their agents fail and have systems in place to catch it before users do.