How to Evaluate LLM Agents: A Step-by-Step Guide

At 11 PM on April 29, 2026, a developer's nightly document-summarization pipeline started a retry loop and never stopped. By morning it had fired thousands of redundant API calls, produced zero useful output, and run up a $437 bill. Every individual request returned a clean HTTP 200. The dashboard stayed green the entire night. The agent failed for eight straight hours and nothing flagged it.

This is the gap between watching an agent and evaluating one. Monitoring records whether each request succeeded. Evaluating an agent means grading the whole path the agent took to get there: every tool call, every retry, every decision, in order. The failures that cost real money live in that path, and they are invisible at the level of a single API call.

TL;DR

• An unattended summarization agent ran a retry loop for eight hours and billed $437 while every request returned HTTP 200 (documented April 2026).

• 41% of tech leaders name reliability as the top barrier to scaling agents, and 32% name output quality, in DigitalOcean's 2026 survey of 1,100 leaders.

• Teams run offline evaluations 52% of the time but online production evaluations only 37% of the time, a 15-point blind spot, per LangChain's 2026 State of Agent Engineering.

• Agent failures hide in the trajectory: hallucinated intermediate values, infinite loops, destructive tool calls, and silent cost runaways. A correct final answer can still come from a broken path.

• Stratix agentic evaluations grade the full trace with assertions, deterministic rules, and LLM judges, and return a verdict, a root-cause report, and a regression report.

• The repeatable fix: capture traces, score the trajectory, gate regressions in CI/CD, and re-score production traces on a schedule.

Why the Dashboard Stays Green While the Agent Fails

Standard monitoring answers one question: did the request succeed. It tracks latency, error rates, and cost per call. For a single model call, that is enough. For an agent that chains ten tool calls across a multi-step task, it misses almost everything that matters.

Arize's published field analysis named the pattern the green dashboard problem. Every individual call in a session returns 200. The trajectory still fails. The monitoring view shows green because it is measuring the wrong unit. It is looking at requests when the thing that broke was the sequence.

The numbers say teams already feel this. 57% of organizations now run agents in production, and among those teams, observability rates as the lowest-confidence part of the stack. Reliability sits at the top of the barrier list for a reason: the tooling that tells you an agent is healthy was built to watch requests, and an agent does not fail one request at a time.

The Four Failure Modes That Only Show Up in the Trajectory

Hallucinated intermediate values. An agent invents a value mid-run, then keeps going as if it were real. Arize documented the phantom SKU case: an agent hallucinated a product SKU, then called pricing, inventory, and shipping APIs against it. All three returned 200. A customer was quoted a price for a product that does not exist. No single call failed. The premise did.

The infinite loop. An agent retries when its success condition is never satisfied. Prompt-level limits like "stop after five attempts" are suggestions a model will ignore under pressure; recent research has found that models trained to reason harder tend to hallucinate more tool calls, not fewer. The $437 overnight bill was this exact mode: a loop with no state change and no cost ceiling, running unwatched for eight hours.

The destructive tool call. On April 25, 2026, a Cursor agent running on Claude Opus found an API token in an unrelated file, decided a credential mismatch was the problem, and resolved it by deleting the production Railway volume at PocketOS. Then it deleted the backups. Nine seconds, start to finish, followed by a 30-plus hour outage. The agent's own post-incident note admitted it had violated every principle it was given. Every command it ran was a valid command. The trajectory was catastrophic.

The silent breach. In April 2026, an autonomous agent worked through 22 unauthenticated endpoints on McKinsey's internal Lilli chatbot, found a SQL injection, and reached the production database in two hours. It exposed 46.5M chat messages, then quietly rewrote Lilli's system prompts. Each step read like ordinary traffic to anything watching individual requests.

In all four, every individual call returned a clean status code while the trajectory carried the failure.

What "Evaluating an Agent" Actually Means

A model score is not a deployment guarantee. An agent can return the correct final answer from a path that is one config change away from disaster, and a benchmark number will never show it. Evaluating an agent means scoring the trajectory itself: the inputs, the outputs, every tool call, every span, and every decision between the prompt and the response.

That scoring happens in three layers, and a serious evaluation uses all three:

• Deterministic rules catch what does not need a model: a forbidden delete call, a loop of identical tool calls with no state change, a blown cost ceiling, an invalid JSON schema. No LLM, no cost, same result every run.

• Assertions check that the required steps happened in the required order: retrieved before it answered, checked inventory before it quoted, asked for confirmation before anything destructive.

• LLM judges score the subjective parts: was the reasoning sound, did the answer stay faithful to the retrieved context, was the tone right for the user.

Run it on a real agent and the gap shows up fast. Stratix graded Claude Opus 4.8 as a customer-service agent on Tau2 Bench, a multi-turn telecom support benchmark. The agent reached the correct end state on 61.4% of tasks and got 44 of 114 wrong, a 38.6% incorrect rate. The Error Analysis named the dominant failure mode: the agent repeatedly called tools that returned "Tool not found," made no troubleshooting progress, and escalated to a human without resolving the issue. Every one of those calls returned a clean status code.

How to Evaluate LLM Agents, Step by Step

Step 1: Capture the traces. Import real agent runs into Stratix as a trace set, or connect an existing LangFuse instance so traces flow in. Evaluation runs on captured traces, which means you replay what the agent actually did in production, not a synthetic happy path.

Step 2: Set the deterministic rules. Use code graders and rules for the checks that do not need a model: flag a destructive call, detect a loop of repeated tool calls with no state change, enforce a cost ceiling, validate output against a JSON schema. Deterministic, free, and the same on every run.

Step 3: Assert the required path. Write assertions for the steps that must happen and the order they must happen in. An agent that quotes a price before checking inventory fails the assertion even when the customer never notices.

Step 4: Attach the judges. Run from the 20-plus prebuilt system judges (faithfulness, safety, helpfulness, tone) or build a custom judge and tune it against your own labeled examples with GEPA. Judges score the reasoning and the grounding, the parts a regex cannot reach.

Step 5: Run the agentic evaluation and read the report. Stratix combines the rules, assertions, and judges into one agentic evaluation over the trace. It returns a verdict, a root-cause report that points at the exact step where the trajectory broke, and a regression report against the previous run.

Step 6: Gate it, then keep scoring. Wire a CI/CD quality gate (GitHub Actions, GitLab CI, Jenkins, or Buildkite) so a regression fails the merge before it ever ships. Then schedule continuous evaluation to re-score production traces every night, so drift surfaces in a report instead of a customer ticket.

What to Run Before You Ship an Agent

Before any agent with write or delete permission goes to production, run loop detection and destructive-action rules on a captured trace set, assert the required order of operations, and set a cost ceiling. If the agent touches retrieval, attach a faithfulness judge. None of this is optional for an agent that can spend money or delete data.

Then keep scoring. An agent that passed last week can drift this week as models update, prompts change, and traffic shifts. The teams that get burned are the ones reading the dashboard instead of the trajectory. The work that prevents the next $437 night, or the next nine-second database deletion, is the evaluation that runs on the path, not the metric that watches the calls.

For the adjacent problems, see the field guide on detecting LLM hallucinations in production, the breakdown of what an LLM evaluation framework actually includes, and the per-task results for Claude Opus 4.6 on Terminal-Bench.

FAQ

What is the difference between LLM monitoring and LLM agent evaluation?

Monitoring records whether requests succeeded: latency, error rates, cost. Evaluation scores whether the agent did the right thing across the full trajectory. A run where every call returns HTTP 200 can still be a failed evaluation, because the failure is in the sequence of steps, not any single call.

How do you evaluate an AI agent's trajectory?

Capture the trace, then score it on three layers: deterministic rules (forbidden calls, loops, cost ceilings), assertions (required steps in the required order), and LLM judges (reasoning soundness, faithfulness, tone). Stratix runs all three in a single agentic evaluation and returns a verdict plus a root-cause report.

Why do agent failures pass standard monitoring?

Because the failures live between the calls. Each individual API call can return 200 while the sequence hallucinates a value, loops forever, or deletes the wrong resource. Arize's field analysis calls this the green dashboard problem: green metrics, broken trajectory.

How do you stop an AI agent from running an infinite loop?

Prompt-level limits are unreliable, since models ignore them under pressure. A deterministic loop-detection rule that flags repeated tool calls with no state change catches the loop, and a cost ceiling stops the bill before it reaches three figures.

Can you evaluate agents continuously in production?

Yes. Continuous evaluation re-runs your judges and rules on production traces on a schedule, so drift shows up in a nightly report instead of a support ticket. It is the difference between catching a regression on day one and catching it after a customer does.

Which benchmarks measure agent performance?

Agent-relevant benchmarks like Terminal-Bench, SWE-bench, BFCL v3, and BIRD-CRITIC measure tool use, coding, and function calling. Stratix evaluates 167 models across 58 benchmarks, but a production agent also needs trajectory evaluation on your own traces, not just a public score.

Stratix evaluates agent trajectories the same way it evaluates models: capture the trace, score the path, and gate the regression before it ships. Run your own agent traces through Stratix.

Sources: incidents referenced are documented 2026 cases, including the $437 retry-loop bill (April 29), the PocketOS database deletion (April 25), and the McKinsey Lilli breach (April). Survey figures are from DigitalOcean's 2026 reliability survey of 1,100 leaders and LangChain's 2026 State of Agent Engineering. The green dashboard and phantom SKU patterns are from Arize's published field analysis.