The Compounding Math Engineers Miss When They Dismiss Agent Failures
Author:
The LayerLens Team
Last updated:
Published:
TL;DR
A 2025 study analyzing 1,642 multi-agent traces across 7 production frameworks found failure rates of 41-86.7% in real systems, not in adversarial test conditions.
A 5% per-step failure rate across a 10-step agent produces a 40% full-run failure rate. The math is multiplicative, not additive.
68% of practitioners in a cross-domain survey cap agents at 10 or fewer autonomous steps before requiring human review. Reliability is the stated reason.
Engineers dismiss individual step failures because they experience them as isolated. The system-level failure rate is a derived number that does not appear in a debugger.
Trajectory-level evaluation scores the complete run. Step-level benchmarks do not.
There is a specific conversation that happens at teams building with AI agents for the first time. An engineer runs a benchmark, the model scores 92%. Someone raises a concern about reliability. The engineer says 92% is good enough to ship. The thing ships. Three weeks later, support tickets arrive that do not match what the benchmark predicted.
The gap is compounding math. It is not intuitive. And because it is not intuitive, it keeps getting dismissed.
What 41-86.7% Failure Rates Look Like in Practice
In March 2025, researchers from Berkeley, MIT, and Stanford published MAST: the first systematic failure taxonomy for multi-agent LLM systems. They analyzed 1,642 annotated execution traces across 7 popular multi-agent frameworks and found:
Failure rates ranged from 41% to 86.7% depending on the framework and task type.
Not in adversarial conditions. In standard benchmark runs across coding, math, and general agent tasks.
The failure modes clustered into three categories: system design issues, inter-agent misalignment, and task verification failures. Specification failures alone accounted for 41.8% of failures. Coordination breakdowns accounted for 36.9%.
The researchers' conclusion: "performance gains on popular benchmarks are often minimal" for multi-agent systems, and the failures "require more sophisticated solutions" than the current evaluation apparatus catches.
The Arithmetic Behind the Dismissal
The reason individual engineers dismiss step-level failure rates comes down to where the system-level number lives: not in the debugger, but in a derived calculation they rarely make.
An engineer debugging a failed step sees a single failure. They fix it. They move on. They see another failure on a different step. They fix that. The collection of step-level failures looks manageable (each one is a concrete bug to address).
What they are not seeing: the probability that any given full run completes without error.
The math is straightforward. If each step in a multi-step agent fails at rate f, the probability that a complete N-step run succeeds is (1 - f)^N.
At 5% per-step failure rate:
5 steps: 23% of runs fail
10 steps: 40% of runs fail
20 steps: 64% of runs fail
The 90% step accuracy that feels solid produces 65% full-run success at 15 steps. One in three users hits a failure the benchmark did not surface.
This is not a novel finding. A 2025 Zartis analysis of the compounding errors problem in multi-agent systems reaches the same conclusion: the per-component metric and the system-level metric tell different stories.
Why "Everyone Says This" Does Not Fix It
The compounding failure message sounds like a sales line because it gets repeated in every piece of content about AI reliability. When it is repeated without new evidence, it stops registering.
The MAST study provides the new evidence. 1,642 real traces. 7 production frameworks. 41-86.7% failure rates in practice, not in theory. That number deserves attention not because it is alarming in the abstract but because it explains what teams are already seeing in their support queues.
A separate practitioner survey across 306 respondents in 26 domains found that 68% cap agents at 10 or fewer autonomous steps before requiring human review. The primary stated reason: reliability. Teams have already internalized the problem through painful production experience. The evaluation practice has not caught up.
What Changes When You Score Trajectories
Stratix agentic evaluations score the trajectory: a complete run from first prompt to final output, with every tool call, branch decision, and intermediate result scored.
The number that appears is trajectory success rate: the fraction of complete runs that meet the evaluation criteria end to end.
When that number is 60%, the compounding math is visible in the output. When per-step accuracy is 90%, the compounding math requires a calculation the engineer is unlikely to do in a code review.
The goal is not to replace step-level debugging. Both are necessary. Trajectory-level evaluation makes the system-level risk legible before it shows up in production.
Frequently Asked Questions
Is this specific to multi-agent systems or does it apply to single-agent pipelines too?
Any multi-step system has the same compounding math. Single-agent pipelines with tool calls, retrieval steps, or multi-turn loops compound in the same way. MAST focuses on multi-agent systems, but the arithmetic applies to any sequential process where steps fail independently.
What per-step failure rate should we target?
It depends on how many steps are in the pipeline and what failure means for users. A 3-step pipeline with 2% per-step failure rate produces a 94% full-run success rate (probably acceptable). A 15-step pipeline with the same rate produces 74% (probably not). The right question is what run-level success rate your use case requires, then work backwards to per-step targets.
Do structured outputs between steps reduce compounding?
Validators reduce per-step failure rates by catching format errors and some logic errors at step boundaries. They do not catch semantic failures (the right format, the wrong content) and they do not prevent subtly wrong upstream outputs from propagating. Trajectory-level evaluation is still necessary.
We cap agents at short runs. Does that solve this?
Shorter pipelines have lower compounding risk. The practitioner survey finding (68% cap at 10 or fewer steps) reflects this. But it also reflects a design constraint driven by reliability concerns: teams are limiting capability to manage risk they cannot fully measure. Better trajectory measurement enables longer reliable runs.