The test your agent never took

Before an aircraft engine gets certified, the FAA makes it start 100 times. The endurance rules require at least 150 hours of running, with 25 of those starts after 2-hour shutdowns, and a separate test throws rain and hail into the intake. Trust comes from repetition. An agent headed for production gets cleared by a different kind of test: a benchmark score, pass@1, which gives each task a single attempt and counts the wins. The single attempt is the norm, and it mostly goes unexamined: in a review of 445 LLM benchmarks, only 16% ran any statistical test on their scores. 3 new benchmarks brought repetition to agents, running the same tests again and again under rougher conditions. The scores dropped, some by more than half.

TL;DR

• Microsoft's Thinkingbox benchmark runs agents through 507 stateful business workflows 20 times each: the strongest model passes 65.36% of tasks once but only 25.25% of tasks all 20 times.

• Stratix holds 2 recorded runs of Qwen3.7 Max on SWE-bench Verified (mini-swe-agent, 500 tasks): headline scores 70.6% and 72.8%, but 53 tasks (10.6% of the suite) changed verdict between the runs.

• The same 2 runs support any headline from 66.4% (passed both) to 77% (passed at least once), depending on the repeatability standard applied.

• AnTrap tested 16 GUI agent models against 10 categories of runtime anomalies: every model degraded, with a top performer falling from 74.2% to 66.5%.

• LongRCA Bench: across 1,140 real failed agent trajectories, the best training-free method locates the exact failing step 24.1% of the time.

• Before trusting a pass rate, ask for the repeat rate, the flip list, and the failing step.

Run it 20 times and 65% becomes 25%

Microsoft built Thinkingbox, a sandbox of 507 business workflows across retail, hospitality, insurance, banking, and IT support, the kind of work where an agent has to hold state and follow policy across many turns. Every task ran 20 times. The strongest model passed 65.36% of tasks on the first try. Held to passing all 20 tries, it kept 25.25%.

The drop compounds step by step: an agent that gets each of 30 steps right 98% of the time finishes a 30-step workflow 54.5% of the time. The certification bar for the engine was 100 starts. This test asked for 20.

2 identical runs, 53 different verdicts

One of those rerun logs already exists, sitting in the Stratix record. It holds 2 recorded runs of Qwen3.7 Max on SWE-bench Verified under mini-swe-agent, a minimalist bash-only coding agent harness, 500 human-verified software issues per run, same model, same harness, same suite. The 2 headline scores are 70.6% and 72.8%, which reads as a tidy 2.2-point wobble.

[INSERT IMAGE: te05-two-runs.png - Branded chart: same model, same benchmark, two different runs]
Image URL: https://mcusercontent.com/82dd3f8e78273bcc7c72a2f8c/images/c2578ec3-f59f-d7c6-cea2-ecc87d266f69.png

The per-task records underneath say more: 53 of the 500 tasks changed verdict between the 2 runs, 21 from pass to fail and 32 from fail to pass. So the tasks that passed both runs number 332, or 66.4%, and the tasks that passed at least once reach 385, or 77%. The same 2 runs can honestly support any headline from 66% to 77%, depending on how much repeatability the reader demands. A caveat belongs here: some of those 53 flips could be infrastructure at work, a timed-out request here or a flaky test step there, with the model itself blameless. The records cannot separate the 2 causes yet, and that cuts the same way: whatever moved the verdicts, a single run would have hidden the movement entirely.

[INSERT IMAGE: stratix-compare-models-results.png - Real Stratix screenshot: per-prompt comparison view for 2 evaluation runs read side by side]
Image URL: https://i.imgur.com/g38XsIz.png

Add pop-ups and 74% becomes 67%

Clean conditions are the other problem. AnTrap added the interruptions a real phone produces, ad popups, system alerts, rendering glitches, frozen screens, and ran 16 GUI agent models through Android tasks seeded with them. All 16 dropped. The top performer went from 74.2% to 66.5%, and the frozen-screen failures did not improve with training. Production is full of interruptions the test never shows.

The failing step gets found 1 time in 4

Then there is the failure itself. Picture a 145-step agent run that fails somewhere in the middle. Which step broke it? LongRCA Bench built the answer key for exactly this question: 1,140 real failed runs, each labeled by human annotators with the part of the agent system that failed and the earliest wrong step. Automated methods then tried to find those labels. The best one found the failing part 51.1% of the time and the exact step 24.1% of the time. That is the success rate with the full trace saved. A step-level trace is the black box of an agent run, and most runs fly without one, which is why the usual response to a failed run is a shrug and a rerun.

All 3 point the same way: rerun the test, disturb it, keep the trace. Objections to the increased cost of multiple runs can be short-sighted. The 2nd Stratix run alone exposed a 10.6% churn, and catching that before production protects the brand and every decision resting on the score. The expensive habit is promoting an agent on a number that changes when you look at it twice.

Before trusting a pass rate, ask for the 3 numbers it hides: the repeat rate (same suite, N runs, tasks that passed every time), the flip list (which tasks changed verdict between runs), and the failing step (where the failed runs died). An agent that survives all 3 questions is ready for passengers.

Frequently Asked Questions

What is the difference between pass@1 and pass^k?

pass@1 is the share of tasks an agent completes on a single attempt. pass^k requires the agent to complete the same task on all k attempts. Thinkingbox reports both: 65.36% pass@1 against 25.25% pass^20 for the strongest model it tested, which makes pass^k the closer proxy for production reliability.

How many reruns are enough?

Thinkingbox uses 20. Even 2 runs of a 500-task suite exposed a 10.6% verdict churn in recorded Stratix data, so the 2nd run already buys most of the signal, and more runs tighten the estimate. Budget-permitting, 3 to 5 runs separate stable tasks from coin flips.

Why do agent scores vary between identical runs at all?

Model sampling is stochastic, and multi-step workflows compound it: small per-step variation multiplies across dozens of steps. Environment state, tool latency, and timeouts add more variation on top.

Does temperature 0 fix run-to-run variance?

No. Greedy decoding reduces one source of randomness, but serving-stack nondeterminism, tool responses, environment state, and timing effects still vary between runs, and agent trajectories diverge after the first differing token.

What should an eval record keep so failures can be attributed?

Step-level traces. With human-labeled ground truth on 1,140 failed trajectories, LongRCA Bench found the best automated method locates the exact failing step 24.1% of the time. Attribution needs the full trajectory recorded per run, per task, per step.

Where do repeat runs live in Stratix?

Every evaluation run is stored as its own record with per-task results, so rerunning a suite produces comparable records. The comparison in this post came from 2 such records read side by side.

Run your own repeat evaluations at stratix.layerlens.ai.