Trace Evidence #08: Can You Trust Your AI Agent Audit?

An evaluation that cannot detect its own failure modes is not an evaluation.

A benchmark reports 0.99 accuracy. A production dashboard holds at 0.87 for five straight weeks. Nothing in either number tells you the benchmark scorer is ignoring its own test inputs, or that the production system has collapsed to three output categories. The evaluation instrument has failure modes of its own, and they are silent: the score holds steady while the thing it measures falls apart. Two studies published this year opened the pipeline from opposite ends and found the same defect.

The Benchmark Side: Zero of Ten Cells Pass

On the benchmark side, Li, Fan, and Zhuang applied a six-point due-diligence gate to five safety benchmarks (TruthfulQA, BBQ, ToxiGen, CrowS-Pairs, XSTest) across two open-weight instruction-tuned models. Zero of ten model-benchmark cells reached confirmatory status.

There were serious failures. On TruthfulQA and BBQ, the canonical multiple-choice scorer returned identical answers for every perturbation: a 100% no-op rate. Perturbations existed. The scorer ignored them. For Qwen-2.5-7B on TruthfulQA, only 44% of outputs were parseable under the legacy renderer. The remaining 56% defaulted to an incorrect answer. On CrowS-Pairs, an inverted scoring convention made Mistral-7B's 0.99 accuracy represent maximal stereotypical preference, the opposite of what the benchmark was built to measure.

Five failure classes appeared across the ten non-confirmatory cells: silent no-op perturbations, regex-extraction artifacts, non-faithful scoring (three subtypes: inverted convention, harness data ordering, scorer truncation), broken bootstrap pairing, and metric archetype mismatch. Three cells were ineligible due to fundamental scoring or pipeline defects, three had unvalidated scorers, two failed statistical gates, and two reached exploratory status only.

The Production Side: Accuracy Holds While Everything Else Breaks

On the production side, Pandey tracked seven failure modes in deployed agentic systems. In one experiment, accuracy held between 0.86 and 0.88 across five weekly measurement windows. Output diversity collapsed from 0.200 to 0.030 over the same period, a 6.5x reduction. Categories dropped from 20 to 3. The repeat rate at the final window reached 1.000. The system was returning correct answers, just three kinds of them, over and over. In a second experiment on tool degradation, external accuracy shifted by only 0.03 across four stages while partial response rates climbed from 4% to 58%. ROUGE and BERTScore caught none of the seven failure modes on their own. Accuracy and AUC flagged three, each with a detection lag of one to two evaluation cycles.

The Evaluation Needs Its Own Evaluation

Five failure classes on the benchmark side, seven on the production side, and the headline metric caught none of them in time. A smoke detector with dead batteries looks exactly like one that works. So does a benchmark that scores 0.99 while its scorer ignores every perturbation.

Did the perturbation actually change the output? Does the scorer's convention match the benchmark's definition? Is the metric tracking what the system actually does? Li, Fan, and Zhuang's six-point gate is one design for benchmarks. Pandey's PAEF framework is one design for production. An evaluation pipeline without a validity layer checking the evaluation itself is grading its own homework.