What a benchmark score actually measures

Copper melts at 1,085°C. Any lab on any continent gets that number, with any thermometer, and nobody has to publish their equipment list to be believed. The number belongs to the copper.

Claude Opus 4.8 scores 81.8 on SWE-bench Verified. That sentence is built the same way, which makes it easy to read as a fact about the model. It is a fact about one run, on one day, under conditions nobody wrote down.

LayerLens ran Qwen3.7 Max against SWE-bench Verified twice this June on Stratix, same mini-swe-agent harness both times. The scores came back 70.6 and 72.8. The two runs also differ in a way neither score mentions: median content latency of 596 ms on the first and 3,869 ms on the second, with p95 moving from 2.4 seconds to 15.5.

Nothing in the record says which of those two scores is correct. It does show that the two runs were not doing the same thing, and a published score almost never lets a reader get that far.

TL;DR

• The same model on the same benchmark with the same harness scored 70.6 and 72.8 on two consecutive June runs on Stratix, with median content latency 6.5x apart between them.

• Five inputs decide a benchmark number: model version and endpoint, decoding config, harness, task set, and grader. The model is one of them.

• Across 101,955 reported eval results covering 638 benchmarks from 31 organizations, 96.5% are missing at least one field needed to reproduce the setup. Sampling temperature is absent 93.9% of the time.

• 98.2% of model-benchmark pairs are reported by exactly one party, so roughly one published result in twenty-eight could be reopened by another team.

• OpenAI audited SWE-bench Pro in July and withdrew its own recommendation: an automated pipeline flagged 200 of 730 tasks as broken (27.4%), human reviewers put it at 249 (34.1%).

• Recording eleven configuration values alongside every score is what separates a number you can defend in six months from one you quietly re-run.

A score is the output of a setup

A benchmark score comes out of a setup. Five parts of that setup decide the number, and the model is one part.

The model version and endpoint: the exact build behind the name. Providers change what sits behind a stable alias, so a number collected in May can fail to reproduce in July under the same name.

The decoding config: temperature, top-p, max tokens, and whether reasoning was switched on. Temperature decides how often a model commits to its highest-probability answer, which moves the score on any task with one correct response. It is also the field most often missing from published results, so it is usually the first thing a reader cannot check.

The harness: the scaffold around the model. How many tool calls it gets, how the prompt is assembled, how failures are retried, how much of the repository or screen it can see. The machine underneath counts too. Anthropic's engineering team measured this and found a 6-point gap on Terminal-Bench 2.0 between the most and least resourced machines, p<0.01, with the model, the harness and the task set all held constant. Sprinting has handled this for a century: a tailwind over 2.0 metres per second voids the record, and nobody calls that an attack on the sport.

The task set: which subset, in which order, how many. Teams usually run a slice, and a slice behaves differently from the whole.

The grader: whether a test suite or a model decides correctness, and how strictly. A binary grader scores a nearly-correct answer the same as a blank one. When the grader is itself a model, two properties get collapsed into one: reliability, whether it gives the same answer twice, and validity, whether that answer is right. A bathroom scale reading three kilos heavy is perfectly reliable and wrong every time, and its consistency is what stops you noticing. Across 21 judges and 541,000 judgments, raw agreement with human labels ran 33 to 41 points above chance-corrected Cohen's κ on MT-Bench, so a judge advertised at 85% agreement is somewhere between 44% and 52%.

Change any of the five and the number moves. That is how measurement works everywhere. The gap is that almost nobody writes the five down.

Almost no published result can be reopened

The EvalEval Coalition published an audit of eval reporting in June. Across 101,955 reported results covering 638 benchmarks from 31 organizations, 96.5% are missing at least one field needed to reproduce the published setup. Sampling temperature is absent 93.9% of the time. Max tokens, 95.6%. The median benchmark ships documentation that is 10.7% complete.

Nobody is checking the ones that do get published, either. Exactly one party reports 98.2% of model-benchmark pairs, and three organizations put GPT-5's MATH-500 result across an interval of 84.7% to 98.9%, a 14.2-point spread on one model and one benchmark. That leaves roughly one published result in twenty-eight that anyone else could reopen.

When the task set is the problem

That is the configuration problem. The tasks themselves can also be wrong, and OpenAI documented that twice this year.

In February it said SWE-bench Verified no longer measured frontier coding capability and recommended SWE-bench Pro instead. On July 8 it audited SWE-bench Pro and withdrew that advice. An automated pipeline flagged 200 of 730 tasks as defective, 27.4%. Human reviewers put it at 249, or 34.1%.

Four ways a task breaks:

• The test is stricter than the prompt. It enforces an implementation detail nobody asked for, so a correct solution fails on a variable name or a return shape.

• The prompt is underspecified. The author left out a requirement the hidden tests enforce, so the model cannot pass because nothing told it what passing meant.

• The test has no coverage. It runs green without exercising the requested behavior, so an empty stub scores as a win.

• The prompt misleads. The task text steers toward the wrong behavior, then grades the model for going there.

Graders fail the same quiet way. Faros AI checked its own rubric-based judge against the test results on a subset of SWE-bench Pro and found 152 cases where the two disagreed. Seventy-one were near misses: answers that were substantially correct and scored zero, because binary grading has no way to say "almost."

A task set with a third of its tasks broken still produces a clean-looking number, and nothing about the number shows the damage. So the only place to catch it is the task set itself. Pull twenty tasks out of your own golden set and put those four questions to each one. It costs about an hour, and the sets that have gone longest without that hour are usually the ones a team leans on hardest.

Record the score and the recipe together

Eleven values, every run: model version, endpoint, temperature, top-p, max tokens, reasoning setting, prompt template version, harness name, harness version, task subset and ordering, and the date.

If a team keeps only one, keep the harness version. Without it nobody can reconstruct the conditions the model ran under, which is the difference between a number you can defend six months from now and one you will quietly re-run.

Frequently Asked Questions

Does a benchmark score tell you nothing at all?

It tells you what one system scored under one configuration. That is useful when the configuration travels with the number. A score quoted without its setup cannot be compared to a score produced under a different setup, which is what most leaderboard comparisons are doing.

Which of the five inputs moves the number most?

It depends on the task type, but the harness and the machine underneath it are the most commonly underestimated. Anthropic measured a 6-point gap on Terminal-Bench 2.0 between the most and least resourced machines with the model, harness and task set held constant.

How many times should a benchmark be run before the result is trusted?

Enough times to see the spread. Two runs of Qwen3.7 Max on SWE-bench Verified came back 2.2 points apart on Stratix. A single run gives you a point with no error bar, and a two-point improvement inside that spread is not evidence of anything.

What is the difference between a reliable judge and a valid one?

Reliability is whether the judge gives the same answer twice. Validity is whether that answer is right. A judge can be perfectly consistent and consistently wrong, and its consistency is what hides the error. Raw agreement figures conflate the two, which is why chance-corrected Cohen's κ runs 33 to 41 points lower on the same judges.

How do you check whether your own task set is broken?

Pull twenty tasks from the golden set and ask four questions of each: is the test stricter than the prompt, is the prompt underspecified, does the test actually exercise the behavior, and does the prompt steer toward the wrong answer. About an hour of work.

Why does temperature matter if the model is deterministic at temperature zero?

Most published results are not run at temperature zero, and 93.9% of them do not say what temperature was used. Temperature decides how often the model commits to its highest-probability answer, which moves the score on any task with a single correct response.

Run evaluations you can reopen

Stratix records the configuration alongside the score on every run, so a number produced today can be reconstructed and defended months later. See how evaluations run on Stratix.

This article is adapted from Trace Evidence, a weekly read on AI evaluation from LayerLens. Sources are linked inline. Stratix run data comes from evaluations 6a29ed5f and 6a2b64fb, run 10 and 12 June 2026.