LLM Benchmarks Explained: What the Scores Mean
Author:
The LayerLens Team
Last updated:
Published:
[VIDEO EMBED: Evals 101, Episode 5, once the YouTube upload is live. Leave out if the post publishes first.]
Suppose a team runs one model through SWE-bench Verified twice, two weeks apart, on the same harness and the same task set. The two scores land 2.2 points apart, and the median latency on the second run is several times the first. Nothing on the model card changed between the runs.
A person who sees only the higher number calls it a stronger model. A person who sees only the lower number calls it a weaker one. Both are reading a fact about one run as if it were a fact about the model.
TL;DR
A benchmark is a fixed task set with a fixed grader. A score is what happened when one model, configured one way, ran through that task set on one day.
Five inputs decide the number: model version and endpoint, decoding config, harness, task set version, and grader. Change any one and the score moves.
The EvalEval Coalition audited 101,955 published results across 638 benchmarks and found 96.5 percent missing at least one field needed to reproduce the setup. Sampling temperature was absent from 93.9 percent.
Three questions sort any score in the wild: which task set and version, who ran it, whether the decoding config is published. Two answers make it useful, three makes it reproducible.
Internal numbers need the same discipline. Record eleven values with every run or the pass rate in a March Slack message is unrecoverable by September.
A fully documented public score narrows the shortlist. The team's own twenty cases decide what ships.
A Benchmark Is a Property of the Test and a Score Is a Property of the Run
A benchmark is a fixed set of tasks with a fixed way to grade them. SWE-bench Verified is a set of real GitHub issues, and the grade is whether the model's patch passes the repository's tests. BFCL is a set of function-calling prompts, and the grade is whether the call the model produced matches the expected one. The shape is the same as the twenty-case eval set from LLM Evaluation for Beginners, at a scale of hundreds or thousands of cases, shared across everyone who wants to run it.
A score is a different kind of object. It records what happened when one specific model, configured one specific way, went through that task set under one specific harness on one specific day. The benchmark stays fixed while the score belongs to the run.
Keeping those two apart is most of the skill in reading benchmark results. A benchmark can be well designed and a score on it can still be uninformative, because the score arrived with none of the details that would let anyone repeat it.
Five Inputs Decide the Number
Every score is the output of five inputs, and most published scores name one of them.
Model version and endpoint. A family name such as "Qwen3.7 Max" identifies a family. The provider and the endpoint, together with the run date, decide which weights answered the prompts. Two endpoints serving the same family name can run different quantizations or different snapshots.
Decoding config. Temperature, top-p, max output tokens, and the reasoning budget. Each one moves the score. A run at temperature 0 and a run at temperature 0.7 on the same model are two different experiments.
The harness. The scaffolding around the model: system prompt, tool definitions, retry logic, timeouts, how many attempts count. Two harnesses on the same model produce two different agents, and the harness is the input most often left out of a headline.
The task set and its version. Which tasks, which release, which subset. SWE-bench Verified and SWE-bench Lite share a name and little else. A benchmark that gets a new release changes its own difficulty.
The grader. What counts as a pass, and who decided. A test-suite pass is deterministic. A judged grade carries the judge model and the judge prompt as two more hidden inputs.
Take the two runs from the opening and compare them on these five. If all five match, the 2.2-point gap is run-to-run variance, and the honest number for that model on that benchmark is a range with both runs in it. If one input differs, that input is the explanation and belongs in the same sentence as the score.
[INSERT IMAGE: llm-benchmarks-explained-five-inputs.png - Branded data visualization: two run cards for the same model on the same benchmark, scores 2.2 points apart, with a five-row table beneath them. Rows: model version and endpoint, decoding config, harness, task set version, grader. Each row shows a value for run A and run B and a match or differ marker. Alt text: Diagram comparing two benchmark runs of one model across the five inputs that decide a score, with the input that differs highlighted.]
Most Published Scores Arrive With Four of Five Inputs Hidden
The EvalEval Coalition audited 101,955 reported evaluation results, covering 638 benchmarks from 31 organizations. In that set, 96.5 percent of results were missing at least one field needed to reproduce the setup. Sampling temperature was absent from 93.9 percent of them. And 98.2 percent of model-benchmark pairs had exactly one party reporting a result, which means roughly one published number in twenty-eight had a second source anyone could check it against.
Three Questions to Ask Before Trusting a Score
Every time a score crosses a feed, three questions sort it.
Which task set, and which version? "SWE-bench" alone is a family name, the same way a model name is. The answer needs the variant and the release.
Who ran it, and is that the same party that built the model? A lab's own number and a third party's number are different kinds of evidence, and both have a place. They should never be averaged together or read as confirming each other without checking the other four inputs.
Is the decoding config published? Without a temperature and a token limit nobody can rerun the score, so nobody can tell whether it would hold.
Two of three answered makes a score useful for choosing what to try. All three makes it reproducible. Zero means skip it and wait for a better source. Even a fully reproducible public score only narrows the shortlist; the team's own 20 cases decide what ships.
Record Eleven Values With Every Run
Everything above applies to a team's internal numbers. A pass rate posted in Slack in March, with no config attached, cannot be defended in September when a new model release raises the question of whether the old number still holds.
Record these eleven values with every evaluation run.
Model name and full version string.
Provider and endpoint.
Run date and time.
Temperature and top-p.
Max output tokens and reasoning budget.
System prompt, as full text or a hash of it.
Tool definitions and harness version.
Retry and timeout settings.
Task set name, version, subset, and prompt count.
Grader or judge name and version.
Input and output token counts and latency per case.
The first ten reproduce the run. The eleventh makes the score comparable, because a model that passes by spending three times the tokens costs three times as much per pass, and the score alone hides that.
LayerLens records most of this without setup. A benchmark evaluation detail page carries the model, the benchmark version, the prompt count, the sampling settings, and the score on one screen, and the run page shows token counts per evaluation. For judged runs, every verdict stores a judge snapshot with the judge name, version, evaluation goal, and the judge model used, so value ten is pinned to the result and does not drift when the judge is edited later. The live post on pinned traces covers the same idea from the trace side: hold the inputs fixed so the comparison means something.
[Marin question: confirm the benchmark evaluation detail page shows model, benchmark version, prompt count, and sampling settings next to the score, and which of the eleven values LayerLens records without setup.]
[INSERT IMAGE: stratix-benchmark-evaluation-detail.png - Real LayerLens screenshot of a benchmark evaluation detail page. The configuration fields are visible (model, benchmark version, prompt count, sampling settings) alongside the score. Alt text: LayerLens benchmark evaluation detail page showing the model, benchmark version, prompt count, and sampling settings recorded next to the final score.]
Reading a Score Is a Habit and Recording One Is a Checklist
The same rule covers both directions. When reading someone else's number, ask which of the five inputs are visible and weight the number accordingly. When publishing a number, whether in a paper or a Slack channel, attach the eleven values so the next reader can ask the same questions and get answers.
Handled this way, a score works as a record. Two records of the same model on the same benchmark that disagree by 2.2 points are useful, because the record shows whether the runs differed on an input or the model differed with itself. One number on a slide, with the inputs missing, can only be believed or ignored.
Frequently Asked Questions
What is an LLM benchmark?
A fixed set of tasks with a fixed grader, shared so that many models can be run through the same test. SWE-bench Verified (real GitHub issues, graded by the repository's tests) and BFCL (function-calling prompts, graded by whether the call matches) are two examples.
What does an LLM benchmark score mean?
It records what one model, configured one way, scored on one version of the task set under one harness on one day. The score describes that run. Reading it as a property of the model requires knowing the other inputs matched.
Why does the same model get different scores on the same benchmark?
Because one of five inputs differed: model endpoint or snapshot, decoding config, harness, task set version, or grader. If all five match, the gap is run-to-run variance and the honest number is a range.
Can you compare scores across different benchmarks?
No. Each benchmark has its own difficulty and its own grader, so a 70 on one and a 40 on another say nothing about which task is harder for the model. Compare scores only within one benchmark at one version, with matching inputs.
How should a team record its own benchmark runs?
With eleven values per run. The first ten reproduce it: model and version, provider and endpoint, date, temperature and top-p, token limits and reasoning budget, system prompt, tool definitions and harness version, retry and timeout settings, task set with version and count, grader or judge version. The eleventh, per-case token counts and latency, makes it comparable.
Are self-reported benchmark scores reliable?
They are one kind of evidence, and a third-party run of the same setup is another. Neither replaces the other. A self-reported score with all five inputs published is more useful than a third-party score with none.
Next in Evals 101: AI Agent Evaluation: How to Test Your Agents leaves single answers behind and grades an agent that took twelve steps to reach one. Further reading: What Are AI Evals? A Plain-English Guide, the live post on pinned traces, and the benchmark evaluation guides in the LayerLens docs. Run a benchmark with all five inputs recorded in LayerLens.