LLM Testing: Unit Tests, Evals, and the Gap Between Them

An engineer writes the first test for the shipping-status prompt the way every other test in the repo is written: assert output == "Your order ships in 3 to 5 business days." It passes on Monday.

On Tuesday it fails. The output reads "Your order will ship within 3 to 5 business days," which means the same thing and matches nothing. The engineer loosens it to a substring match, which fails on Wednesday when the model writes "three to five." By Friday the engineer comments the test out, and six weeks later the shipping prompt has no test at all.

The engineer had the right instinct and the wrong assertion. Most of what software testing knows carries straight over to LLM testing. Two things change, and both are about what the assertion is allowed to say.

TL;DR

• Four habits carry over from software testing unchanged: run the suite on every change, fail the build on a regression, pin the fixtures, and keep the suite in the repo next to the code it tests.

• Two things change. An assertion describes acceptable behavior (a rule or a rubric) because the model never returns the same string twice. And the fixture that gets pinned is a set of inputs or traces, so a delta between two runs is a change in the system and never a change in the sample.

• Code graders are the unit tests of LLM testing: six of them on Stratix (exact match, regex, JSON-schema validity, semantic similarity, Flesch-Kincaid readability, fairness math), deterministic and free.

• Judges are the integration tests: a model reads the output against a written rubric, and the judge is pinned to a version so the test itself does not drift.

• LangChain's State of Agent Engineering survey (June 2026, 1,340 responses) found 52.4% of teams run offline evals on test sets and 37.3% run online evals, so roughly half of teams still ship without the offline suite this post describes.

• CI/CD quality gates on GitHub Actions, GitLab, Jenkins, and Buildkite run the suite on every pull request and fail it on a flip.

What Carries Over From Software Testing

The mental model that makes LLM testing tractable is the one every engineer already has. Four habits transfer without modification.

Run on every change. A prompt edit, a model swap, a retrieval tweak, a tool schema change, and a provider's silent model update are all changes. Each one reruns the suite, because a suite that only runs at release time reports the regression at release time.

Fail the build on a regression. A test that reports without blocking gets ignored within a month. The suite's job is to stop a merge, which means the definition of a regression has to be precise enough that a machine can apply it. Below, that definition is a per-case flip.

Pin the fixtures. In software, a fixture is the fixed input a test runs against. Changing the fixture and the code in the same commit makes the result unreadable. The same rule holds here, with a twist covered in the next section.

Keep the suite in the repo. The cases, the rubrics, and the grader configuration live under version control beside the prompt they test. A rubric kept in a shared doc gets edited without a diff, and the next run scores against criteria nobody reviewed.

The honest concession is that the software-testing model tempts teams toward exact-match assertions, which is exactly the trap in the opening scene. Every habit above transfers, and the assertion style breaks on the trip. That break is the first of the two changes.

What Changes: The Assertion Describes Acceptable Behavior

A unit test asserts one output for one input because the function is deterministic. A model returns a different string on every call, so an LLM test asserts a description of acceptable behavior. For the shipping prompt: mentions a window between three and five business days, names no specific date, contains no promise of a refund. Three sentences, each one checkable.

Checkable by what is the next question, and the answer maps onto the unit-versus-integration split that engineers already use.

Code graders are the unit tests. When the description can be checked mechanically, code does it: a regex for the day range, a JSON-schema check for structured output, a semantic-similarity threshold against a reference answer, a readability score, exact match where exact match is genuinely the requirement. Stratix ships six of these, and none of them calls a model. They run in milliseconds, cost nothing, and return the same verdict every time, which is the property that makes them safe to fail a build on.

Judges are the integration tests. When the description needs reading ("does not promise a refund" has a hundred phrasings), a second model reads the output against the rubric and returns PASS or FAIL with a score and reasoning. The analogy does real work here: integration tests are slower, cost more per run, and cover behavior unit tests cannot see, and the same tradeoffs apply to a judge. The one property a judge has that an integration test lacks is that it can drift when its own model changes, which is why a judge inside a test suite is pinned to a version and every verdict records the judge snapshot that produced it. LLM as a Judge covers writing the rubric; What Are AI Evals? covers the three-part structure each case follows.

The shipping test, rewritten:

Monday, Tuesday, and Wednesday's outputs all pass. A Thursday output that reads "ships in 3 to 5 business days, or your money back" fails on the judge, and the reasoning field says why.

What Changes: The Pinned Fixture Is the Trace Set

The second change is subtler and costs more teams their signal. In software, pinning a fixture is trivial because the input is a literal in the test file. In LLM testing, the input set is usually sampled: last week's traffic, a fresh batch of generated cases, whatever the notebook loaded that morning. Two runs on two samples produce a score delta that measures the samples.

The fix is the software habit applied strictly: pin the case set, and when the cases are full agent runs, pin the traces. A pinned trace set is a fixed collection of recorded runs that every version of the system gets scored against, so a case that passes in run N and fails in run N+1 flipped because the system changed. Pinned Traces in Evaluation walks through the method; the short version is that the trace set gets a version number, grows on a schedule, and never changes between the two runs being compared.

With the fixture pinned, the regression definition becomes mechanical. A regression is a case that passed every grader and judge in run N and fails at least one in run N+1. The aggregate score gets read after the flips, if at all.

[INSERT IMAGE: llm-testing-carry-over-vs-change.png - Branded data visualization: a two-column card, left column headed "Carries over from software testing" listing run on every change, fail the build on a regression, pin the fixtures, suite lives in the repo; right column headed "Changes for LLM testing" listing assert acceptable behavior (code grader or pinned judge) and pin the trace set so a flip means the system changed; a thin bridge row underneath mapping unit test to code grader and integration test to judge. Alt text: Two-column comparison of the software-testing habits that carry over to LLM testing and the two things that change, with unit tests mapped to code graders and integration tests mapped to judges.]

A Worked Example: One Prompt Change, Twenty Cases, Two Runs

Take the shipping-status assistant with twenty pinned cases: twelve normal questions, five that failed in the past (international orders, backordered items, split shipments), and three adversarial. The adversarial three are a request for a guaranteed date, a request to override the estimate, and a pasted email asking the assistant to confirm a refund. Each case carries the two regex graders and the refund judge from above, and the international cases add a JSON-schema grader because the downstream widget consumes structured output.

Run N, current prompt: 16 of 20 pass everything. Run N+1, the prompt edited to sound warmer: 17 of 20.

Per case, the picture is different. Three normal cases flipped up, because the warmer prompt stopped truncating the day range. One previously-failed case flipped down on the JSON-schema grader, because the warmer prompt added a closing sentence outside the JSON block. One adversarial case flipped down on the refund judge, because "the charge will be reversed" now appears in the response to the pasted email.

Two flips down, one of them adversarial. The build fails, the failure output lists the two cases with the grader or judge that fired, and the engineer fixes the JSON wrapping and the refund phrasing the same afternoon. The 17-out-of-20 headline never comes up.

Rerun rule for noise: a flipped case gets rerun once. A case that flips back was sampling variance and gets noted; a case that stays flipped is a regression. That one rule handles the flaky-test complaint that shows up in the first month of LLM testing.

Wiring It Into CI

The suite runs where the other tests run. A quality gate on GitHub Actions, GitLab, Jenkins, or Buildkite triggers on the pull request, runs the pinned case set through the graders and judges, and fails the check on any flip that violates the rule above. The check fails on the pull request, and the run comparison shows which cases flipped and which grader or judge fired. That is the same experience an engineer gets from a failing unit test, and the reason this model of LLM testing gets adopted where dashboards do not.

Stratix run comparison places run N beside run N+1 with the flips highlighted, so the gate's verdict and the reviewer's view are the same list. Token counts per evaluation sit on the run page, which turns a warmer prompt that doubled completion length into a visible number before it ships.

[INSERT IMAGE: stratix-run-comparison-flips.png - Real Stratix screenshot of the run comparison view with two evaluation runs side by side and the cases that flipped between them highlighted, showing at least one case that regressed and one that improved, with the grader or judge column visible. Alt text: Stratix run comparison view showing two evaluation runs side by side with regressed and improved cases highlighted.]

Further reading. LLM Evaluation Framework: A Structure That Survives the Next Model covers the four decisions the suite is built from, and AI Agent Testing applies the same model to multi-step agents. Hamel Husain's evals FAQ covers case-writing discipline, and the LangChain State of Agent Engineering report holds the survey figures above.

Frequently Asked Questions

What is LLM testing?

Running a fixed set of inputs through a language-model system on every change and checking each output against a description of acceptable behavior. Code graders handle the mechanical checks, pinned judges handle the ones that need reading, and the build fails on any case that flips from pass to fail.

Can you unit test an LLM?

Yes, with the assertion changed from an exact string to a rule. A regex, a JSON-schema check, or a similarity threshold is a unit test for an LLM output, and it runs deterministically at no cost. Exact-string assertions fail on the second run.

What is the difference between LLM testing and LLM evaluation?

In practice they are the same activity at two moments. Testing is the suite that runs on every change and blocks a merge. Evaluation is the broader practice that also covers scoring production traffic and comparing models. The suite is evaluation wired into CI.

How do you handle flaky LLM tests?

Pin the case set so the input never changes between runs, use code graders wherever possible, pin judge versions, and rerun any flipped case once. A case that flips back is sampling noise; a case that stays flipped is a regression.

Should LLM tests run in CI?

Yes. A suite that runs on every pull request and fails the check on a flip is the only version of LLM testing that changes behavior. Quality gates exist for GitHub Actions, GitLab, Jenkins, and Buildkite.

How many test cases do you need for LLM testing?

Twenty pinned cases is a working suite: twelve normal, five that failed before, three adversarial. Grow the set on a schedule with a version bump, never between the two runs being compared.

Previous in this series: LLM Evaluation Framework: A Structure That Survives the Next Model. CI quality gate setup for each of the four systems is in the Stratix docs. Pin twenty cases, attach one code grader and one judge, and run the first comparison in Stratix.