AI regression testing: your quality gate passes while the agent gets worse

AI regression testing rests on a comparison between two runs. A pass-rate floor catches a collapse. Decay walks through it, and the sample gate most teams start from cannot tell the difference.

An agent scored 0.94 on its evaluation set in June. In July it scores 0.86. The CI gate is set to 0.85, the pipeline goes green, and the change merges. Eight percentage points of quality left the product while every automated check reported success.

The gate behaved correctly the whole time. A threshold answers whether the system sits above a floor right now, and it holds no memory of any earlier run, so it has nothing to say about the question a team asks after two months in production. Those are separate measurements. Most teams have built only the first one.

What makes this hard to notice is that the failure produces no error. A code regression throws a stack trace or trips an assertion. An agent drifting from 0.94 to 0.86 returns well-formed answers the whole way down, support tickets rise a few weeks later, and nobody connects them to a merge from three sprints back.

Turning a floor into a regression check takes three things: a test set that holds still between runs, a sample large enough to resolve the change you care about, and a judge that has not moved underneath the comparison. Reading the LayerLens CI sample closely shows that the version most teams will copy has none of the three wired up.

TL;DR

• The LayerLens Python SDK ships a floor gate at samples/cicd/quality_gate.py. It scores recent traces against every configured judge and exits 1 when the aggregate pass rate falls under --threshold, which defaults to 0.85.

• That sample calls client.traces.get_many(page_size=50), which returns the fifty most recent traces. Two runs a month apart score two different populations, so the comparison is confounded before it starts.

• Sizing the run is where most gates go wrong. At 540 scored results a three-point drop is exactly significant at a one-sided 5% level, but you would only catch a real three-point drop half the time. Catching it 80% of the time takes about 1,240 scored results per run. The sample caps total evaluations at 200, and fifty traces against one judge gives a one-sided false-alarm rate near 31% on a three-point band.

• The SDK's built-in diff, client.public.comparisons.compare(), works on benchmark evaluations rather than trace evaluations. It answers whether a model changed. It does not diff two runs of your own agent.

• Every trace evaluation carries a judge_snapshot with the judge name, version and evaluation goal, which is how you detect the judge moving underneath a comparison.

Why a floor is not AI regression testing: at 0.85, a 0.94 looks the same as a 0.86

Every score from 0.851 upward reads identically to the pipeline. A run at 0.94 and a run at 0.86 both print PASSED. The gate answers a question about the floor while the team reads the green check as an answer about the trend.

Two failure shapes live in that gap. Slow decay drifts a few points per release until the score crosses the floor six months later, at which point the gate finally fires and nobody can identify which of forty merges caused it. Sudden decay drops eight points in one release and still clears the floor, so a signal that was perfectly attributable arrives labeled as a pass.

The second shape is the expensive one, because all the information needed to catch it existed in the run. The gate discarded it by comparing against a constant and leaving the previous run unused.

[INSERT IMAGE: IMG-1 - Pass rate declining from 0.94 to 0.86 across six releases, staying above the 0.85 threshold line.]

What the shipped sample actually does

The LayerLens Python SDK is at version 1.8.0. The package name is layerlens even though the client class is Stratix, and the CLI sits behind an optional extra. This install gives you both the library and the stratix binary:

pip install --extra-index-url https://sdk.layerlens.ai/package "layerlens[cli]

pip install --extra-index-url https://sdk.layerlens.ai/package "layerlens[cli]

pip install --extra-index-url https://sdk.layerlens.ai/package "layerlens[cli]

The script builds a Stratix() client, pulls recent traces through client.traces.get_many(), pulls every configured judge through client.judges.get_many(), then creates one trace evaluation per trace and judge pair with client.trace_evaluations.create(). It polls each result, counts passes against failures, prints a summary, and calls sys.exit(1) when the aggregate pass rate lands under the threshold. A MAX_EVALUATIONS constant caps the run at 200 pairs to hold down API volume.

The bundled workflow at samples/cicd/github_actions_gate.yml runs it on pull requests into main that touch src/, prompts/ or agents/, with a fifteen minute job timeout and the threshold carried in a PASS_RATE_THRESHOLD variable.

One practical warning the workflow does not carry: creation and polling both run sequentially, and the SDK's own helper documents judge execution at five to sixty seconds. Two hundred pairs will not finish inside fifteen minutes at either end of that range: even five seconds per pair is nearly seventeen minutes. Size the pull request gate well under the cap and put the large run somewhere without a timeout.

[INSERT IMAGE: IMG-2 - Stratix evaluation results view showing per-prompt pass and fail outcomes with scores.]

The sample re-samples its own test set every run

This is the defect to catch before anything else, because it invalidates the comparison instead of merely adding noise to it.

client.traces.get_many(page_size=max_traces) returns the most recent traces in the project. Run the gate in June and it scores June's traffic. Run it in July and it scores July's traffic. The two pass rates describe different populations of user behavior, so a drop between them measures some mixture of agent quality and whatever changed about who was using the product.

For a floor check that behavior is defensible, because "are recent interactions above the bar" is a reasonable production question. For a regression check it removes the fixed harness the whole comparison depends on.

The fix is to stop letting recency choose the test set. Curate a stable set of traces, store their IDs in the repository next to the code, and pass those IDs explicitly instead of taking whatever get_many hands back. A regression gate needs the same inputs every time, for the same reason a unit test suite does.

Sizing AI regression testing: fifty traces cannot detect a three-point drop

The second missing piece is knowing how large a change has to be to mean anything.

Pass rate on n scored results behaves like a binomial proportion. Take a true pass rate near 0.90. The standard error of a single run is the square root of p(1 minus p) over n. Comparing two runs adds a second source of the same error, so the standard error of the difference is larger by a factor of the square root of two:

Now apply a three-point failure band. With fifty scored results, the standard error of the delta is six points, so a perfectly healthy run trips a three-point band about 31% of the time. Roughly one pull request in three fails for no reason, and the gate gets switched off inside a month.

Running it the other direction gives the number to design against, and this is the step where the arithmetic usually stops one line too early. Setting the band so a three-point drop clears a one-sided 5% threshold needs the standard error of the delta near 1.8 points, which lands at about 540 scored results. That is the significance calculation, and on its own it is not a detection guarantee. At exactly 540 the three-point drop sits right on the critical value, so a real three-point regression is caught about half the time. Sizing for detection means adding a power term, and adding it puts 80% power against a three-point drop at roughly 1,240 scored results per run.

Put differently, 540 results will reliably catch a 4.5-point drop and coin-flip on a three-point one. Both numbers are well past the sample's 200-pair cap and far past anything that belongs in a pull request.

One point of vocabulary, because these are easy to conflate. The drop is the regression you want to catch and it stays fixed at three points. The band is the threshold that fires the alarm, and it shrinks as the sample grows: three points at 540 results, two points at 1,240. Sizing the run tightens the band, which is the entire reason to run more.

The pull request gate therefore stays a floor check on a small sample, because that is the only job a small sample can do honestly. The regression check moves to a scheduled run against a curated set large enough to resolve the effect you care about.

That split has an obvious cost, and it should be named plainly: a nightly job cannot block the merge that caused the problem. The regression is already on main by the time the run finishes. The schedule buys attribution, and prevention is a separate purchase. A nightly run against a fixed set narrows the cause to the commits from one day, which is the difference between reverting one of six changes and bisecting forty merges six months later. Make the nightly failure page someone, record which commit range it covers, and treat a confirmed drop as a revert candidate instead of filing a ticket. Teams that need prevention at merge time have to pay for it in latency, which means a curated set large enough to matter running on the pull request and a much slower pipeline.

Raising the sample also breaks the shipped sample's execution model, and this is the part worth checking before committing to a number. The sample creates and polls sequentially. At the five to sixty seconds per pair the SDK documents, 1,240 pairs runs between 1.7 and 20.7 hours. The slow end does not fit in a night. A nightly regression run at this size needs concurrent creation and polling, and at a mid-range twenty seconds per pair, sixteen workers bring 1,240 pairs to roughly twenty-six minutes. Size the concurrency to the judge latency you actually observe, and measure that before you schedule anything, because the sequential sample will quietly turn a nightly job into a two-day job.

One caveat on both numbers. The arithmetic treats every scored result as an independent draw, and results are not fully independent when the same trace is scored by several judges, because the judges see the same underlying behavior. Correlated results carry less information than their raw count promises, so the effective sample is smaller than the number of rows and both figures are the optimistic end. Widening the eval set beats adding judges to the same traces.

A team that has not done this arithmetic does not yet know how large a delta has to be to mean anything, and choosing a round number instead is how a gate loses its credibility on the first false alarm.

The third requirement: a judge that has not moved, and the diff you write yourself

The SDK does ship a comparison, and the question is which regression it actually answers.

client.public.comparisons.compare() takes two evaluation IDs and returns a per-prompt breakdown with the prompt, the ground truth, and both scores. Its outcome_filter argument accepts comparison_fails, which returns exactly the cases that succeeded in the first evaluation and failed in the second. Note the path: comparisons hangs off client.public, not off the Stratix client directly, and it operates on benchmark evaluations, meaning a model measured against a benchmark.

That is the right tool for the case where a provider ships a point release under an unchanged model name and an agent gets worse with no commit on your side. What it will not do is diff two runs of your own agent, and nothing in v1.8.0 will. The gate sample produces trace evaluations, which live in a different resource with a different ID space, so the delta on your own traces is arithmetic you write:

from layerlens import Stratix

client = Stratix()

BASELINE_PASS_RATE = 0.90      # from the promoted baseline run, stored in the repo.
                               # Must come from a run of comparable size: the sqrt(2)
                               # in the band assumes both runs carry similar error.
BASELINE_JUDGE_VERSION = 7     # judge_snapshot.version at the time of that run
REGRESSION_BAND = 0.020        # 2.0 points: the 5% critical value at ~1,240 results,
                               # which is 1.645 * SE(delta), NOT SE itself.
                               # Re-derive this if you change the eval set size.

# score the curated set; get_many would re-sample by recency
JUDGE_ID = "your-judge-id"     # the judge that produced the baseline

with open("eval_set_v3.txt") as f:      # ~1,240 scored results at one judge per trace
    TRACE_IDS = [line.strip() for line in f if line.strip()]

evaluation = client.trace_evaluations.create(trace_ids=TRACE_IDS, judge_id=JUDGE_ID)
client.trace_evaluations.wait_for_completion(evaluation.id)
evaluation = client.trace_evaluations.get(evaluation.id)   # re-fetch: snapshot fills in on completion
results = client.trace_evaluations.get_results(evaluation.id)

if evaluation.judge_snapshot.version != BASELINE_JUDGE_VERSION:
    raise SystemExit("Judge moved to v%d. Re-baseline before trusting this delta."
                     % evaluation.judge_snapshot.version)

if not results:
    raise SystemExit("No results returned. Check the trace IDs and judge.")

current = sum(1 for r in results if r.passed) / len(results)
delta = current - BASELINE_PASS_RATE

if delta < -REGRESSION_BAND:
    failures = [r for r in results if not r.passed]
    print("\n".join("scored %.2f: %s" % (r.score, (r.reasoning or "")[:120]

from layerlens import Stratix

client = Stratix()

BASELINE_PASS_RATE = 0.90      # from the promoted baseline run, stored in the repo.
                               # Must come from a run of comparable size: the sqrt(2)
                               # in the band assumes both runs carry similar error.
BASELINE_JUDGE_VERSION = 7     # judge_snapshot.version at the time of that run
REGRESSION_BAND = 0.020        # 2.0 points: the 5% critical value at ~1,240 results,
                               # which is 1.645 * SE(delta), NOT SE itself.
                               # Re-derive this if you change the eval set size.

# score the curated set; get_many would re-sample by recency
JUDGE_ID = "your-judge-id"     # the judge that produced the baseline

with open("eval_set_v3.txt") as f:      # ~1,240 scored results at one judge per trace
    TRACE_IDS = [line.strip() for line in f if line.strip()]

evaluation = client.trace_evaluations.create(trace_ids=TRACE_IDS, judge_id=JUDGE_ID)
client.trace_evaluations.wait_for_completion(evaluation.id)
evaluation = client.trace_evaluations.get(evaluation.id)   # re-fetch: snapshot fills in on completion
results = client.trace_evaluations.get_results(evaluation.id)

if evaluation.judge_snapshot.version != BASELINE_JUDGE_VERSION:
    raise SystemExit("Judge moved to v%d. Re-baseline before trusting this delta."
                     % evaluation.judge_snapshot.version)

if not results:
    raise SystemExit("No results returned. Check the trace IDs and judge.")

current = sum(1 for r in results if r.passed) / len(results)
delta = current - BASELINE_PASS_RATE

if delta < -REGRESSION_BAND:
    failures = [r for r in results if not r.passed]
    print("\n".join("scored %.2f: %s" % (r.score, (r.reasoning or "")[:120]

from layerlens import Stratix

client = Stratix()

BASELINE_PASS_RATE = 0.90      # from the promoted baseline run, stored in the repo.
                               # Must come from a run of comparable size: the sqrt(2)
                               # in the band assumes both runs carry similar error.
BASELINE_JUDGE_VERSION = 7     # judge_snapshot.version at the time of that run
REGRESSION_BAND = 0.020        # 2.0 points: the 5% critical value at ~1,240 results,
                               # which is 1.645 * SE(delta), NOT SE itself.
                               # Re-derive this if you change the eval set size.

# score the curated set; get_many would re-sample by recency
JUDGE_ID = "your-judge-id"     # the judge that produced the baseline

with open("eval_set_v3.txt") as f:      # ~1,240 scored results at one judge per trace
    TRACE_IDS = [line.strip() for line in f if line.strip()]

evaluation = client.trace_evaluations.create(trace_ids=TRACE_IDS, judge_id=JUDGE_ID)
client.trace_evaluations.wait_for_completion(evaluation.id)
evaluation = client.trace_evaluations.get(evaluation.id)   # re-fetch: snapshot fills in on completion
results = client.trace_evaluations.get_results(evaluation.id)

if evaluation.judge_snapshot.version != BASELINE_JUDGE_VERSION:
    raise SystemExit("Judge moved to v%d. Re-baseline before trusting this delta."
                     % evaluation.judge_snapshot.version)

if not results:
    raise SystemExit("No results returned. Check the trace IDs and judge.")

current = sum(1 for r in results if r.passed) / len(results)
delta = current - BASELINE_PASS_RATE

if delta < -REGRESSION_BAND:
    failures = [r for r in results if not r.passed]
    print("\n".join("scored %.2f: %s" % (r.score, (r.reasoning or "")[:120]

Two numbers now govern the pipeline. The floor catches a collapse, and the band catches decay that clears the floor.

The judge_snapshot check in the middle is the part teams skip. Every trace evaluation carries one, holding the judge name, its integer version, its evaluation goal and the model backing it. An evaluation scored by judge v7 and compared against a run scored by v8 measures the judge at least as much as it measures the agent, and the snapshot is what lets a script refuse that comparison instead of reporting it.

Three more ways a comparison stops meaning anything

The prompt set changes. Adding ten hard cases to a suite of a hundred lowers the pass rate with nothing having regressed. Version the evaluation set and record which version produced the baseline.

The baseline goes stale. Pinning one run stops a rolling average from following the product downward, and it introduces the opposite problem: promote too rarely and the reference no longer resembles the system. Write down a promotion rule before the first argument about it. Be clear, too, about which evaluation job you are actually buying for, because a tool built to gate changes and a tool built to measure models keep different things constant.

The provider ships a point release. This is the one no amount of internal discipline catches, because the model name in the API call does not change. It is the case client.public.comparisons.compare() is built for, and the reason continuous model measurement and agent evaluation stay separate exercises.

Where to run it and what it costs

A pull request gate should be small and fast. Fifty traces against one judge on the pull request, scoped to changes in prompt and agent code, finishing inside the workflow's fifteen minute timeout at the fast end of that latency range, or at any latency with a handful of workers. It blocks an obvious break before a human reviews the change.

The regression comparison belongs on a nightly schedule against the curated set, where four figures of scored results are affordable given enough concurrency, and where no job timeout applies. stratix ci report --limit 20 writes a markdown summary of recent evaluations, and redirecting it with >> $GITHUB_STEP_SUMMARY puts the table in the Actions UI. That command reports rather than gates, with one exception worth knowing: it exits 1 when it finds no evaluations at all.

Cost scales with trace count multiplied by judge count and with the length of each trace. client.trace_evaluations.estimate_cost(trace_ids=[...], judge_id=...) returns the estimate before the run, which is the check to put in front of a nightly job that just grew from 50 traces to over a thousand scored results.

What this does not solve

A delta gate compares runs. It has no opinion about whether the evaluation set was worth running.

The most common way it fails in practice is a baseline promoted from a run that was never good. That locks in a mediocre reference and reports every subsequent release as healthy, and no amount of statistical care detects it, because the arithmetic is working correctly on a bad number.

It also inherits the blind spots of the judge doing the scoring. A judge that consistently misreads a class of failure misses it in both runs, and the delta comes back clean while a real regression passes through. The judge_snapshot check catches a judge that changed. It does nothing about a judge that was wrong from the start.

And it only sees regressions inside the evaluation set. A production failure mode nobody wrote a case for stays invisible to a delta gate for exactly the reason it stays invisible to a floor gate.

Frequently Asked Questions

Can I just run the regression check on every pull request?

Only if you accept one of two costs. A pull-request-sized sample of fifty results carries a six-point standard error on the delta, so a three-point band fires on roughly one clean run in three. Running a set large enough to resolve three points means 1,240 scored results, which needs concurrent execution to finish in minutes rather than hours. Most teams keep a floor check on the pull request and move the regression check to a schedule.

What if my baseline pass rate is not 0.90?

Re-derive the band. The arithmetic here uses p = 0.90 because the standard error of a proportion depends on p, and it peaks at 0.50 and shrinks toward the extremes. At a 0.95 baseline the same three-point drop needs fewer results; at 0.70 it needs more. The formula in the sizing section takes any p, and the one number you should not reuse from this post is the band itself.

How do I know whether the judge changed or the agent did?

Compare judge_snapshot.version between the two runs before you compare pass rates. If the versions differ, the delta measures both and the comparison should be refused rather than reported. This is why the check belongs in the script and not in a runbook: a human reading a dashboard has no way to see that the scorer moved.

Does a scheduled regression run replace a staging environment?

No. It tells you a change of a certain size happened somewhere in a commit range, which narrows attribution but does not reproduce the failure or tell you which user-visible behavior broke. The failing traces and their reasoning strings are the starting point for that, and they are the reason the script prints them rather than only exiting non-zero.

The cheapest thing to fix this week

Two of the three requirements from the top of this post cost nothing to address. Stop letting get_many choose the test set, and check judge_snapshot.version before trusting any comparison. Both are a few lines, and both address confounders that make a delta meaningless rather than merely noisy.

The arithmetic on sample size is the one that reshapes the pipeline, because a run of that size will not fit in a fifteen minute pull request job and the regression check has to move to a schedule. Better to find that in a spreadsheet than in six months of green builds.

Read the CI/CD samples in the Stratix Python SDK.