New in Stratix: Evaluation Run Comparison and Judge Automation Rules
Author:
The LayerLens Team
Last updated:
Published:
By the LayerLens Team. LayerLens builds Stratix, the evaluation platform for AI agents and models.
TL;DR
Evaluation Run Comparison is live in Stratix: pick two completed runs of the same benchmark and see every prompt with each run's answer, the ground truth, and pass or fail for both.
In the launch example, one model on one 30-prompt benchmark went from 80.0% in July to 73.3% in September. Two regressed prompts account for the entire 6.7 point drop; 28 prompts were unchanged and none improved.
Judge Automation Rules are live: attach a rule to any judge and every matching trace is judged within about a minute of arriving, with a sample rate and a daily cap you set.
Rules filter on agent, framework, status, and tag. Leave the filter blank to judge every new trace in the project.
Also new: custom scorers can be picked in the evaluation run dialog without being attached to the benchmark first.
Also in this release: Policy Coverage. Upload a compliance document and Stratix proposes judges and scorers from its clauses, each citing the clause it came from, for a person to review and approve.
Two features are live in Stratix today. Evaluation Run Comparison lays two runs of the same benchmark side by side, prompt by prompt, so you can see exactly which prompts got better and which got worse after a model or prompt change. Judge Automation Rules let a judge run on its own the moment a matching trace arrives, with no manual run from the UI. A third feature, Policy Coverage, generates judges and scorers from an uploaded compliance document and ships in the same release.
Both features answer the same complaint from teams doing AI agent regression testing: the aggregate score moves, and nobody can say which prompts moved it or whether the traces that mattered were ever judged at all.
Evaluation Run Comparison
Until now, Stratix could compare two models on one benchmark, but it could not compare two runs of the same model. That is the case regression testing needs: you evaluate, fix what failed, rerun, and want to know what changed. Teams were reading two aggregate scores and guessing.
Evaluation Run Comparison closes that gap. Pick any two completed evaluations on the same benchmark, same model or not, and Stratix shows every prompt with each run's answer, the ground truth, and pass or fail for both. An outcome filter narrows the list to the prompts that moved.
How to use it
Open Model Evaluations and select two completed runs. They have to be on the same benchmark; Stratix will tell you if they are not.
Click Compare. The baseline run is the "before" and the candidate run is the "after". Swap them with one click.
Read the top row: accuracy change in points, then the counts of prompts that improved, regressed, and stayed unchanged.
Click Show regressions. That list is your regression report, with each run's answer and the ground truth on every row.
Click Show improved to confirm which fixes held.
What the example shows
Same model, same benchmark, six weeks apart. The baseline run scored 80.0% and the candidate run scored 73.3%, a 6.7 point drop. Prompt by prompt, 28 prompts were unchanged, none improved, and 2 regressed. Those two prompts account for the entire drop, and the regression view puts both of them on screen with the earlier answer, the later answer, and the ground truth.
The regression list: prompts the baseline run passed and the candidate run failed.
The same view works across models. On the public compare page, two runs of the 198-prompt General Purpose Question Answering benchmark scored 86.9% and 88.9%; inside that two-point gain, 13 prompts flipped from wrong to right and 9 flipped from right to wrong. A rising score is not evidence that nothing broke.
Judge Automation Rules
A Stratix judge is a plain-English evaluation criterion that a judge agent applies to an agent trace. The system library includes a Compliance judge that checks a trace against five policy areas, including data minimization under GDPR Article 5, and returns pass or fail with a finding per area. In the trace evaluation records on the same instance, one judge run on one trace took 25 to 30 seconds.
Until now, running a judge meant opening a trace and clicking Run. Traces nobody opened went unjudged, which is a problem when the agent emits traces all day and the ones that matter to an auditor are not the ones someone happened to look at.
Judge Automation Rules take the click out of the loop. Attach a rule to a judge and every trace that matches it is judged as it arrives through the SDK or one of the 30 adapters in public preview. New traces are picked up within about a minute of arriving, and the verdict is stored with the trace. The manual Run button stays for spot checks.
How to use it
Open any judge and go to its Automation tab, then click New rule.
Name the rule and set the filter: agent, framework, status, tag. Leave everything blank to judge every new trace in the project.
Set a sample rate (every matching trace, or a percentage) and a daily cap, the most runs the rule may launch per day. The dialog shows how many traces matched the filter in the last 24 hours and how many evaluations the rule would run at that rate.
Switch on Activate this rule, or leave it off to save the rule without spending anything yet, then click Create rule.
The Automation tab lists each rule with its sample rate, runs used today, and an active toggle. Edit or pause it from there.
Coverage is the practical change. A manual run only ever covered the traces someone opened. A rule covers every trace that matches its filter, up to the cap you set, and the record shows what was judged.
Also in this release: when you launch an evaluation, you can now pick custom scorers in the run dialog; they no longer have to be attached to the benchmark first.
Policy Coverage: Judges Generated From Your Compliance Documents
Writing a judge from a SOC 2 report or a GDPR mapping by hand means someone reads the document and translates clauses into evaluation criteria one at a time.
Policy Coverage lives in the dashboard sidebar under Catalog and takes the document as the input. Upload a policy, a regulation, or an internal SOP, and Stratix proposes judges and scorers drawn from its clauses. Each proposal cites the clause it came from, so a reviewer can check the criterion against the source text. Nothing is created until a person reviews and approves it. Approved judges are ordinary Stratix judges, with versions and the same automation rules as any other judge, so a document-derived judge can run on every matching trace as it arrives.
Availability
Evaluation Run Comparison and Judge Automation Rules are live now on production for every Stratix plan, and Policy Coverage ships in the same release. No changes are needed to existing judges, benchmarks, or adapters.Judges, automation rules, and the system judge library are documented at docs.layerlens.ai. GDPR Article 5, the data minimization principle the Compliance judge checks, is published in the Official Journal of the European Union.
Frequently Asked Questions
Do the two runs have to use the same model?
No. Both runs have to be on the same benchmark, but they can use the same model, different versions of one model, or two different models. The launch example is one model six weeks apart; the public compare page example is two different models on the 198-prompt General Purpose Question Answering benchmark.
What counts as a regression in Evaluation Run Comparison?
A prompt the baseline run passed and the candidate run failed. Improved is the reverse. Unchanged means both runs passed or both runs failed. The top row of the compare view gives the three counts and the accuracy change in points.
Which traces does a judge automation rule cover?
Every trace that matches the rule's filter (agent, framework, status, tag) as it arrives through the SDK or an adapter, at the sample rate you set, up to the daily cap. A rule with an empty filter covers every new trace in the project.
How quickly is a new trace judged?
New traces are picked up within about a minute of arriving. A single judge run on one trace took 25 to 30 seconds in the trace evaluation records used for this post, so a verdict is typically stored within a couple of minutes of ingest.
How do I keep automated judging spend predictable?
Each rule has a sample rate and a daily cap. The rule dialog shows how many traces matched the filter in the last 24 hours and how many evaluations the rule would run at the chosen rate. You can also save a rule without activating it and switch it on later.
What is Policy Coverage?
Policy Coverage, in the dashboard sidebar under Catalog, turns a compliance document into evaluation criteria. Upload a policy, a regulation, or an internal SOP, and Stratix proposes judges and scorers from its clauses, each citing the clause it came from. A person reviews and approves each proposal before anything is created, and approved judges work like any other judge, automation rules included.
Do existing judges, benchmarks, or adapters need any changes?
No changes are needed. Any existing judge gets an Automation tab, and any two completed runs on the same benchmark can be compared.
Run a comparison on two of your own evaluations in Stratix. Open any judge, open its Automation tab, and create your first rule.
Data note: the 30-prompt same-model comparison is from a Stratix workspace on 2026-09-08 (runs EVL-516c95d719 and EVL-db3add5298). The 198-prompt comparison is evaluations 6a8f10db166b2bdcb81d9117 and 6a88890fadc687171b8edaa8 on the General Purpose Question Answering benchmark, pulled through the Stratix SDK on 2026-09-04; those two runs used different models. Judge timing is from trace evaluation records on the same instance.