LLM as a Judge: Two Runs, Two Scores, No Answer

Author:

The LayerLens Team

Last updated:

Aug 18, 2025

Published:

Aug 18, 2026

Your LLM judge scored 0.91 last Tuesday and 0.84 this Tuesday. The team assumes the agent got worse. Nobody checks whether the judge itself changed between runs.

Two things can move underneath an LLM judge without anyone noticing. The judge changes when someone edits the rubric, adjusts the output format, or the model provider ships a new snapshot. The population changes when the script pulls "the most recent 50 traces" instead of a fixed list. Either shift moves the score, and both look identical to a real regression.

That gap has a name: an unpinned delta. The score changed, and nobody can say whether the agent, the judge, or the test set caused it. Most eval setups that track scores over time are reporting unpinned deltas. The number on the dashboard looks the same whether it carries information or not.

TL;DR

An LLM judge is a measuring instrument with a version. Two things drift underneath it between runs: the judge itself (rubric text, output contract, underlying model snapshot) and the population (which traces got scored).
Pin the judge by storing a judge snapshot with every score. Stratix carries judge_snapshot on trace evaluations with four fields: name, version, evaluation_goal, model_name. Those four pin the instrument.
Pin the population by naming trace IDs explicitly. Any call that returns "recent" data hands back a different set on every invocation.
Any CI script that pulls "recent traces" re-samples its test set on every run. That works for threshold gating (does this batch pass?) and breaks for regression comparison (did the score drop?). The fix is a checked-in list of trace IDs.
Stratix Python v1.8.0 does not diff two trace-evaluation runs. client.public.comparisons.compare() diffs benchmark evaluations, which is a different object. The trace-eval delta is arithmetic you write.
Skip the LLM judge entirely for anything a rule can decide. Code Graders in Stratix cover exact match, regex, JSON-schema validity, semantic similarity, Flesch-Kincaid and fairness math, with no model call and no version to drift.

What an LLM judge is

An LLM judge is a language model handed a rubric and asked to score another model's output against it. The rubric names a dimension (faithfulness to retrieved context, whether an agent took a destructive action, whether a response held the requested tone) and fixes an output contract: binary, a score on a scale, or a label from a set. The judge reads the artifact, applies the rubric, returns a verdict and its reasoning.

Teams adopt the pattern because production cares most about dimensions no assertion reaches. No regular expression catches a support response that stayed factually accurate while steering the customer toward the worse product. A model reading for meaning catches it.

Three places teams tend to land with this. The first group runs a judge by hand when something looks wrong, gets a useful answer, and never schedules it. The second group schedules it, watches the average, and cannot say why the average moved. The third group pins the instrument and the population, so a movement in the number carries information. Most of the teams in group two believe they are in group three, and the belief survives because their dashboard renders either way.

The judge moves

A judge has three moving parts, and each moves for its own reasons.

An engineer sharpens the rubric text because it scored too generously last quarter, which is good work that every calibrated judge needs. It also silently splits the score history in two.

Output contracts shift shape. A binary judge becomes a scored judge, or a scale runs 1 to 5 in March and 0 to 1 in June. Every average downstream keeps computing and starts meaning something different.

Then the provider upgrades the model on its own schedule. A judge pointed at a floating alias inherits every snapshot the provider ships. Nobody on the team touched the rubric, nobody touched the sample, and the scores moved anyway.

Storing a record beats writing a rule about who may edit what. Stratix carries judge_snapshot on trace evaluations, holding the judge name and version, the evaluation_goal that was live at scoring time, and the model_name that returned the verdict. Read those four across a window and judge drift becomes a diff instead of a theory.

pip install --extra-index-url https://sdk.layerlens.ai/package "layerlens[cli]"

pip install --extra-index-url https://sdk.layerlens.ai/package "layerlens[cli]"

pip install --extra-index-url https://sdk.layerlens.ai/package "layerlens[cli]"

Note the --extra-index-url. Dropping "extra" breaks dependency resolution, which is the single most common way this install fails.

from layerlens import Stratix

client = Stratix()

# A pinned regression set. Boring, stable, checked into the repo.
TRACE_IDS = [
    "trc_01hq9y...",
    "trc_01hq9z...",
    # the cases you actually care about regressing
]

JUDGE_ID = "jdg_faithfulness_v4"


def score_pinned_set(client, trace_ids, judge_id):
    """Score a fixed set and return the judge snapshot alongside the scores."""
    results = []
    for trace_id in trace_ids:
        # [VERIFY WITH ENGINEERING: confirm create() returns a populated
        #  score, or whether a poll loop is needed before reading ev.score]
        ev = client.trace_evaluations.create(
            trace_id=trace_id,
            judge_id=judge_id,
        )
        snap = ev.judge_snapshot
        results.append(
            {
                "trace_id": trace_id,
                "score": ev.score,
                "judge_name": snap.name,
                "judge_version": snap.version,
                "judge_model": snap.model_name,
            }
        )
    return results

from layerlens import Stratix

client = Stratix()

# A pinned regression set. Boring, stable, checked into the repo.
TRACE_IDS = [
    "trc_01hq9y...",
    "trc_01hq9z...",
    # the cases you actually care about regressing
]

JUDGE_ID = "jdg_faithfulness_v4"


def score_pinned_set(client, trace_ids, judge_id):
    """Score a fixed set and return the judge snapshot alongside the scores."""
    results = []
    for trace_id in trace_ids:
        # [VERIFY WITH ENGINEERING: confirm create() returns a populated
        #  score, or whether a poll loop is needed before reading ev.score]
        ev = client.trace_evaluations.create(
            trace_id=trace_id,
            judge_id=judge_id,
        )
        snap = ev.judge_snapshot
        results.append(
            {
                "trace_id": trace_id,
                "score": ev.score,
                "judge_name": snap.name,
                "judge_version": snap.version,
                "judge_model": snap.model_name,
            }
        )
    return results

from layerlens import Stratix

client = Stratix()

# A pinned regression set. Boring, stable, checked into the repo.
TRACE_IDS = [
    "trc_01hq9y...",
    "trc_01hq9z...",
    # the cases you actually care about regressing
]

JUDGE_ID = "jdg_faithfulness_v4"


def score_pinned_set(client, trace_ids, judge_id):
    """Score a fixed set and return the judge snapshot alongside the scores."""
    results = []
    for trace_id in trace_ids:
        # [VERIFY WITH ENGINEERING: confirm create() returns a populated
        #  score, or whether a poll loop is needed before reading ev.score]
        ev = client.trace_evaluations.create(
            trace_id=trace_id,
            judge_id=judge_id,
        )
        snap = ev.judge_snapshot
        results.append(
            {
                "trace_id": trace_id,
                "score": ev.score,
                "judge_name": snap.name,
                "judge_version": snap.version,
                "judge_model": snap.model_name,
            }
        )
    return results

Both runs now carry their own instrument record. Comparing them takes six lines, and writing those six lines yourself is an advantage, because you decide what counts as the same case.

def compare_runs(baseline, candidate):
    """Refuse to report a delta when the instrument changed."""
    base_versions = {(r["judge_version"], r["judge_model"]) for r in baseline}
    cand_versions = {(r["judge_version"], r["judge_model"]) for r in candidate}
    if base_versions != cand_versions:
        raise ValueError(
            f"Judge changed between runs: {base_versions} vs {cand_versions}. "
            "This delta is a judge delta, not an agent delta."
        )

    base_by_trace = {r["trace_id"]: r["score"] for r in baseline}
    return {
        r["trace_id"]: r["score"] - base_by_trace[r["trace_id"]]
        for r in candidate
        if r["trace_id"] in base_by_trace
    }

def compare_runs(baseline, candidate):
    """Refuse to report a delta when the instrument changed."""
    base_versions = {(r["judge_version"], r["judge_model"]) for r in baseline}
    cand_versions = {(r["judge_version"], r["judge_model"]) for r in candidate}
    if base_versions != cand_versions:
        raise ValueError(
            f"Judge changed between runs: {base_versions} vs {cand_versions}. "
            "This delta is a judge delta, not an agent delta."
        )

    base_by_trace = {r["trace_id"]: r["score"] for r in baseline}
    return {
        r["trace_id"]: r["score"] - base_by_trace[r["trace_id"]]
        for r in candidate
        if r["trace_id"] in base_by_trace
    }

def compare_runs(baseline, candidate):
    """Refuse to report a delta when the instrument changed."""
    base_versions = {(r["judge_version"], r["judge_model"]) for r in baseline}
    cand_versions = {(r["judge_version"], r["judge_model"]) for r in candidate}
    if base_versions != cand_versions:
        raise ValueError(
            f"Judge changed between runs: {base_versions} vs {cand_versions}. "
            "This delta is a judge delta, not an agent delta."
        )

    base_by_trace = {r["trace_id"]: r["score"] for r in baseline}
    return {
        r["trace_id"]: r["score"] - base_by_trace[r["trace_id"]]
        for r in candidate
        if r["trace_id"] in base_by_trace
    }

That raise is the whole point. A comparison function that refuses to run on a changed instrument catches an unpinned delta before it reaches a dashboard.

[INSERT IMAGE: pinned-vs-unpinned-delta.png - Comparison showing how an unpinned delta cannot attribute score movement while a pinned delta isolates agent behavior] Image URL: https://litter.catbox.moe/ybf4yd.png

[INSERT IMAGE: stratix-evaluation-detail.png - Stratix trace evaluation detail showing a per-row judge score with the recorded judge snapshot name, version, evaluation goal and judging model beside it] Image URL: [CATBOX URL TO BE ADDED]

The population moves

The second drift hides in the code that fetches the test set, which is why it survives code review.

Most evaluation scripts pull "the most recent N traces" because that is what monitoring needs. A dashboard wants fresh data. A regression comparison needs the same data both times. Score Monday's 50 traces, score next Monday's 50 traces, subtract, and the number describes a week of shifting production traffic rather than a week of engineering.

The Stratix Python SDK's CI gating sample (samples/cicd/quality_gate.py) does exactly this: client.traces.get_many(page_size=50) returns whatever is most recent. As a pass/fail gate against a threshold, that works. As a regression baseline, it scores a different population every time. Two more properties matter if you copy this pattern: the sample caps at 200 evaluations and runs them sequentially, while the companion GitHub Actions workflow sets timeout-minutes: 15. At 5 to 60 seconds per judge call, a full 200-evaluation run can exceed that window. Raise the timeout or cut the set, and know which one you chose.

One clarification for anyone searching the CLI for a shortcut: stratix ci run does not exist. The ci group in v1.8.0 has one subcommand, report, which writes a markdown summary and does not gate. Gating ships as the sample script, not as a built-in command.

Pin the judge-side four, log the full seven

Two different minimums operate here, and conflating them causes arguments.

The judge-side four pin the instrument: judge name, judge version, evaluation goal, judging model. Those four decide whether a comparison between two runs is valid at all.

The full seven make a single score reconstructable months later: the judge-side four, plus the exact input identifier, the threshold in force, and the verdict. Six of the seven cost bytes. The seventh is the verdict you already have.

A team pinning only the judge-side four can compare runs. A team logging all seven can also answer why one specific output passed on one specific day, which is a different question and usually arrives from outside engineering.

When the LLM judge is the wrong instrument

An LLM judge costs a model call per row and carries a version that can move. Both argue for a cheaper check whenever a cheaper check settles the question.

Stratix Code Graders settle a larger share than most teams expect: exact match, regex match, JSON-schema validity, semantic similarity, Flesch-Kincaid readability, fairness math. They return the same answer forever, cost nothing per run, and hold no model inside them to drift. Structured-output validity is the clean case. When the contract says "this parses as JSON matching this schema," a schema check answers it completely and an LLM judge only adds latency and variance.

Keep the judge for dimensions that need reading comprehension: faithfulness against retrieved context, whether a tool-call sequence accomplished what the user asked rather than something adjacent, tone, and domain standards that live in a policy document instead of a pattern.

The counterargument worth taking seriously

Pinning costs something real, and teams that sample are buying something real.

A frozen trace set goes stale. Traffic shifts, new failure modes appear, and a set frozen in March stops representing what an agent meets in August. A team that pins everything and never refreshes ends up defending a benchmark that describes a product it no longer ships.

Two sets resolve it. Keep a pinned regression set that changes only on purpose, with a version bump when it does, and run a separate rolling sample for discovery. The pinned set answers whether anything broke. The rolling sample surfaces what production is doing that nobody has tested for yet. Teams reporting unpinned deltas are almost always asking one set to do both jobs.

Frequently asked questions

Does LLM as a judge actually work? It works well enough on subjective dimensions to be the default approach, and its reliability tracks rubric calibration more than model choice. Stratix tunes custom judges with GEPA against 30 or more ground-truth labels, which is the mechanism that turns a plausible rubric into a calibrated one. A judge nobody calibrated returns a guess in a confident format.

Which model should do the judging? Pick one, pin the snapshot, and calibrate against labels from your own domain rather than reaching for whichever model currently leads a general ranking. Calibration is the variable a team controls, and the pinned snapshot is what makes any comparison meaningful afterward.

How can a team tell whether a judge drifted? Compare judge snapshots across the window. Different version or different model_name between two runs means the instrument changed and the score comparison carries no information about the agent. A team that never stored the snapshot cannot answer this retroactively, which is the argument for storing it from run one.

Can one judge serve both benchmark evaluations and production traces? The rubric carries over. The score series should not merge, because benchmark evaluations and trace evaluations are separate objects in Stratix and they diff through different paths. Keep the rubric shared and the two series apart.

What is the minimum to log per evaluation? Judge name, judge version, evaluation goal, judging model, the input identifier, the threshold in force, and the verdict. Four of those pin the instrument for comparison. All seven make one past decision reconstructable.

Do deterministic graders replace LLM judges? They replace the checks a rule can decide, and most teams find that share larger than expected once they inventory what they are actually asking the judge. They cannot reach dimensions that require reading for meaning, which is where an LLM judge earns its cost per row.

Sourcing note: SDK behavior here reflects LayerLens/stratix-python v1.8.0 as verified against the public repository, including the quality_gate.py sampling behavior, the MAX_EVALUATIONS cap, the workflow timeout, and the absence of any trace-evaluation diff.

Pinning is a one-afternoon change with a long tail. The next time a score moves, the useful position to be in is knowing whether the agent moved or the ruler did.

See how Stratix records judge snapshots alongside every trace evaluation at layerlens.ai.

The Setup Shapes the Score: How Grading Logic, Model Behavior, and Token Budgets Move Evaluation Results ›