Your LLM Judge Agrees with Itself. That Does Not Mean It Is Right.

A production LLM judge reports 85% agreement across repeated runs. The team ships the evaluation pipeline. Three weeks later, a model that ranked fifth on one benchmark lands at twentieth on another. The judge never flagged a problem because the judge was never tested for the right kind of reliability. High self-agreement is not validity. It is a consistency score that can mask systematic bias, position sensitivity, and benchmark-dependent ranking collapse. A June 2026 study spanning 21 judges, 541,000 judgments, and 3 benchmarks found that 85% raw agreement corresponded to a Cohen's kappa of roughly 0.48. That is barely above moderate agreement once you correct for chance.

[INSERT IMAGE: post4-hero.png - Visual showing judge agreement illusion vs reality: high raw agreement on the left, low kappa on the right]

TL;DR

• A judge reporting 85% agreement on MT-Bench actually achieves Cohen's kappa of approximately 0.48 once chance agreement is factored in. Raw percentages overstate real reliability.

• 52% of models (11 of 21) shifted 4 or more rank positions when evaluated across different benchmarks. Llama 3.3 70B dropped from rank 5 to rank 20.

• Position bias varies by a factor of 96x across judges: 0.002 for Gemini 2.5 Pro versus 0.192 for Qwen 3 8B.

• Two production judges achieved test-retest reliability above 0.95 while simultaneously showing position bias above 0.10. Consistency and correctness are independent properties.

• Single-trial pairwise verdicts carry plus or minus 5 points of uncertainty. Reliable evaluation requires 20 to 50 trials per item.

• Frontier models fail more than 50% of advanced bias tests in JudgeBiasBench. No model family is exempt.

Raw Agreement Is the Wrong Metric

Most teams measure judge reliability with raw agreement: the percentage of times a judge produces the same verdict on the same input pair. The number looks reassuring. Eighty, eighty-five, ninety percent.

The problem is base rate. If two outputs differ substantially in quality, any judge will agree with itself on the easy cases. Cohen's kappa corrects for this by subtracting expected chance agreement. The June 2026 study (21 judges, 541,000 judgments, 3 benchmarks) found universal kappa deflation. An 85% raw agreement score on MT-Bench translated to kappa of approximately 0.48. By conventional interpretation, that is moderate agreement. Not strong. Not substantial. Moderate.

Teams that report only raw agreement are not lying. They are measuring the wrong thing. Kappa deflation means the judge is coasting on easy comparisons and performing near-randomly on the close calls that actually matter for model selection.

[INSERT IMAGE: post4-kappa-deflation.png - Chart showing kappa deflation across benchmarks]

Rankings Collapse Across Benchmarks

If a judge were genuinely reliable, model rankings would hold across evaluation contexts. They do not. The same study found that 52% of models (11 of 21) shifted 4 or more rank positions when evaluated on different benchmarks. Llama 3.3 70B moved from rank 5 to rank 20. That is not noise in the margins. That is a total reordering of the leaderboard.

This result breaks a common assumption in AI governance frameworks: that a single benchmark evaluation provides a stable quality signal. A model that passes an evaluation gate on Benchmark A may fail on Benchmark B with the same judge. The evaluation did not change. The context did.

Position Bias: A 96x Spread

Pairwise evaluation presents two outputs side by side. The judge picks one. Position bias measures whether the judge systematically favors the output shown first (or second), regardless of quality.

Across the 21 judges studied, position bias ranged from 0.002 (Gemini 2.5 Pro) to 0.192 (Qwen 3 8B). That is a 96-fold difference. At the high end, nearly one in five verdicts is determined by presentation order rather than output quality.

[INSERT IMAGE: post4-position-bias.png - Position bias range across 21 models]

A single-trial pairwise evaluation using a high-bias judge is not an evaluation. It is a coin flip with extra steps. Teams running pairwise comparisons without position-bias controls are generating rankings shaped more by presentation order than by model capability.

The Consistency-Bias Paradox

Here is the finding that should concern every evaluation team: two production judges in the study achieved test-retest reliability of 0.95 or higher while simultaneously exhibiting position bias above 0.10. They were perfectly consistent. They were also systematically wrong.

This is the consistency-bias paradox. A judge can reproduce the same biased verdict every time, achieving excellent reliability scores while embedding a directional error into every evaluation run. Consistency metrics cannot detect this. Only bias-specific tests can.

The implication for production evaluation pipelines is direct. Test-retest reliability is necessary but not sufficient. A judge that passes a consistency check still needs to pass position bias tests, brand bias tests, and verbosity bias tests before it earns trust in a ranking decision.

Identical Prompts, Random Verdicts

"The Coin Flip Judge?" (arXiv 2606.13685) tested what happens when you submit identical prompts to the same judge multiple times. The answer: verdicts varied at rates approximating random chance. The same input, the same judge, different outputs. Not occasionally. Systematically.

This finding challenges the assumption that LLM judges are deterministic evaluation instruments. Even with temperature set to zero, quantization effects, batching differences, and infrastructure variability introduce enough variance to flip close-call verdicts. Teams treating single-trial verdicts as ground truth are building on sand.

Brand Bias Distorts Pairwise Comparisons

A separate study (arXiv 2606.17443) documented brand bias in LLM judges. When model names are visible in pairwise comparisons, judges systematically favor outputs labeled with established model names. The effect is not subtle. It is a measurable, reproducible directional bias that inflates scores for recognized brands and deflates scores for newer or less-known models.

This matters for any team evaluating open-source alternatives against proprietary models. If the judge knows which output came from GPT-4o and which came from a fine-tuned Llama variant, the comparison is contaminated. Blind evaluation (stripping model identifiers before judging) is not optional. It is a prerequisite for valid results. This is one reason why robust agent evaluation frameworks must account for metadata leakage.

Inference Compute Reshapes Rankings

A study published as arXiv 2606.17930 found that performance rankings shift when test-time compute budgets change. A model that outperforms a competitor under standard inference conditions may underperform when the competitor is given additional compute for chain-of-thought reasoning or self-verification.

This means evaluation results are not just benchmark-dependent. They are compute-budget-dependent. Two teams evaluating the same pair of models can reach opposite conclusions by allocating different inference budgets. An evaluation report that does not specify inference compute parameters is incomplete.

No Judge Is Uniformly Reliable

RAND Corporation's Judge Reliability Harness tested judges across multiple reliability dimensions. The conclusion: "no judge was uniformly reliable." Every judge in the study had at least one failure mode. Some were position-biased. Some were verbosity-biased. Some were brand-biased. None passed every test.

Frontier models fare no better. JudgeBiasBench found that frontier models fail more than 50% of advanced bias tests. The assumption that a more capable model makes a more reliable judge does not hold. Capability and judge reliability are different properties measured by different instruments.

The 20 to 50 Trial Threshold

Given the variance documented across these studies, what does reliable pairwise evaluation actually require? The practical threshold is 20 to 50 trials per item before trusting a pairwise verdict. Single-trial claims carry plus or minus 5 points of uncertainty.

That is not a typo. A single pairwise comparison that reports "Model A scores 78, Model B scores 73" cannot distinguish between the two models. The confidence interval swallows the difference. Teams making deployment decisions on single-trial pairwise results are making decisions on noise.

Multi-trial evaluation is more expensive. It is also the minimum viable approach. Anything less is theater.

What Reliable Judge Evaluation Requires

The research converges on a set of minimum requirements for trustworthy LLM-as-judge evaluation:

• Kappa, not raw agreement. Report Cohen's kappa or Krippendorff's alpha. Raw percentages hide chance agreement.

• Position bias testing. Run every pairwise comparison in both orderings. Measure and report the delta.

• Blind evaluation. Strip model identifiers before judging. Brand bias is real and measurable.

• Multi-trial aggregation. Run 20 to 50 trials per item. Single trials are not evaluations.

• Cross-benchmark validation. Test ranking stability across at least two benchmarks. If rankings collapse, the evaluation is benchmark-dependent, not model-dependent.

• Version pinning. Record judge model, version, prompt template, and inference parameters. Judge drift is undetectable without version control.

• Compute budget specification. Document inference compute allocation. Rankings are compute-dependent.

Organizations navigating evaluation decision paralysis often skip these steps because they add complexity. The alternative is shipping evaluation results that do not replicate.

How Stratix Addresses Judge Reliability

Stratix was built around the finding that no single judge is uniformly reliable. Every evaluation run in Stratix pins judge version, evaluation goal, and scoring criteria to a seven-field schema: input identifier, model version, agent configuration, judge name and version, evaluation goal, score, and timestamp. When a judge updates or drifts, the change is visible in the evaluation history.

Stratix's Deliberation Panels address position bias directly. Instead of running a single judge in a single ordering, Deliberation Panels run multiple judges across multiple orderings and aggregate verdicts. The result eliminates position bias by design rather than by hoping a single judge happens to have low bias (a bet that fails for 19 of the 21 judges in the study).

The seven-field schema also makes cross-benchmark comparison possible. Because every evaluation records the benchmark, judge, and configuration, teams can test whether their rankings hold across evaluation contexts or collapse like the 52% of models in the study.

Frequently Asked Questions

What is Cohen's kappa and why does it matter for LLM judges?

Cohen's kappa measures agreement between two raters (or a rater with itself across trials) after correcting for chance. A raw agreement of 85% sounds strong, but if two random raters would agree 70% of the time by chance, the real signal is much smaller. Kappa captures that residual. For LLM judges, kappa reveals that high raw agreement often masks near-random performance on close comparisons.

How many trials do you need for a reliable pairwise comparison?

Research points to 20 to 50 trials per item as the minimum for trustworthy pairwise verdicts. Single-trial comparisons carry plus or minus 5 points of uncertainty, which is large enough to swallow most score differences between competitive models.

Can position bias be fixed by randomizing presentation order?

Randomization reduces the systematic effect but does not eliminate it. A judge with 0.192 position bias will still produce biased individual verdicts. The reliable approach is to run every comparison in both orderings and aggregate, which is what Stratix's Deliberation Panels do by default.

Does using a more powerful model as a judge solve reliability problems?

No. JudgeBiasBench found that frontier models fail more than 50% of advanced bias tests. Gemini 2.5 Pro shows extremely low position bias (0.002) but other frontier models do not share that property. Capability and judge reliability are independent dimensions.

What is brand bias in LLM evaluation?

Brand bias occurs when a judge systematically favors outputs labeled with recognized model names (such as GPT-4o or Claude) over outputs from lesser-known models, regardless of actual quality. The effect was documented in arXiv 2606.17443. Blind evaluation (removing model identifiers before judging) is the standard mitigation.

How does inference compute affect evaluation results?

Performance rankings shift when test-time compute budgets change (arXiv 2606.17930). A model that wins under standard inference may lose when its competitor gets additional compute for reasoning or self-verification. Evaluation reports should specify compute allocation to be reproducible.

What is the consistency-bias paradox?

Two judges in the June 2026 study achieved test-retest reliability above 0.95 (highly consistent) while exhibiting position bias above 0.10 (systematically wrong). A judge can reproduce the same biased verdict perfectly every time. Consistency metrics alone cannot detect directional bias.

How does Stratix detect judge drift over time?

Every Stratix evaluation run records judge name, version, evaluation goal, score, and timestamp in a structured schema. When a judge model updates or its behavior shifts, the change appears as a detectable pattern in the evaluation history. Teams can compare scores before and after a judge update to quantify drift.

Stop Trusting Self-Agreement

The evidence is consistent across six independent studies: LLM judges are not reliable evaluation instruments by default. They exhibit position bias, brand bias, benchmark sensitivity, and verdict instability. High self-agreement masks these failures rather than ruling them out.

The path forward is not abandoning LLM judges. It is treating them as measurement instruments that require calibration, bias testing, and multi-trial aggregation. Teams that do this work get evaluation results they can act on. Teams that skip it get numbers that feel precise and mean very little.

Stratix provides the infrastructure for reliable LLM evaluation: version-pinned judges, Deliberation Panels for position-bias elimination, and structured schemas for drift detection. Start evaluating with Stratix and stop treating judge self-agreement as proof of reliability.