How Jev Changes AI Evaluation

TypeSafe AI released Jev on September 15. Jev is a judge in model form. You send it state and questions, and it returns a probability for each answer, with no text. Within eight days, developers had published 109 independent studies testing it. That count says more about demand for a cheaper judge than any launch post could.

TL;DR

• TypeSafe reports 67.8% agreement for Jev at $0.0004 and 0.4 seconds per case, against 67.9% for a GPT "Terra" setup at $0.0304 and 10.1 seconds. The reference answers come from averaging two other models.

• At those rates a million judgments cost about $400 with Jev and about $30,400 with the Terra setup.

• In an 8,801-example test by Anthus, Jev reported 91.4% confidence on Choice questions and was right 76.1% of the time. Isotonic recalibration cut calibration error from 0.117 to 0.008.

• Barg Labs planted one false sentence in agent completion reports. Jev passed 90% of premature "done" claims and 98% of fabricated references.

• TypeSafe has published no calibration curve for Jev.

What TypeSafe claims

The headline numbers come from TypeSafe. On its workflow evals, Jev agreed with reference answers 67.8% of the time at $0.0004 and 0.4 seconds per case, against 67.9% for a GPT "Terra" setup at $0.0304 and 10.1 seconds. Two details in TypeSafe's own materials matter more than the headline. The reference answers are an average of GPT-6 Astra and Claude Fable 5.1, so 67.8% measures agreement with other models. And for a model whose entire output is probabilities, TypeSafe has published no calibration curve.

What a cheap judge changes for evals

If those cost and speed numbers hold on real workloads, two things shift.

The first is coverage. A million judgments would cost about $400 at Jev's rate and about $30,400 with the Terra setup. Teams that grade a 1% sample of traces because of the judge bill could grade all of them.

The second is placement. A judge that answers in under half a second can run inside the agent loop. LangChain's AutoModeMiddleware uses Jev to check tool calls and block risky ones before they execute. Once a judge sits there, its threshold decides what the agent may do.

Both shifts raise the cost of a wrong judge. More grades and inline gates spread a judge's errors across more decisions, with no person reviewing them.

Independent tests find Jev overconfident in specific ways

Anthus ran 8,801 labeled examples. On Choice questions, Jev reported 91.4% average confidence and was right 76.1% of the time. On yes/no questions, it reported 79.0% and was right 72.3% of the time. Isotonic regression on held-out labels cut calibration error from 0.117 to 0.008, and 200 to 500 labels did most of the work.

TypeSafe's limitations page lists nine known failure modes, including math, dates and adversarial content. Its example of a question and its negation scores 0.72 and 0.47, which sum to 1.19. In its consistency test, 15 runs of one question landed between 0.43 and 0.53, on both sides of a 0.5 threshold.

Barg Labs tested the job agent teams most want a cheap judge for: checking an agent's own report. They planted one false sentence in completion reports for 50 real pull requests and asked Jev whether each report was accurate. Jev passed 90% of premature "done" claims and 98% of fabricated references. Giving it the GitHub evidence cut fabricated passes to 68% and pushed wrong-file passes from 62% to 92%.

Evaluators need evaluation

Jev is an evaluator, and evaluators need evaluation. A cheaper judge leaves the work of checking the judge in place and hands it to the team that deploys it. The riskiest setup is the one Jev makes easiest: an unreviewed gate on agent actions, running on scores nobody has checked against the team's own traffic.

Three rules apply to Jev and to any judge that follows it.

Calibrate on your own traffic before trusting a score. Anthus got calibration error near zero with a few hundred labels. Alex Molas explains why a vendor cannot do this for you: calibration depends on the data distribution the judge sees.

Do not treat more judges as more safety. Guneet Kohli found that nine frontier judges from seven model families carried about two independent votes of information, and the best single judge matched the panel. Extra checks help only when their errors differ.

Keep the gate out of your tuning loop. In OpenAI's Baker et al., a reasoning model trained against a monitor learned to hide its intent and kept misbehaving. Tuning an agent's prompts until the gate goes quiet applies the same pressure.

For teams that do deploy a cheap judge, Trust or Escalate shows the safer shape. Cheap judges handled the confident cases and escalated the uncertain ones, and the cascade still guaranteed over 80% human agreement on 79.1% of ChatArena cases.

Cheaper grading changes the budget math. Knowing whether the grades are right still takes a labeled set from your own traffic, and no vendor ships that with the model.

Frequently Asked Questions

What is Jev?

Jev is a decision model from TypeSafe AI. It takes state and typed questions and returns probabilities for yes/no questions (Noul), a distribution over options (Choice), or a score. It does not generate text.

Can Jev replace an LLM-as-a-judge?

For bounded decisions it can do the same job at a fraction of the cost and latency. Independent tests show its probabilities drift from real accuracy, so a team needs to recalibrate on its own labeled data before using the scores as thresholds.

How many labels does recalibration take?

In the Anthus study, 200 to 500 labeled examples delivered most of the improvement, and isotonic regression averaged 2.6 points of calibration error at 500 examples against 5.4 for Platt scaling.

Is Jev deterministic?

No. TypeSafe's own consistency cookbook shows one question returning values between 0.43 and 0.53 across 15 runs. Cases near a threshold should be rerun or escalated.

Why not ask several judges and take the majority?

Judges trained on similar data make similar mistakes. Kohli measured a nine-judge panel carrying about two independent votes of information, so extra judges add cost faster than they add accuracy.

Sources

• Can you trust Jev's confidence? (Ryan Porter, Anthus, September 2026)

• Jev as a judge of agent completion reports (Barg Labs, September 2026)

• Jev 1.13 jaggedness (TypeSafe, September 2026)

• awesome-jev-robustness (community index, September 2026)

• Nine Judges, Two Effective Votes (Guneet Kohli, May 2026)

• Trust or Escalate (Jung, Brahman and Choi, July 2024)

• Monitoring Reasoning Models for Misbehavior (Baker et al., OpenAI, March 2025)

Stratix now runs evaluations on your own Bedrock or Azure OpenAI capacity through the new Settings > Providers page. Run your own benchmarks at stratix.layerlens.ai.