RAG Evaluation Basics: The Metrics That Matter

[VIDEO EMBED: Evals 101, Episode 8, once the YouTube upload is live. Leave out if the post publishes first.]

Suppose two foundation models post the same score on a public coding benchmark, and a team building a contract-summary assistant tries both on 20 of its own contracts. The first returns 17 usable summaries and misses the indemnity clause in three of them. The second returns 14 usable summaries, and one of the 14 states a termination date that appears nowhere in the contract.

The two models share a public score, and their failures look nothing alike. The first model's misses are visible and boring. The second model's single invented date is the kind of error that reaches a customer with a signature under it. Which one is "better" depends on a question the benchmark never asked.

TL;DR

• "Which model is best" has no answer because it has no object. "Which model, for this task set, under this latency floor and this cost ceiling" does.

• Four inputs decide the choice: the team's own task set, the data the model will see, the latency the surface can tolerate, and the token budget per case.

• Public benchmarks are the right tool for building a shortlist of four or five candidates. They are the wrong tool for the final pick.

• Run the same 20 cases across every candidate with the prompt, temperature, grader, and judge held fixed. Change only the model.

• Read the spread before the mean. Two candidates at 16 of 20 can share 12 passes and fail four different cases each, and the failed cases decide the choice.

• Record the run. The next model release reopens the question, and a comparison without its config cannot be rerun.

Throughout this post, "model" means the foundation model behind an agent or an assistant. The prompt, the tools, the retrieval, and everything else stay fixed while the model swaps.

"Best Model" Has No Object

A ranking answers "best at what," and the what is a task set someone else chose. A coding benchmark ranks models on its repositories, its issues, its grader. A team summarizing contracts works with different documents and a different definition of pass, and its failure costs are its own. The ranking's top entry might be the right pick, and the ranking alone cannot say so.

Public benchmarks still earn their place. A model in the bottom half of a coding benchmark is unlikely to top a coding task, and a reasoning benchmark is a fair first filter for a task that needs multi-step logic. Use the public scores to cut 200 candidates to five. The benchmarks post in this series covers how to read those scores without over-trusting them. The final choice comes from the team's own cases.

The question with an answer is: which of these five, on these 20 cases, within this latency and this token budget, fails in the way this product can tolerate.

Four Inputs Decide the Choice

The task set. The 20 real cases from LLM Evaluation for Beginners: twelve normal, five previously failed, three adversarial. For a contract assistant, twelve ordinary contracts, five that tripped the current model, three with a clause designed to confuse it. That set is the team's own benchmark, built from the inputs the product sees.

The data. The documents and tool outputs the model will see in production, in the formats they arrive in. A model that reads clean prose well can stall on a scanned PDF table or a JSON tool result with nested arrays. If the production input is messy, the test input is messy.

The latency floor. A chat surface needs a first token inside a limit a person will wait for. A nightly batch job does not care. Write the floor down as a p95 number per case before any candidate runs. A model that clears the floor on 19 of 20 cases and takes forty seconds on the twentieth has a problem the mean hides.

The cost ceiling. Tokens per case, multiplied by the monthly volume. Express it relative to the current model: a candidate that passes two more cases at three times the output tokens is a decision with two sides. A team that reads only the pass rate takes the trade without knowing it.

Write the four down before the comparison starts, because a run without them produces a winner nobody can justify next quarter.

[INSERT IMAGE: ai-model-comparison-four-inputs.png - Branded data visualization: a four-quadrant card showing the inputs to a model choice. Task set (20 cases: 12 normal, 5 failed, 3 adversarial), data (the production input formats), latency floor (a p95 number per case), cost ceiling (tokens per case times volume). Alt text: Diagram of the four inputs that decide a foundation model choice: task set, data, latency floor, and cost ceiling.]

Run the Same 20 Cases Across Every Candidate

The comparison is only valid if one thing changes. Same prompt, same temperature, same tool definitions, same grader, same judge and judge version, same day. Only the model swaps. A comparison where the prompt was tuned for one candidate and copied to the others is a comparison of the prompt.

Each case carries the pairing from the metrics post: one deterministic grader and one judge. For the contract assistant, a regex that fails any date not present in the source document, and a Hallucination Detection judge reading the summary against the contract. Both columns record per case, along with tokens in, tokens out, and latency.

In Stratix the same evaluation runs across any of 200 or more foundation models, and a BYOK endpoint registers any OpenAI-compatible model, so a fine-tuned or self-hosted candidate sits in the same table as the API models. Five candidates on one 20-case dataset means five runs and one table.

Then run each case three times per candidate. A case that passes two of three is a coin flip, and a candidate whose pass rate depends on the seed is telling the team something the single run would hide.

Read the Spread Before the Mean

Five pass rates come back, and sorting them by mean throws away most of what the run produced.

Two candidates at 16 of 20 can share 12 passes and fail four different cases each. One fails the three adversarial cases and one hard contract. The other passes every adversarial case and fails four ordinary ones. The means match and the risk profiles are opposite, and only the per-case view shows it.

Read the comparison in this order.

• Flips. Which cases pass on one candidate and fail on another. The flipped cases are where the models differ, and everything else is shared ground.

• Failure type. A missed clause and an invented clause are both failures and they cost different amounts. Sort the failures by what the judge reasoning says went wrong.

• Rerun variance. Cases that flip between runs of the same candidate. A candidate with three coin-flip cases has a wider true range than its single pass rate.

• The latency tail. The p95 per candidate, checked against the floor. A candidate with a fast median and a slow tail fails the floor for one user in twenty.

• The token spread. Which cases each candidate spends the most tokens on. A model that passes the hard cases by writing three times as much output is buying those passes with tokens, and the cost ceiling has to absorb that.

Stratix run comparison puts two evaluation runs side by side with the flipped cases highlighted, which is the first view on the list. For five candidates, compare each one against the current production model and read the flips in both directions.

[INSERT IMAGE: stratix-run-comparison-two-candidates.png - Real Stratix screenshot of the run comparison view with two evaluation runs of the same 20-case dataset on two different foundation models, the flipped cases highlighted, per-case grader and judge verdicts visible. Alt text: Stratix run comparison view showing two candidate models on the same test set, with the cases that flipped between pass and fail highlighted.]

Record the Run So the Choice Survives the Next Release

The comparison answers the question for one week. A new model release, a silent provider update, or a change in the team's own prompt reopens it, and the answer only holds if the original run can be rerun the same way.

Record the config with the result: model version and endpoint for each candidate, temperature, the prompt, the tool definitions, the dataset version, the grader and judge versions, and the per-case token counts and latency. Stratix stores a judge snapshot with every verdict (judge name, version, evaluation goal, model used), so the judge side of the record is pinned without setup. The live post on pinned traces covers holding the rest of the comparison fixed.

The table will point at one candidate. It will also show, per case, what that candidate gets wrong, which is the thing to write down next to the choice. A team that picks a model knowing its four failure cases can write those four into the next run and watch whether the next release fixes them.

Frequently Asked Questions

How do you compare AI models for a specific use case?

Build a 20-case test set from real inputs, fix the prompt, temperature, grader, and judge, and run every candidate model through the same set on the same day. Read the per-case flips, failure types, rerun variance, latency tail, and token spread before looking at the mean pass rate.

Which AI model is the best?

The question needs an object. For a given task set, data, latency floor, and cost ceiling, one candidate will fail in the way the product tolerates best, and that is the pick. The answer changes with the task and with each new release.

How many test cases do you need to compare models?

Twenty is a working start, split twelve normal, five previously failed, three adversarial. Run each case three times per candidate to see which passes are stable.

Can public benchmarks be used to choose a model?

For the shortlist, yes. A model near the bottom of a relevant benchmark is unlikely to top a similar task. For the final pick, the team's own cases decide, because the benchmark's task set is not the team's.

How do you compare a self-hosted or fine-tuned model against API models?

Register its endpoint alongside the others so it runs the same dataset with the same grader and judge. A BYOK registration of any OpenAI-compatible endpoint puts it in the same comparison table.

How often should a model comparison be rerun?

Whenever a candidate changes: a new release or provider update on the model side, or a change to the team's own prompt or tools. With the config recorded, the rerun is the same evaluation with one input changed.

Next in Evals 101: the series loops back to LLM Evaluation for Beginners: How It Actually Works, which builds the 20-case set every comparison in this post assumes. Further reading: LLM Benchmarks Explained: What the Scores Mean, the live post on pinned traces, and the model comparison guides in the Stratix docs. Run your 20 cases across five candidates in Stratix.