LLM Evaluation Metrics: The Only 7 You Need

[VIDEO EMBED: Evals 101, Episode 3, once the YouTube upload is live. Leave out if the post publishes first.]

Suppose a returns assistant gets the question "Can this be returned after 45 days?" and answers in four sentences. The window is 30 days, a manager can approve exceptions, opened items need the original receipt, and a 15 percent restocking fee applies. The reference answer in the eval set is one sentence long and never mentions a fee, because no fee exists.

Run seven metrics on that single output and seven verdicts come back. Exact match fails it on the first character. Semantic similarity passes it, because the answer talks about the right policy. A faithfulness judge fails it on the invented fee. A relevance judge passes it. Tool-call correctness fails it, since the assistant never looked up the order. The cost column shows the answer used about three times the tokens of the reference. Every verdict is correct, and each one is measuring a different thing.

TL;DR

• Seven metrics cover nearly every LLM test a team will write: exact match, semantic similarity, faithfulness, relevance, tool-call correctness, trajectory pass rate, and cost with latency.

• Three of the seven run as code with no model involved (exact match, semantic similarity, cost and latency). The other four need a judge that reads the output.

• Every metric has a blind spot. Exact match fails paraphrases, similarity passes fluent wrong answers, faithfulness ignores relevance, and an average latency hides the p95.

• The pairing rule: each test carries one deterministic metric and one judged metric. Disagreement between the two is the most useful signal on the run page.

• The MT-Bench study (arXiv:2306.05685, Table 5, setup S2) measured 85 percent judge-human agreement against 81 percent human-human, so a judged metric is about as reliable as a second reviewer once calibrated.

• In Stratix the deterministic side comes from six code graders and the judged side from prebuilt judges such as Hallucination Detection and Tool Usage Quality, with token counts on the same run page.

Exact Match Is Free and Fails Every Paraphrase

Exact match compares the output to a reference string, character for character. It costs nothing to run, and the verdict is identical every single time. For a classification label, a SQL statement, a yes/no field, or a structured record, it is the right first metric and often the only one needed.

For free text it fails almost everything. "The window is 30 days" and "Returns are accepted within 30 days" say the same thing, and exact match rejects one of them. Regex match and JSON-schema validity belong to the same family: cheap and deterministic, and narrow in what they catch. A regex catches a percent sign the assistant was told never to quote. A schema check confirms the two required fields exist before anyone reads the values.

Stratix ships six code graders in this family: exact match, regex match, JSON-schema validity, semantic similarity, Flesch-Kincaid readability, and fairness math. None of them calls a model. Reach for one of these first on every test, and add a judge only where the rubric needs reading comprehension.

Semantic Similarity Passes Fluent Wrong Answers

Semantic similarity embeds the output and the reference as vectors and measures the angle between them. Above a threshold, pass. It handles paraphrase, which is the whole reason it exists. No judge is needed.

It lies in a specific way. A fluent wrong answer sits very close to the right one in embedding space. "The window is 30 days" and "The window is 60 days" score almost identically, because both sentences are about a return window. The metric measures whether the output discusses the same thing as the reference. It has no opinion on whether the output is true.

It also lies in the other direction. A correct answer written as a bulleted list can score below threshold against a reference written as a paragraph. So pick the threshold from data. Run the grader over twenty outputs a person already marked good and twenty marked bad, and set the line where the two groups separate. Expect some overlap, and hand that overlap to a judge.

Faithfulness and Relevance Catch Two Different Failures

Both of these need a judge: a second model that reads the output against a written criterion and returns pass or fail with a reason. The live post on LLM as a judge covers how the judge prompt works and how to calibrate one.

Faithfulness asks one question: does the material the model was given support every claim in the answer? The material might be a policy document, a retrieved chunk, or a tool result. The restocking fee fails here, because nothing in the policy mentions a fee. The Hallucination Detection judge runs this check and returns a score with its reasoning.

Relevance asks a different question: does the answer address what the user asked? An answer about the exchange policy can be perfectly faithful to the handbook and still fail a return question. The two metrics fail independently. An output can be faithful and off-topic, or on-topic and invented, and a team that runs only one of them will miss the other failure entirely.

Judges carry biases, which is where these two metrics lie. A judge model tends to prefer outputs from its own family and tends to reward longer answers. The fix is calibration against human labels, and the ceiling is known. The MT-Bench human-agreement study, arXiv:2306.05685, Table 5, setup S2, found 85 percent agreement between judge and human against 81 percent between two humans. A calibrated judge disagrees with a reviewer about as often as reviewers disagree with each other.

[INSERT IMAGE: llm-evaluation-metrics-seven-verdicts.png - Branded data visualization: one returns-assistant answer on the left and seven metric rows on the right. Rows: exact match, semantic similarity, faithfulness, relevance, tool-call correctness, trajectory pass rate, cost and latency. Each row shows its verdict on that answer plus a one-line "where it lies" note. Alt text: Table of seven LLM evaluation metrics applied to one assistant answer, showing each verdict and the failure each metric misses.]

Tool-Call Correctness Grades the Name, the Arguments, and the Timing

Any assistant that calls a function needs this metric. It has three parts, and teams usually check only the first. Did the model call the right tool? Did it pass the right arguments? Did it call the tool at the point where it needed the result?

The deterministic version compares the tool name and the argument JSON to an expected call. Exact match works well here because the call is structured data. The judged version reads the call in context. The Tool Usage Quality judge asks whether the agent called the lookup when it needed the order, whether it passed the order id it was given, and whether it used the result in the answer that followed.

The deterministic version lies when the expected call is too strict. Two valid orderings of the same two calls should both pass, and "2026-09-01" and "September 1, 2026" are the same argument to a human and different strings to a grader. Write the expected call as loosely as the task allows, then let the judge cover the rest.

Trajectory Pass Rate Is the Only Metric That Sees the Path

The first five metrics grade one output. An agent produces a sequence: read the task, pick a tool, read the result, pick again, and eventually answer. Trajectory pass rate is the share of runs where the whole path met its rules. No forbidden action, within the step budget, required steps present, each decision sound given what the agent knew.

The rules split into two kinds. Assertions are code: never exceed ten calls per task, never call a destructive tool without a confirmation step, never loop the same call with the same arguments twice. Judged rules need reading: the Agent Trajectory judge scores whether each step made sense from where the agent stood. The companion post on AI agent evaluation walks through both.

This metric lies when the rule set encodes one path as the only acceptable path. An agent that reaches the right answer in four steps by a different route than the reference should pass. Write trajectory rules as constraints on what must never happen and what must happen at least once, and leave the order open.

Cost and Latency Belong in the Same Table as Accuracy

Tokens in, tokens out, and wall-clock time per case are metrics, and they get reported next to the pass rate or they get ignored. A prompt change that raises the pass rate by three points and doubles the output tokens is a trade, and the team should see both numbers before deciding whether to take it.

Latency lies when it arrives as an average. A p50 of two seconds with a p95 of nineteen seconds describes a system that feels fast in the demo and stalls for one user in twenty. Record the distribution per case, and read the tail before the middle.

Stratix shows token counts per evaluation on the run page, so the cost column sits in the same row as the grader verdict and the judge verdict. Keeping the seven on one page means a case that passes accuracy and fails the token budget still reads as a failure.

[INSERT IMAGE: stratix-run-page-grader-judge-tokens.png - Real Stratix screenshot of an evaluation run page with per-case results. The table shows a code grader column, a judge verdict column, and the token count per evaluation. Alt text: Stratix evaluation run page listing test cases in rows, with a code grader result, a judge verdict, and token counts side by side for each case.]

The Pairing Rule: One Deterministic Metric and One Judge per Test

Every test gets two metrics. One deterministic (exact match, regex, schema, similarity, or a cost budget) and one judged (faithfulness, relevance, tool usage, or trajectory). The deterministic one is stable and free and catches format regressions on every commit. The judge catches meaning.

The pairing pays off when the two disagree, which is the main thing to read on a run page. Judge pass with grader fail usually means the reference or the regex is wrong, and the case needs editing before the model does. Grader pass with judge fail means the output has the right shape and the wrong content, which is the failure that reaches customers. When both pass, the case ships, and when both fail the team has a real bug.

For the returns assistant from the opening, the five cases pair like this. The 45-day question takes a regex that fails any percent sign plus the Hallucination Detection judge. The JSON case takes a schema check plus the Relevance judge. The order-lookup case takes exact match on the tool name and arguments plus the Tool Usage Quality judge. Twenty cases produce forty verdicts on one page.

Frequently Asked Questions

What are LLM evaluation metrics?

Ways of scoring a language model's output against a test case. They split into deterministic metrics computed by code (exact match, regex, schema validity, semantic similarity, token cost, latency) and judged metrics where a second model reads the output against a written criterion (faithfulness, relevance, tool usage, trajectory quality).

Which LLM evaluation metric should a team start with?

Whichever deterministic one fits the output format. Exact match for labels and structured records, JSON-schema validity for structured output, regex for a forbidden phrase. Add a judged metric on the same test once the deterministic one is passing.

What is the difference between faithfulness and relevance?

Faithfulness checks whether the material the model was given supports every claim in the answer. Relevance checks whether the answer addresses the question asked. An answer can pass one and fail the other, so both run on any test where the model works from provided context.

Do you still need BLEU or ROUGE for LLM evaluation?

Rarely. Both compare word overlap with a reference and fail paraphrase the same way exact match does, with less precision. Semantic similarity replaces them for free-text closeness, and a judge replaces them for correctness.

What is a good semantic similarity threshold?

One chosen from data. Score twenty outputs already labeled good and twenty labeled bad, and set the threshold where the groups separate. A threshold copied from another team's dataset is a guess.

How many metrics should one test case have?

Two: one deterministic and one judged. More than two on a single case usually means the case is testing two things and should be split.

Next in Evals 101: RAG Evaluation Basics: The Metrics That Matter takes faithfulness and relevance into a retrieval pipeline, where the wrong chunk can produce a fluent, cited, and false answer. Further reading: What Are AI Evals? A Plain-English Guide, the live post on LLM as a judge, and the code grader and judge reference in the Stratix docs. Pair a grader with a judge on your own twenty cases in Stratix.