RAG Evaluation Basics: The Metrics That Matter

Suppose an internal knowledge assistant gets the question "What is the notice period for a contractor?" and answers in two clean sentences: 30 days, with a citation to the handbook, section 4. The answer reads well, carries a citation, and is wrong, because the contractor handbook says 14 days.

The retriever pulled the employee handbook chunk, which does say 30 days. The generator wrote a faithful summary of that chunk. The citation formatter attached the section number correctly. Every component did its job, and the pipeline still gave a person a wrong number with a source attached, which is worse than no answer at all.

TL;DR

• A RAG pipeline fails in three places (retrieval, generation, the whole), and one end-to-end score cannot tell them apart.

• Retrieval precision and recall grade the chunks before a word is generated. They run as code once each question has its gold chunks labeled.

• Faithfulness asks whether the retrieved context supports every claim in the answer. An answer can be fully faithful to the wrong chunk.

• A 30-question RAG test set splits 18 single-document questions, six that span two documents, three the corpus cannot answer, and three where two documents conflict.

• The failure that slips past the end-to-end score is the right answer with the wrong source. It passes the end-to-end score and fails the next question that looks like it.

• The LayerLens RAG Quality judge runs five checks (retrieval relevance, retrieval coverage, faithful synthesis, citation accuracy, appropriate hedging), passes at 0.8, and fails on any unsupported claim.

RAG Fails in Three Places and One Score Cannot Tell Them Apart

Retrieval-augmented generation has two moving parts and one result. The retriever picks chunks from a corpus. The generator writes an answer from those chunks. The user sees the answer. A team that scores only the answer knows the pipeline failed and has no idea where.

So RAG evaluation is three measurements. Retrieval quality asks whether the right chunks came back. Generation quality asks whether the answer stayed inside those chunks. End-to-end quality asks whether the user got the right answer. Each one can pass while another fails, and the opening scene is the proof: generation passed while end-to-end failed, and the cause sat in retrieval.

Many teams check one of the three, usually the last. That is the score a stakeholder asks for, and it is the only one of the three that hides its own cause.

Retrieval Precision and Recall Grade the Chunks Before Any Words Exist

For each test question, someone labels which chunks in the corpus contain the answer. Those are the gold chunks. The retriever returns its top k, and two numbers follow.

Precision is the share of returned chunks that are gold: relevant retrieved divided by total retrieved. Recall is the share of gold chunks that came back: relevant retrieved divided by total relevant. A retriever that returns five chunks, two of them gold, out of three gold chunks in the corpus, scores a precision of 0.4 and a recall of 0.67.

Both run as code with no judge. Once the gold labels exist, a script compares chunk ids and writes two columns. The labeling is the work, which is why the test set section below spends most of its space on it.

Precision lies when k is small and the corpus has near-duplicates: a retriever can return two copies of the same paragraph and score well. Recall lies when the labeler missed a chunk, which puts a ceiling on the score that has nothing to do with the retriever. Label carefully and then trust the two numbers.

Faithfulness Asks Whether the Answer Stayed Inside the Context

Faithfulness needs a judge: a second model that reads the answer next to the retrieved chunks and checks each claim against them. The live post on LLM as a judge explains how such a judge is written and calibrated. For RAG, the criterion is narrow. Every factual claim in the answer must trace to a sentence in the retrieved context. A claim with no support fails the answer, however true it happens to be.

The LayerLens RAG Quality judge runs five checks on a trace: retrieval relevance, retrieval coverage, faithful synthesis, citation accuracy, and appropriate hedging. It passes at a score of 0.8 or above and fails outright if the retrieved documents leave any claim unsupported. The verdict comes back with reasoning that names which check failed and why.

Faithfulness is relative to the context, and that is the property to hold onto. The opening answer was faithful. The generator wrote exactly what the employee handbook chunk said. A faithful answer to the wrong chunk fails the user and passes this metric, which is the reason the retrieval columns exist beside it.

End-to-End Answer Quality Is the Score the User Feels

The third measurement compares the final answer to a reference answer. Semantic similarity gives a cheap first pass, and a judge with a correctness rubric gives the real verdict. This is the score that appears on a dashboard, and it is the right one to report as long as the other two sit next to it.

Read the three columns together and the failure sorts itself.

• Retrieval fails: the chunking or the embedding model needs work. The generator never had a chance.

• Retrieval passes and faithfulness fails: the generator invented or drifted. The prompt or the model is the problem.

• Retrieval and faithfulness pass and the answer fails: the reference is wrong or the question is ambiguous, and either way the test case needs editing before the pipeline does.

• All three pass: ship.

A team with only the third column sees the first three rows as the same failure and fixes the wrong component first.

Build the 30-Question Test Set From Real Questions

The questions come from users. Search logs, support tickets, the Slack channel where people ask the question they could not find an answer to. Invented questions test the corpus the way the team imagines it gets used, and real ones test it the way it does.

Thirty questions is enough to learn something, and the split matters more than the count.

• Eighteen questions with a single clear answer in one document. These test the basic path and should mostly pass.

• Six questions whose answer spans two documents. These test recall, because a retriever that returns one of the two gold chunks produces a half answer that reads as complete.

• Three questions the corpus cannot answer. The right response is "the documents do not cover this," and these test the hedging check. A pipeline that answers them anyway has a faithfulness problem hiding behind confidence.

• Three questions where two documents conflict, like the employee and contractor handbooks. These test source selection and produce the failure the next section is about.

Each question carries four fields: the question text, the gold chunk ids, a reference answer, and a one-line rubric for the judge. The pattern is the same twenty-case shape from LLM Evaluation for Beginners, with the gold chunks added as a column.

Right Answer, Wrong Source Is the Failure a Green Run Hides

Take a question where the pipeline returns the correct number, faithful to its chunk, matching the reference. Then look at the cited chunk id. It is one the labeler never marked gold. The pipeline found the right answer in the wrong document, because a different document happened to contain the same value.

End-to-end and faithfulness both pass, and precision is the only column that shows the miss, provided someone reads it.

This failure matters for two reasons. The next question with a different answer in that wrong document will fail, and the team will not see it coming because the previous run was green. And a cited answer that is right by accident cannot be defended to anyone who checks the citation, and in a regulated setting the citation exists so that someone will check it.

The check is cheap. Compare the cited chunk ids to the gold chunk ids as a code grader, and let the RAG Quality judge's citation accuracy check read the cases where the ids differ. Count the right-answer-wrong-source cases on the first run. That count is the number of correct answers the pipeline got by luck.

Frequently Asked Questions

What is RAG evaluation?

Measuring a retrieval-augmented generation pipeline in three places: whether the retriever returned the right chunks (precision and recall), whether the generated answer stayed inside those chunks (faithfulness), and whether the final answer was correct (end-to-end quality).

What metrics are used to evaluate RAG?

Three: retrieval precision and recall against labeled gold chunks; faithfulness of the answer to the retrieved context; end-to-end answer quality against a reference. Citation accuracy and appropriate hedging cover two failures the first three miss.

How do you measure retrieval quality in RAG?

Label the gold chunks for each test question, then run the retriever and compare chunk ids. Precision is gold chunks returned divided by chunks returned; recall is gold chunks returned divided by gold chunks that exist. Both run as code.

What is faithfulness in RAG evaluation?

Whether a sentence in the retrieved context supports every claim in the answer. A judge reads the answer beside the chunks and fails any claim without support. Faithfulness is relative to the context, so a faithful answer to the wrong chunk still passes it.

How many test questions does a RAG eval set need?

Thirty is a working start: eighteen single-document questions, six that span two documents, three the corpus cannot answer, and three where two documents conflict. Each carries the question, gold chunk ids, a reference answer, and a one-line rubric.

Can you evaluate RAG without ground truth?

Faithfulness can run without a reference answer, because it compares the answer to the retrieved chunks. Retrieval precision and recall need gold chunk labels, and end-to-end quality needs a reference answer or a rubric a judge can apply.

Next in Evals 101: AI Model Comparison: How to Pick a Model runs the same test set across several candidate models and reads the spread of per-case results before the mean. Further reading: LLM Evaluation Metrics: The Only 7 You Need, the live post on LLM as a judge, and the RAG evaluation recipes in the LayerLens docs. Label thirty questions and run all three measurements on your own pipeline in LayerLens.