Your RAG Pipeline Scores 0.91 Faithfulness. It Still Hallucinates.

By The LayerLens Team | September 1, 2026 | 11 min read

A legal research team deployed a RAG pipeline with 0.91 faithfulness. The number looked solid. Six weeks later, they discovered that one in six customer responses missed critical case information. Context recall had silently dropped to 0.62 on multi-hop queries. Faithfulness never flagged the problem because faithfulness does not measure what most teams think it measures.

Retrieval-augmented generation promised to fix hallucination by grounding LLM outputs in retrieved documents. The idea was sound. The evaluation layer was not. Most RAG evaluation stacks measure retrieval and generation independently, then report the higher number. The pipeline breaks at the seam between them, and the metrics miss it.

[INSERT IMAGE: post7-hero.png - Visual showing RAG faithfulness score vs actual production hallucination gap]
Image URL: https://litter.catbox.moe/egysgz.png

TL;DR

• Retrieval accuracy explains only 60% of RAG output-quality variance. The other 40% depends on how well the generator uses the retrieved context, and no standard metric captures this.

• 47% to 67% of queries show the generator ignoring the top-ranked retrieved document. The retrieval step succeeded. The generation step discarded it.

• 22% of retrieved evidence is duplicated across chunks, wasting context budget on redundant information that displaces relevant passages.

• Systems scoring 0.92 offline on gold-set evaluations dropped to 0.78 in production (arXiv:2504.14891). Offline faithfulness overstates production faithfulness by 15 to 20 points.

• 70% to 80% of enterprise RAG projects never reach production. The evaluation layer signs off on metrics that do not predict deployment success.

• 26% of medical advice sampled from Microsoft Copilot contained incorrect or potentially harmful information, despite the system using retrieval grounding.

Faithfulness Measures the Wrong Boundary

Faithfulness scores whether the generated answer is supported by the retrieved context. If the retrieved context is wrong, incomplete, or stale, a faithful answer reproduces those errors. A RAG pipeline can achieve 0.95 faithfulness while serving confidently wrong answers, because faithfulness does not evaluate the retrieval step.

This is the core design flaw in most RAG evaluation stacks. They decompose the pipeline into retrieval metrics (precision, recall, NDCG) and generation metrics (faithfulness, answer relevancy) and evaluate each independently. The failure mode lives at the boundary: the generator receives context, decides what to use, and produces an answer. That decision is where hallucinations enter, and neither metric family captures it.

The legal RAG example illustrates the failure pattern. Faithfulness stayed at 0.91 because the generator accurately reflected the documents it chose to use. Context recall dropped to 0.62 because the generator systematically ignored documents containing multi-hop reasoning chains. The system was faithfully incomplete.

The Generator Ignores What Retrieval Finds

A 2026 analysis of production RAG systems found that 47% to 67% of queries resulted in the generator ignoring the highest-ranked retrieved document. The retrieval step surfaced the right information. The generation step used a lower-ranked passage instead, or synthesized an answer from partial fragments across multiple chunks.

This pattern explains why retrieval accuracy predicts only 60% of output quality. The remaining 40% depends on context utilization: how effectively the generator incorporates retrieved information into its answer. Context utilization above 40% is considered acceptable. Most production systems fall below that threshold on complex queries.

[INSERT IMAGE: post7-retrieval-gap.png - Chart showing retrieval accuracy vs end-to-end answer quality gap across different query types]
Image URL: https://litter.catbox.moe/d4isr1.png

The problem compounds with chunk duplication. 22% of retrieved evidence in typical RAG deployments is duplicated across chunks. Each duplicate consumes context window budget without adding information. A system retrieving 20 chunks at k=20 effectively operates with 15.6 unique chunks. On long-context queries where every chunk matters, that 22% loss is the difference between a complete answer and a partial one.

Offline Scores Do Not Predict Production Performance

A survey of RAG evaluation methods (arXiv:2504.14891) documented a consistent pattern: systems scoring 0.92 on offline gold-set evaluations dropped to 0.78 in production. The 14-point gap is not random variance. It is structural.

Gold-set evaluations use curated question-answer pairs where the correct retrieval targets are known. Production queries are messier. They contain ambiguity, implicit context, and multi-step reasoning requirements that gold sets rarely represent. The retrieval step optimizes for the gold set during development and never encounters the distribution of real queries until deployment.

This is the same pattern documented in agent production failures: evaluation in controlled environments produces numbers that do not transfer. RAG pipelines add a specific failure mode. The retrieval index changes as documents are added, updated, or deleted. A pipeline that scored 0.92 against a static corpus in March may score 0.78 against the same corpus in June because the index has drifted.

The Five RAG Failure Modes Standard Metrics Miss

Research from 2026 (arXiv:2506.00054, arXiv:2605.11330) identifies five failure modes that standard RAG metrics routinely miss.

Context window poisoning. When retrieved chunks contain contradictory information, the generator must choose between them. Standard retrieval metrics score both chunks as relevant. The generator picks one, and the answer depends on which one it picks. Faithfulness scores high either way. The answer may be wrong.

Multi-hop reasoning failure. Queries requiring information from multiple documents expose a gap between chunk-level retrieval and answer-level synthesis. A system can retrieve all necessary chunks (high context recall) and still fail to connect them into a coherent answer. The legal RAG failure was this pattern: all relevant documents were retrieved, but the generator could not chain the reasoning across them.

Temporal drift. Documents in the retrieval corpus become stale. A financial RAG that retrieved a Q2 earnings report when Q3 data was available produces a faithful, outdated answer. No standard RAG metric checks temporal validity.

Confidence without coverage. The generator produces a confident, well-structured answer from a subset of the retrieved context. The answer reads as complete. It is not. The missing information was retrieved but not used. Faithfulness passes. The user does not know what they are missing.

Retrieval-generation feedback loops. In agentic RAG systems where the model generates follow-up queries based on initial retrieval results, errors compound. A slightly off initial query retrieves slightly wrong context, which generates a slightly worse follow-up query. Three iterations later, the system is confidently answering a question the user did not ask.

Real-World RAG Failures Tell the Same Story

Air Canada's chatbot invented a bereavement fare refund policy that the airline never offered. The system retrieved real fare policy documents. The generator synthesized a policy from fragments across multiple documents that sounded plausible but did not exist. Air Canada was held legally liable for the chatbot's fabrication. Faithfulness to individual retrieved passages was high. The synthesized answer was fiction.

NYC's MyCity chatbot advised small business owners to take actions that violated city regulations. The retrieval step found relevant regulatory documents. The generator extracted permissions without the associated restrictions, producing answers that were technically grounded in real documents but practically illegal.

Microsoft Copilot's medical advice showed 26% of sampled responses containing incorrect or potentially harmful information, despite retrieval grounding. The system retrieved medical literature. The generator simplified, elided qualifications, or combined findings from incompatible studies.

Every case follows the same pattern: retrieval worked, generation failed, and the evaluation metrics reported success.

[INSERT IMAGE: post7-failure-modes.png - Diagram of the 5 RAG failure modes with examples mapped to each]
Image URL: https://litter.catbox.moe/7neuvw.png

What RAG Evaluation Actually Requires

The research converges on production thresholds that go beyond faithfulness and retrieval precision. Organizations operating mature evaluation frameworks track six metrics simultaneously.

Faithfulness above 0.8 minimum, above 0.9 for regulated industries. This is the floor, not the ceiling. Faithfulness alone tells you whether the generator stayed within the retrieved context. It does not tell you whether the retrieved context was correct, complete, or current.

Context recall above 0.75 at k=20. This measures whether the retrieval step found the information needed to answer the query. Below 0.75, the generator is working with incomplete context on at least one in four queries.

Context precision above 0.7 for narrow domains, above 0.5 for broad domains. This measures how much of the retrieved context is actually relevant. Low precision means the generator must filter noise, which increases the chance of hallucination.

Answer relevancy above 0.75. This measures whether the generated answer actually addresses the question asked. A faithful, well-grounded answer to the wrong question is still a failure.

Context utilization above 40%. This is the metric most teams do not track. It measures what fraction of the retrieved context the generator actually used. Below 40%, the system is retrieving information and discarding it.

Hallucination rate below 5%. This requires a separate evaluation pass, not a derivation from faithfulness. Hallucination detection benchmarks (arXiv:2605.08462, arXiv:2505.04847) show that faithfulness-based hallucination estimates undercount by 15% to 30%.

End-to-End Evaluation Beats Component Testing

The fundamental problem with RAG evaluation is component decomposition. Testing retrieval in isolation and generation in isolation produces two passing grades that, combined, produce a failing pipeline. The fix is end-to-end evaluation: feed the pipeline a query, capture the full trace (query, retrieved chunks, chunk rankings, generated answer, cited sources), and evaluate the trace as a unit.

This is the approach that caught the legal RAG failure. Component metrics showed 0.91 faithfulness and 0.83 context recall. End-to-end trace evaluation showed that multi-hop queries produced answers missing critical information 16% of the time. The component metrics were both true and misleading. The trace told the real story.

How Stratix Evaluates RAG Pipelines

Stratix evaluates RAG at the trace level, not the component level. Every evaluation captures the full pipeline trace: the input query, the retrieval results with rankings, the generated answer, and the citations. The seven-field schema (input identifier, model version, agent configuration, judge name and version, evaluation goal, score, timestamp) applies to each trace as a unit.

Stratix's retrieval-generation coupling analysis measures the gap between what was retrieved and what was used. When context utilization drops below threshold, Stratix flags the specific queries where the generator ignored high-ranked retrieval results. Teams can see exactly which queries produce faithful but incomplete answers.

Deliberation Panels evaluate RAG traces across multiple judges and multiple orderings, the same approach that eliminates position bias in pairwise comparisons. For RAG evaluation, this means the faithfulness score is not dependent on a single judge's interpretation of "supported by the context."

Frequently Asked Questions

Why does high faithfulness not prevent hallucination in RAG systems?

Faithfulness measures whether the generated answer is supported by the retrieved context. If the retrieval step returns wrong, incomplete, or contradictory information, a faithful answer reproduces those problems. The generator can also synthesize plausible-sounding answers from fragments across multiple documents that, individually, support the synthesis but collectively create fiction. Air Canada's chatbot demonstrated this pattern.

What is context utilization and why does it matter?

Context utilization measures what percentage of retrieved information the generator actually incorporated into its answer. Production RAG systems routinely retrieve relevant documents and then ignore them. When context utilization falls below 40%, the system is discarding more information than it uses, which means retrieval improvements will not improve answer quality until the generator starts using what it receives.

How large is the gap between offline and production RAG performance?

Systems scoring 0.92 on offline gold-set evaluations typically drop to 0.78 in production, a 14-point gap documented across multiple deployments (arXiv:2504.14891). The gap is structural: gold sets do not represent the distribution of real queries, and the retrieval corpus changes over time while the gold set stays static.

What causes multi-hop reasoning failures in RAG?

Multi-hop queries require connecting information from multiple retrieved documents. The retrieval step may surface all necessary chunks (high context recall), but the generator fails to chain reasoning across them. This produces answers that are partially correct based on individual documents but miss the full picture. Legal and medical domains are especially vulnerable because answers often depend on combining regulations, precedents, or clinical findings from separate sources.

Can chunk size tuning fix RAG hallucination problems?

Chunk size affects retrieval precision and context window usage but does not fix the retrieval-generation boundary problem. Smaller chunks improve precision at the cost of losing cross-document context. Larger chunks preserve context but introduce noise. The 22% duplication rate across chunks suggests that chunk overlap settings, not just size, need evaluation. Tuning chunk parameters without measuring end-to-end answer quality is optimizing a component metric that may not improve pipeline performance.

How does Stratix detect RAG pipeline drift?

Stratix records every evaluation trace with a timestamp and the retrieval corpus version. When the corpus changes (documents added, updated, or deleted), Stratix can compare evaluation results before and after the change. A faithfulness score that holds steady while context recall drops signals corpus drift. The seven-field schema makes these comparisons queryable without re-running historical evaluations.

Stop Evaluating Components. Start Evaluating Traces.

RAG evaluation has a measurement problem, not a model problem. The metrics exist. The failure is in how teams combine them. Component-level evaluation produces two passing grades for a failing pipeline. Trace-level evaluation catches the failures that live at the boundary between retrieval and generation.

The path forward is not more metrics. It is measuring the right thing: the full pipeline trace, evaluated as a unit, across the distribution of production queries, with temporal drift detection built into the evaluation loop.

Stratix provides trace-level RAG evaluation with retrieval-generation coupling analysis, temporal drift detection, and Deliberation Panels for judge-independent faithfulness scoring. Start evaluating your RAG pipeline end-to-end, not component by component.