How to Test a RAG System for Hallucinations (Faithfulness & Grounding)

Retrieval-Augmented Generation is supposed to stop hallucination by grounding the model in retrieved evidence. In practice it introduces a subtler, more dangerous failure: the model receives the right context and still asserts something that context does not say. Testing a RAG system for hallucinations means testing that specific contract - faithfulness to the retrieved context - and it needs a different method from testing a plain LLM.

This is the RAG-specific companion to our hallucination testing techniques guide. Here is the workflow end to end.

Grounding Hallucination Is the Failure That Matters

In a RAG system there are three ways an answer can be wrong, and only one is unique to retrieval:

• The answer is factually wrong (a general LLM problem).
• The answer is irrelevant to the question.
• The answer is not grounded in the retrieved context - it adds, distorts, or invents claims the passages do not support.

The third is grounding hallucination, and it is the one your test suite must isolate. It is especially insidious because a grounded-but-wrong retrieval (bad source document) and a hallucinated generation (good source, ungrounded answer) look identical to the user but require opposite fixes. Measuring one number for “accuracy” hides which is happening.

Step 1: Split Retrieval From Generation

Before scoring answers, decide which stage you are testing. RAG has two, and they fail for different reasons:

• Retrieval - did the system surface the passages that contain the answer? Measure context precision (are the retrieved passages relevant) and context recall (did retrieval capture everything needed).
• Generation - given those passages, did the model answer faithfully? Measure faithfulness (groundedness) and answer relevancy.

The diagnostic rule is simple. Low context recall means the answer failed because the evidence was never retrieved - fix chunking, embeddings, or the retriever. High recall but low faithfulness means the model had the evidence and hallucinated anyway - fix the prompt, the model, or add a grounding guardrail. Testing a single blended score cannot tell you which.

Step 2: Score the RAG Triad

The cleanest framing is the RAG triad: context relevance, groundedness, and answer relevance. Score all three per response and the failing stage reveals itself.

• Context relevance low → retriever is pulling noise.
• Groundedness / faithfulness low → generation is straying from context.
• Answer relevance low → the model is grounded but not answering the question.

TruLens implements the triad directly by instrumenting the app; RAGAS exposes the same dimensions as offline metrics you can run over a test set.

Step 3: Verify at the Claim Level

A holistic “is this answer faithful?” verdict is too coarse - a mostly-correct answer with one fabricated sentence scores as “faithful enough” and ships. The reliable method is claim-level decomposition:

1. Break the response into atomic claims.
2. For each claim, classify it against the retrieved context as supported, contradicted, or not mentioned.
3. Faithfulness is the proportion of claims that are supported.

This is exactly what RAGAS faithfulness does under the hood, and what purpose-built models like Patronus Lynx and Vectara HHEM do cheaply enough to run on every request. Claim-level scoring catches the single fabricated sentence a whole-answer judge waves through.

Step 4: Build a Context-Grounded Evaluation Set

Unlike a plain-LLM test set, a RAG evaluation set carries the context with each case: the question, the passages that should ground the answer, and a domain-expert-verified ground-truth answer. Target 100-200 pairs reflecting real query distribution.

Two categories are essential and often skipped:

• Answer-not-in-corpus cases - questions whose answer your knowledge base does not contain. A well-behaved system abstains (“I don’t have that information”); a hallucinating one invents an answer. This directly tests the failure that hurts most.
• Conflicting-source cases - questions where two retrieved documents disagree, testing whether the model flags the conflict or silently picks one.

If your knowledge base itself is stale or contradictory, even a faithful model will look wrong - which is why a training data quality audit is often the first fix.

Step 5: Wire It Into CI

Grounding behaviour is fragile: a new chunk size, a swapped embedding model, a reworded prompt, or a model upgrade can all reintroduce hallucination. Treat faithfulness as a regression test.

Expose a subset of your evaluation set as programmatic checks with RAGAS or DeepEval, gate the build on a faithfulness threshold, and use a lightweight model such as Lynx or HHEM where you need per-request scoring cheap enough to run continuously. A RAG change that is not re-scored is a RAG change that is untested.

The Short Version

Testing a RAG system for hallucination is not “measure accuracy.” It is: split retrieval from generation, score the RAG triad, verify faithfulness at the claim level, build a context-grounded set that includes abstention cases, and gate it in CI. Do that and grounding hallucination stops being a surprise in production and becomes a number you watch on every release.

For the full technique catalogue see AI hallucination testing techniques, and for the tools that implement each check see the hallucination detection tools comparison. When you want a context-grounded RAG test suite built and benchmarked for you, that is our AI Product QA and LLM Evaluation work.