A RAG system that gives a wrong answer could have retrieved the wrong documents, retrieved the right documents and ignored them, or retrieved the right documents and misread them. A single pass/fail accuracy score can’t distinguish between these — which is exactly what component-level RAG metrics are for.
Retrieval-augmented generation has two distinct failure surfaces — the retrieval step (did it…
Context relevance — of the documents or chunks retrieved, how many are actually relevant to…
The diagnostic value is in how these metrics combine.
Retrieval-augmented generation has two distinct failure surfaces — the retrieval step (did it find the right source material?) and the generation step (did it use that material correctly?) — and a system can fail at either independently of the other. Evaluating only the final answer’s correctness conflates both failure modes into one score, which tells you something is wrong without telling you where to fix it.
The diagnostic value is in how these metrics combine. Low context relevance with any groundedness score points at retrieval — fix the retriever, embedding model, or chunking strategy. High context relevance paired with low groundedness points at generation — the model is ignoring or misusing good source material, which is a prompting or model-selection problem, not a retrieval one. That pattern is what makes component-level evaluation worth the extra setup over a single end-to-end accuracy check.
These metrics are most useful run continuously against a curated test set as the system changes — a new embedding model, a different chunking strategy, an updated prompt — following the same continuous evaluation pipeline pattern used for LLM applications generally, rather than as a one-time assessment at launch that goes stale as the underlying document corpus and query patterns evolve.
Context relevance and groundedness are the two with the highest diagnostic value for distinguishing retrieval from generation failures — a reasonable starting pair if you need to prioritize, with context sufficiency and attribution added once the basic retrieval-versus-generation split is in place.
Several can be measured with an LLM-as-a-judge approach (scoring groundedness and context relevance against the retrieved content) at scale, though periodic human validation of the automated scores — the same discipline that applies to LLM-as-a-judge generally — is worth building in rather than trusting the automated scores indefinitely.
They’re a more granular, RAG-specific version of groundedness scoring, one of the four measurement approaches covered in our broader piece on LLM hallucination testing. These metrics add the retrieval-versus-generation diagnostic layer that general hallucination testing doesn’t always provide on its own.