RAG Evaluation Guide: How to Measure Retrieval Quality, Answer Accuracy, and LLM App Reliability

Overview

RAG evaluation is the process of measuring whether an application retrieves useful evidence and uses that evidence correctly when generating an answer. It applies to internal knowledge assistants, document search, support tools, research interfaces, and other LLM applications that combine a model with a changing knowledge base.

The central mistake in many evaluations is treating the generated answer as the only thing worth testing. A response can sound convincing while relying on irrelevant passages, omitting the key fact, misrepresenting a source, or answering a question that the retrieved context does not support. A useful evaluation therefore separates the system into observable stages:

• Query handling: Was the user’s request interpreted correctly?
• Retrieval: Did the search return the passages needed to answer the request?
• Context construction: Were relevant passages selected, ordered, and formatted effectively?
• Generation: Did the model answer using the supplied evidence?
• Presentation: Are citations, uncertainty, and source boundaries clear to the user?
• Operations: Does the application meet expectations for latency, cost, availability, and failure handling?

This decomposition makes RAG evaluation actionable. If an answer is wrong, you can determine whether the problem came from a missing document, weak vector search, poor chunking, an overly aggressive filter, an unclear prompt, or unsupported model behavior. For broader production practices, see the developer tooling checklist for shipping an LLM app.

Template structure

Use a versioned evaluation record for every test case. Store it in a format your team can review manually and run automatically. A practical record can contain the following fields:

{
  "id": "billing-001",
  "question": "What is the refund window for an annual plan?",
  "expected_answer": "The documented refund window is ...",
  "reference_documents": ["billing-policy-v3"],
  "required_facts": ["annual plan", "refund window"],
  "forbidden_claims": ["a refund is guaranteed outside the policy"],
  "expected_citations": ["billing-policy-v3"],
  "category": "policy question",
  "difficulty": "multi-condition",
  "metadata": {
    "language": "en",
    "tenant": "example",
    "created_at": "YYYY-MM-DD"
  }
}

The question should reflect an actual user task rather than an artificial test sentence. The expected answer is a reference for reviewers, not necessarily a string that the model must reproduce exactly. Include required facts when completeness matters and forbidden claims when a plausible but unsupported interpretation would create risk.

For retrieval tests, add document or chunk-level labels. Mark which passages are relevant, partially relevant, irrelevant, or required for a complete answer. This supports common retrieval metrics:

• Recall at K: How often the required evidence appears within the top K retrieved results.
• Precision at K: How much of the retrieved set is relevant.
• Mean reciprocal rank: How highly the first relevant result appears.
• nDCG or graded relevance: How well the ranking places highly useful results ahead of marginal ones.

These metrics answer different questions. High recall with low precision may produce a noisy context. High precision with low recall may omit a document that contains a necessary condition. Evaluate several values of K because the best retrieval depth for a short factual query may not suit a question requiring evidence from multiple documents.

For answer evaluation, record separate judgments for correctness, completeness, groundedness, and citation accuracy. Groundedness asks whether the answer is supported by the supplied context. Citation accuracy asks whether each citation actually supports the claim attached to it. A response can be factually correct by coincidence yet fail groundedness, or cite a relevant document without supporting the specific sentence.

How to customize

Begin with a small, representative test set instead of trying to cover every possible question. Sample queries from support tickets, search logs, domain reviews, and known failure cases. Remove personal or confidential information before adding examples to a shared evaluation repository. Include both ordinary questions and cases designed to expose weaknesses.

A balanced set commonly includes:

• Direct fact lookups with one authoritative source.
• Questions requiring evidence from several documents.
• Ambiguous questions that should trigger clarification.
• Questions about information absent from the knowledge base.
• Conflicting or versioned documents.
• Long, conversational questions with references such as “that policy” or “the earlier option.”
• Requests that should be refused, redirected, or answered with uncertainty.

Define a scoring rubric before reviewing outputs. For example, correctness can use a pass, partial-pass, or fail label; groundedness can require every material claim to be supported; and citation accuracy can require a reviewer to locate the cited evidence quickly. Keep the rubric narrow enough that two reviewers can apply it consistently. If disagreement is frequent, clarify the rubric with examples rather than silently averaging the results.

Pair quality scores with operational measurements. Capture retrieval latency, generation latency, total response time, token usage, error rates, empty-result rates, and the number of retrieved chunks. These measurements help explain why a quality change occurred. For example, increasing retrieval depth may improve recall while increasing context size and response time. Model selection also affects this balance; the guide on choosing a model for an AI app provides a useful framework for comparing speed, context, cost, and accuracy requirements.

Automated evaluation is useful for fast feedback, but it should not replace targeted human review. An evaluator model can classify obvious cases, compare an answer with reference criteria, or flag unsupported claims. Treat its output as a measurement with its own error rate. Periodically sample passing and failing cases for human inspection, especially when the application handles policy, financial, legal, health, security, or other high-consequence information.

Examples

Consider a document assistant that answers questions about product configuration. A retrieval test might label the setup guide as highly relevant, an outdated guide as partially relevant, and a marketing page as irrelevant. If the current guide appears below the outdated one, the retrieval failure is visible even before the model generates an answer.

An answer test for the same system could require three facts: the supported configuration, the prerequisite version, and the applicable limitation. The reviewer then scores whether all three facts appear, whether they are supported by the retrieved passages, and whether each citation points to the correct section. This is more informative than comparing the output with one expected sentence.

For an absent-information case, the test record should state that the correct behavior is to acknowledge that the source set does not contain the answer. The evaluation should penalize an invented value even if the response is fluent. This type of case is especially important when prompts instruct the model to be helpful without defining how it should handle missing evidence. Guidance on reducing hallucinations in production applications can complement this test design.

For regression testing, run the same records against a baseline and a candidate configuration. The candidate might change the embedding model, chunk size, metadata filters, reranker, system prompt, model, or citation format. Compare retrieval and answer results by category rather than relying only on one overall score. A change that improves direct lookups but harms multi-document questions may be acceptable for one product and unsuitable for another.

Use traces to connect each evaluation result to the actual request path: rewritten query, filters, retrieved IDs, chunk text, prompt version, model configuration, tool calls, output, and reviewer judgment. The article on LLM observability tools covers how tracing and evaluation data can work together during diagnosis.

When to update

RAG evaluation is not a one-time acceptance test. Revisit the suite whenever the underlying inputs or workflow change. Important triggers include a new document collection, revised chunking or indexing, changed metadata filters, a new embedding or reranking approach, a model or prompt update, a modified citation format, or a change in user tasks.

Update the test set when production feedback reveals a new failure pattern. Add the original query, a sanitized version of the relevant context, the expected behavior, and the reason the previous system failed. Retain old cases so improvements do not erase evidence of earlier problems. Mark superseded records when a policy or product behavior genuinely changes, and preserve the version of the source material used for the judgment.

A practical maintenance cycle is simple: run automated checks on every meaningful code or configuration change, review a smaller quality sample on a regular schedule, and conduct a broader evaluation before release. Track changes by retrieval configuration, prompt version, model, and knowledge-base snapshot. Set release thresholds for the dimensions that matter most to your application rather than chasing a single universal score.

To put this framework into practice, create 25 to 50 representative records, label the required evidence, define pass criteria, and capture a baseline trace for each case. Then change one component at a time and compare the results. Link failures to a specific remedy—better chunking, improved filters, clearer prompting, a different model, human approval, or a data correction. This turns RAG evaluation from a vague quality exercise into a repeatable developer tool workflow that can keep pace with an evolving LLM application.