When a retrieval-augmented generation (RAG) application produces irrelevant, incomplete, or weakly cited answers, changing the language model is rarely the best first step. This troubleshooting checklist helps you isolate whether the failure begins in ingestion, chunking, metadata, embeddings, vector search, reranking, context assembly, or answer generation—and gives you a repeatable way to test each stage.
Overview
A RAG response is the result of several dependent stages. Documents are parsed and split, chunks receive metadata and embeddings, a search system retrieves candidates, an optional reranker orders them, and the application passes selected context to the model. Citations are then created from the retrieved evidence and presented with the answer. A defect at any stage can look like a prompt or model problem.
Start by treating retrieval quality as a measurable pipeline rather than a single vector-database setting. For each test question, record the expected source, the expected passage, the returned chunks, their scores, the final context sent to the model, and the citations shown to the user. This separates “the system found the wrong evidence” from “the model failed to use the right evidence.”
Maintain a small evaluation set containing common questions, ambiguous questions, questions requiring multiple documents, and questions for which the correct response is “not enough information.” Include the expected document identifiers and, where practical, the relevant passage or facts. Re-run this set after changing a parser, chunking rule, embedding model, metadata filter, search parameters, reranker, or prompt.
For broader testing practices, see the LLM evaluation frameworks guide and the walkthrough on building an LLM evaluation dataset.
Checklist by scenario
When results are irrelevant
- Run the query directly against the index and inspect the top results before invoking the model.
- Check that the query and document embeddings were created with compatible dimensions and the intended model configuration.
- Confirm that the query is not being altered by an overly aggressive rewrite, translation, truncation, or chat-history merge.
- Compare semantic search with a keyword or hybrid search path for exact names, product codes, error messages, and technical identifiers.
- Inspect filters. A missing, incorrect, or stale tenant, permissions, language, product, or date filter can remove the relevant document or admit unrelated content.
- Test several top-k values. If the right passage appears only at a large k, retrieval may need reranking or better chunking rather than a different generation prompt.
When results are relevant but incomplete
- Check whether a single concept has been split across chunk boundaries. Preserve headings, list items, table rows, and nearby definitions where their meaning depends on context.
- Inspect whether the context budget is filled with redundant chunks, leaving no room for distinct evidence.
- Test parent-document or adjacent-chunk expansion for questions that require a definition followed by an exception, procedure, or example.
- Verify that pagination, deduplication, and result limits are not silently discarding evidence.
- Ask whether the question requires multiple retrieval passes. A comparison or multi-part request may need separate subqueries and a final synthesis step.
When search looks good but the answer is wrong
- Log the exact context supplied to the model, not only the chunks returned by the database.
- Check for context ordering problems. Relevant evidence may be buried between long or contradictory passages.
- Use a prompt that distinguishes instructions from retrieved text and requires the answer to stay within the supplied evidence.
- Require the model to associate each factual claim with a source identifier or passage reference.
- Test whether the model is answering from conversation history or general knowledge instead of the retrieved material.
RAG applications that expose tools or external actions also need isolation between retrieved text and executable instructions. The guidance in Prompt Injection Defense for LLM Apps is useful when documents may contain hostile or misleading instructions.
When citations are missing or poorly matched
- Give every chunk a stable document ID, source URL or path, title, section, and location marker during ingestion.
- Carry those identifiers through retrieval, reranking, context assembly, and response formatting.
- Check that a citation points to the passage supporting the claim, not merely to a document that happened to rank highly.
- Separate answer generation from citation formatting when necessary: first identify supported claims, then render the references.
- Test refusal behavior for unsupported questions. A system that cites unrelated material instead of acknowledging missing evidence has a grounding problem.
What to double-check
Ingestion and chunking
Inspect the text before it reaches the embedding service. Confirm that OCR, HTML, PDF, and table extraction preserve the words users search for. Remove navigation, repeated headers, and unrelated boilerplate when they dominate chunks. Use boundaries that reflect document structure rather than relying only on a fixed character count. Store the original text alongside normalized text so a citation can reproduce the source accurately.
There is no universally correct chunk size. Evaluate several sizes and overlap settings against your own test cases. Smaller chunks can improve precise matching but may lose context; larger chunks can preserve context while diluting the relevant signal and consuming more context space. Measure the trade-off instead of selecting a value by convention.
Embeddings and index configuration
Record the embedding model, preprocessing steps, vector dimension, distance metric, index settings, and ingestion timestamp for every index version. Re-embedding only part of a corpus can create inconsistent search behavior. After changing models, rebuild or clearly version the affected index and rerun the same retrieval benchmark.
Search, filters, and reranking
Log query text, filters, top-k, returned IDs, scores, and latency. Scores from different models or indexes are not automatically comparable, so use them for diagnosis within a consistent configuration. Reranking can help when initial retrieval returns a broad candidate set, but it cannot recover a passage that was never retrieved. If a candidate is absent, investigate query formulation, chunking, embeddings, filters, or hybrid search first.
Context assembly and prompting
Inspect the final prompt as an artifact. Look for truncation, duplicated chunks, contradictory versions, lost metadata, and instructions that conflict with the evidence policy. Define what the model should do when sources disagree or do not answer the question. Structured output and validation can make citation fields easier to verify; the structured-prompt templates guide covers this pattern.
Common mistakes
- Changing the model first: A stronger model cannot reliably answer from evidence that was not retrieved or was truncated.
- Evaluating only final answers: Store intermediate retrieval and context data so you can identify the failing stage.
- Using one test query: A single successful demo can hide failures involving synonyms, exact terms, permissions, long documents, and unknown answers.
- Optimizing recall without controlling noise: Increasing top-k may surface the right passage but also crowd out useful context with duplicates and distractors.
- Ignoring metadata: Source type, version, access scope, language, date, and section often matter as much as semantic similarity.
- Rebuilding without versioning: Keep index, parser, embedding, and prompt versions with benchmark results so improvements and regressions can be explained.
- Treating citations as decoration: A citation should support a specific claim and be traceable to the retrieved source location.
Keep operational constraints visible as well. More retrieval passes, larger contexts, and reranking can affect latency and token use. The LLM cost optimization guide provides a framework for reviewing those trade-offs without sacrificing the quality checks above.
When to revisit
Revisit this checklist before seasonal planning cycles, major content refreshes, and production releases that change the corpus or workflow. Run the benchmark whenever you alter parsing, chunking, metadata fields, embedding models, vector indexes, search filters, reranking, context limits, prompts, or citation formatting. Also review it when users report a new failure pattern, such as outdated answers, permission leaks, missing sections, or citations that do not support claims.
Keep a compact regression report with retrieval results, answer quality, citation accuracy, latency, and estimated usage for each version. When a failure appears, reproduce it with the smallest possible test case, identify the first stage where expected evidence is lost, change one variable, and rerun the full set. This workflow is slower than guessing once, but it produces a RAG system that can be maintained as documents, tools, and workflows change.
Action plan: choose 20–50 representative questions, label the expected sources, log every pipeline stage, and establish a baseline before changing settings. Then fix the earliest failing stage—ingestion, chunking, retrieval, ranking, context assembly, or generation—rather than masking the symptom with a larger model or a longer prompt.