A reliable retrieval-augmented generation (RAG) application is not created by adding a vector database to a chatbot and stopping there. This RAG tutorial explains how to structure ingestion, chunking, embeddings, retrieval, prompting, citations, evaluation, and monitoring—and what to review monthly or quarterly as your documents, models, and user questions change.
Overview
RAG combines information retrieval with LLM generation. Instead of asking a model to answer entirely from its trained knowledge, the application first searches a trusted document collection, then places relevant passages into the model’s context. The model uses those passages to produce an answer, ideally with citations or links back to the source material.
A maintainable RAG system usually contains six stages:
- Ingestion: collect documents and preserve useful metadata.
- Parsing: extract readable text, headings, tables, and other relevant structure.
- Chunking: divide content into passages that can be retrieved and understood independently.
- Indexing: create embeddings and store them in a vector database or hybrid search index.
- Retrieval and generation: find relevant passages, assemble a controlled prompt, and generate a response.
- Evaluation and monitoring: measure retrieval quality, answer quality, latency, cost, and failure patterns.
The central design principle is separation of responsibilities. Retrieval should locate evidence; the language model should interpret and present it. If the retrieved context is incomplete or irrelevant, a more elaborate prompt will not reliably correct the underlying problem. For a broader architecture checklist, see How to Build a Production-Ready RAG Application.
What to track
1. Document coverage and freshness
Track which sources are indexed, when each source was last processed, and whether ingestion completed successfully. Store identifiers such as document title, URL, department, version, publication date, and access scope alongside each chunk. These fields make filtering, citations, and troubleshooting easier.
Watch for documents that changed without triggering re-indexing. A RAG application can appear healthy while answering from an outdated copy. Define a freshness expectation for each collection—for example, near-real-time for operational records or a scheduled refresh for stable technical documentation. The right interval depends on how quickly the underlying information changes.
2. Chunking and metadata quality
Review whether chunks retain enough context to stand on their own. A chunk that contains a procedure without its heading, prerequisites, or exception may be semantically similar to a question but still produce an incomplete answer. Useful approaches include preserving heading paths, keeping paragraphs together, and using overlap only where it prevents important context from being split.
Track average chunk length, unusually small or large chunks, missing metadata, and parsing failures. Tables, scanned PDFs, code samples, and nested web pages often require different handling from ordinary prose. Treat these as ingestion quality issues rather than problems that the model should solve at generation time.
3. Retrieval quality
Measure whether the relevant source appears in the retrieved results. Useful checks include recall at a selected result count, ranking position of the expected passage, duplicate results, and the proportion of queries that return no useful evidence. If users regularly reformulate questions before receiving a good answer, retrieval may be too literal, too broad, or missing useful query expansion.
Compare vector search with keyword or hybrid search where terminology matters. Product names, error codes, policy identifiers, and version strings can be difficult for semantic search alone. Apply metadata filters for tenant, region, product version, permissions, or document type before the final context is assembled.
4. Answer quality and citations
Track whether answers are supported by retrieved evidence, whether citations point to the correct source, and whether the response clearly states when the available context is insufficient. A useful answer should not merely include a citation; the cited passage should actually support the relevant claim.
Create a small evaluation set containing common questions, difficult questions, questions with no answer in the corpus, and questions that could be confused by similarly named concepts. Review both helpfulness and unsupported claims. The RAG Evaluation Guide provides a focused framework for retrieval quality, answer accuracy, and reliability.
5. Operational signals
Monitor latency by stage: query embedding, retrieval, reranking if used, model generation, and post-processing. Also track token usage, error rates, timeouts, empty retrieval results, malformed citations, and user feedback. Keep prompts, model settings, index versions, and document collection versions identifiable in logs so a later regression can be reproduced.
Cadence and checkpoints
Use a review cadence that matches the system’s risk and rate of change. A monthly checkpoint is often practical for active internal knowledge applications; a quarterly review may be sufficient for stable documentation. High-impact systems may need automated checks on every index or prompt change rather than relying on a calendar.
At each ingestion run
- Confirm that expected sources were discovered and processed.
- Compare document counts and changed-file counts with the previous run.
- Inspect parsing failures, empty documents, and missing metadata.
- Verify that deleted or superseded content is no longer retrievable where appropriate.
Monthly review
- Run the fixed evaluation set and compare results with the previous baseline.
- Sample production questions, including unanswered and low-rated queries.
- Review retrieval misses, citation errors, and repeated user reformulations.
- Check latency, token consumption, service errors, and unusual traffic patterns.
- Confirm that access filters are applied consistently across retrieval paths.
Quarterly review
- Reassess chunking and metadata rules against newly added document types.
- Compare embedding, retrieval, reranking, and model configurations using the same test set.
- Remove obsolete indexes, duplicate documents, and unused processing steps.
- Review whether the corpus still matches the application’s intended scope.
- Update runbooks and document the reason for each material change.
Keep a changelog for index versions, ingestion code, prompts, model choices, and evaluation results. This makes it possible to distinguish a retrieval regression from a source-content change or a generation change.
How to interpret changes
When answer quality falls, start by separating retrieval failures from generation failures. Inspect the retrieved passages for a sample of affected queries. If the correct evidence is absent, investigate parsing, chunking, metadata filters, query construction, and index freshness. If the evidence is present but the answer ignores it, inspect context ordering, prompt instructions, answer length, citation handling, and model behavior.
A rise in “no answer” responses is not automatically negative. If the corpus truly lacks the requested information, a controlled refusal is preferable to an unsupported answer. Classify these cases into genuine knowledge gaps, retrieval misses, ambiguous questions, permission-filtered results, and expected out-of-scope requests.
Changes in the document collection can also alter evaluation results without any code change. Record corpus versions when comparing scores. Likewise, a lower latency number may come from retrieving less context, which could reduce answer quality. Interpret operational metrics together rather than optimizing one variable in isolation.
Use targeted experiments instead of changing several components at once. For example, compare two chunking strategies while holding the embedding model, retrieval count, prompt, and evaluation set constant. This produces a more useful signal than making simultaneous changes to the index and generation layer.
When to revisit
Revisit this RAG implementation on a monthly or quarterly schedule, and immediately after any event that can change its behavior. The most important triggers are:
- A major change to the document source, format, ownership, or update frequency.
- A new embedding model, language model, reranker, prompt, or vector database configuration.
- A noticeable shift in user questions, domains, languages, or response expectations.
- Repeated citation mistakes, unsupported answers, retrieval misses, or user reformulations.
- Changes to permissions, tenant boundaries, retention requirements, or document scope.
- Unexpected increases in latency, failures, token use, or infrastructure load.
For the next review, choose a small set of actions: sample recent questions, run the fixed evaluation set, inspect the worst retrieval misses, verify source freshness, and compare results with the last recorded baseline. Then make one controlled change, rerun the checks, and record what improved or regressed.
A RAG application becomes more dependable through this repeatable cycle: keep the corpus current, preserve document structure, test retrieval separately from generation, verify citations, and monitor the system after release. When the application grows beyond a simple question-answering flow, connect these checks to your broader developer tooling checklist for shipping an LLM app and maintain the evaluation set as a living engineering asset.
