How to Build a Production-Ready RAG Application: Architecture, Retrieval, and Evaluation

Overview

RAG combines information retrieval with LLM app development. Instead of asking a language model to answer from its learned parameters alone, the application retrieves relevant passages from a controlled knowledge source and places them in the model’s context. The model then uses that context to produce an answer, ideally with citations or source references.

A typical production architecture has six stages:

1. Ingestion: collect files, web pages, tickets, database records, or other approved sources.
2. Normalization: extract text, preserve useful metadata, remove duplicates, and identify access permissions.
3. Chunking and indexing: divide content into retrievable units, create embeddings, and store them in a search index or vector database.
4. Retrieval: interpret the user’s question, apply filters, search for candidate passages, and optionally rerank them.
5. Generation: provide the selected context to the LLM with instructions about evidence, uncertainty, and output format.
6. Evaluation and operations: measure retrieval quality, answer quality, latency, cost, failures, and user feedback.

Vector search is useful for finding semantically related text, but it should not be treated as a complete search strategy. Keyword or lexical search can be valuable for product names, error codes, legal terms, identifiers, and exact phrases. A hybrid retrieval approach can combine both methods, while metadata filters can restrict results by tenant, department, document type, date, or permission.

Access control belongs in retrieval, not only in the final interface. Every indexed item should carry enough metadata to determine who may retrieve it. The application should apply those rules before context reaches the model. A prompt instruction such as “do not reveal confidential information” is not a substitute for authorization logic.

For implementation teams, the most useful design principle is separation of concerns. Keep ingestion, indexing, retrieval, prompting, model calls, and evaluation independently testable. This makes it easier to change an embedding model, adjust chunking, or replace a vector store without rewriting the entire application. For related production planning, see the developer tooling checklist for shipping an LLM app.

Maintenance cycle

RAG systems are maintained systems, not one-time builds. Documents change, content becomes obsolete, permissions are revised, and model behavior can shift after configuration changes. Establish a review cycle before launch and record the decisions that affect retrieval quality.

1. Review the source pipeline

Confirm that every important source is still being collected and that failed imports are visible. Track document identifiers, source timestamps, versions, and deletion events. If a source document is removed or replaced, the corresponding indexed records should be updated or deleted rather than left searchable.

2. Inspect chunking and metadata

Chunking should preserve enough context to make a passage understandable while remaining focused enough for retrieval. Test headings, tables, lists, code, and long paragraphs separately. Store metadata such as title, URL, section, owner, revision date, language, and access group where relevant. Metadata is often as important as the embedding when users need filtered or cited answers.

3. Recheck retrieval with a fixed test set

Maintain a small, representative collection of questions with expected source documents or passages. Run it after changes to chunk size, embedding configuration, search thresholds, filters, reranking, or query rewriting. Measure whether the expected evidence appears in the retrieved set before judging the generated answer.

4. Review answer behavior

Check whether the application answers from retrieved evidence, distinguishes missing information, cites the right sources, and follows the required output format. A response can sound fluent while relying on the wrong passage. Evaluation should therefore inspect both retrieval and generation. The RAG evaluation guide provides a useful framework for separating these measurements.

5. Monitor operational costs and failures

Record retrieval latency, model latency, token usage, timeout rates, empty-result rates, and failed ingestion jobs. Monitor these values by workflow or tenant where possible. A quality improvement that makes every request substantially slower may not be appropriate for all use cases; a cheaper configuration that increases unsupported answers may create a different operational problem.

Signals that require updates

Some changes justify an immediate review rather than waiting for the next scheduled maintenance window.

• Users report incorrect or outdated answers: inspect the retrieved passages first. The problem may be stale indexing, duplicate content, weak ranking, or an overly broad prompt.
• Answers cite irrelevant documents: examine query interpretation, metadata filters, chunk boundaries, and reranking. Citation formatting alone will not correct poor evidence selection.
• New content types are introduced: PDFs, spreadsheets, scanned documents, source code, and tables often need different extraction and chunking rules.
• Permissions or tenancy change: recheck filter logic, cached results, document inheritance, and index isolation before exposing the updated system.
• The model, embedding model, or search service changes: rerun the evaluation set. Even a seemingly compatible replacement can alter ranking, context use, latency, or output structure.
• Search intent shifts: if users begin asking comparative, procedural, or diagnostic questions instead of simple lookups, review query rewriting, retrieval depth, and response templates.
• Fallback answers increase: determine whether the corpus is incomplete, retrieval thresholds are too strict, or the application is failing to interpret valid queries.

Keep an update log that records the change, reason, affected components, evaluation results, and rollback plan. This turns maintenance into an engineering process rather than a series of unexplained prompt edits.

Common issues

Overly large or overly small chunks

Large chunks may contain several unrelated topics and consume valuable context. Small chunks may lose definitions, headings, or the conditions that make a statement meaningful. Compare multiple strategies using real questions, and preserve structural information such as section titles and parent document names.

Retrieval without authorization

Filtering results only after generation is unsafe because restricted text may already have entered the model context. Apply authorization at query time and test with users who have different access levels. Also review logs and caches so that one user’s retrieved context cannot be reused for another user incorrectly.

Assuming a high similarity score means relevance

Similarity scores are useful signals, not universal truth. A passage can be semantically close but fail to answer the question. Use a reranker, hybrid search, metadata filters, or a query decomposition step when the task requires exact matching or multiple pieces of evidence.

Prompting the model to compensate for weak retrieval

A detailed system prompt cannot reliably recover evidence that was never retrieved. Keep the generation prompt focused: define the task, identify the supplied context, require an explicit uncertainty response when evidence is insufficient, and specify the citation or structured-output format. For robust JSON responses, see the guide to structured output from LLMs.

Evaluating only the final answer

Final-answer review hides the source of failure. Label evaluation cases for retrieval success, evidence sufficiency, factual support, citation correctness, refusal or abstention behavior, and formatting. This makes remediation more precise and helps teams decide whether to change the index, retriever, prompt, or model.

Ignoring human review for consequential workflows

Some applications should route uncertain, sensitive, or high-impact outputs to a person. Define clear approval conditions, show the supporting passages, and record the reviewer decision. A human-in-the-loop design can be especially useful while the evaluation set is still growing; see this guide to approval steps in AI workflows.

When to revisit

Use a two-level maintenance schedule. Perform a lightweight operational review on a regular cycle: check ingestion failures, empty retrievals, latency, access-control errors, user feedback, and a sample of citations. Perform a deeper review on a less frequent scheduled cycle or whenever a major dependency changes. Revisit chunking, embedding and retrieval settings, prompts, model selection, evaluation cases, and cost assumptions together.

An update is also warranted when the application expands to a new department, language, document type, or use case. Those changes can alter the meaning of relevance and may require new metadata, test questions, permissions, and response rules. If users begin searching for procedures, troubleshooting steps, or multi-document comparisons, expand the evaluation set to represent those tasks instead of relying on the original lookup questions.

Before each release, run a small regression checklist:

1. Verify that new and deleted documents are reflected in the index.
2. Test authorized and unauthorized access paths.
3. Run retrieval tests against known expected passages.
4. Review citations and unsupported-answer behavior.
5. Check structured output validation, latency, and failure handling.
6. Record the change and retain a rollback option.

Finally, keep the architecture documentation current. A production-ready RAG application is not defined by a particular model, vector database, or framework. It is defined by repeatable evidence handling, controlled access, measurable quality, and a maintenance process that responds when the system or its users change.