Evaluating an LLM application is more than checking whether a few answers look good. This guide provides a repeatable framework for measuring accuracy, safety, cost, latency, and regression risk before and after changing a model, prompt, retrieval pipeline, or tool workflow.
Overview
An LLM evaluation framework is a set of test cases, metrics, review methods, and release rules used to determine whether an AI application is performing acceptably. It applies to chat assistants, RAG systems, document automation workflows, classifiers, summarizers, and tool-using agents.
The central principle is to evaluate the application as a system rather than judging the language model in isolation. A model may produce fluent answers while the application still fails because it retrieves the wrong document, exposes sensitive content, calls a tool incorrectly, exceeds its token budget, or gives inconsistent results after a prompt change.
A practical evaluation process usually combines four layers:
- Offline tests: Repeatable test cases run against a fixed dataset before deployment.
- Human review: Subject-matter experts assess quality dimensions that are difficult to reduce to a single score.
- Model-graded evaluation: A separate evaluator checks outputs against explicit criteria, with human spot checks to detect grading errors.
- Production monitoring: Real usage data reveals failures, cost increases, latency changes, and emerging edge cases.
Use the same evaluation set when comparing candidate models, prompt versions, retrieval settings, or orchestration changes. This makes differences easier to attribute and turns AI regression testing into an engineering practice rather than an informal demonstration.
For reliable structured outputs, pair evaluation with a schema and validation workflow such as structured prompt templates with JSON schema validation. If the application uses external knowledge, also test retrieval separately from generation by following a RAG evaluation workflow.
How to estimate
Start by converting the application’s goals into measurable evaluation dimensions. Avoid treating “quality” as one universal number. A support assistant, extraction pipeline, and autonomous agent need different checks.
| Dimension | What to measure | Useful evaluation method |
|---|---|---|
| Accuracy | Whether the response or extracted value is correct | Reference answers, exact-match checks, field-level comparison, or expert review |
| Grounding | Whether claims are supported by approved context | Citation checks, retrieval inspection, and claim-level review |
| Safety | Whether the system avoids prohibited actions, leakage, or unsafe tool use | Adversarial cases, policy-specific checks, and human escalation review |
| Consistency | Whether similar inputs produce acceptable results | Repeated runs, paraphrased prompts, and structured output validation |
| Cost | Estimated spend per request or workflow | Input and output token counts multiplied by the applicable rate assumptions |
| Latency | Time to first response and time to completion | Application traces and percentile-based monitoring |
For each dimension, define a target, a warning threshold, and a release threshold. For example, a team might require every mandatory extraction field to pass validation, allow a small number of non-critical formatting failures, and block release when a safety case fails. The exact thresholds depend on the application’s consequences and should be documented rather than inferred from a single impressive demo.
A simple comparison score can help organize decisions, but it should not hide critical failures. One possible approach is:
Weighted quality score = (accuracy × weight) + (grounding × weight) + (consistency × weight) − penalty for critical failures
Keep safety and data-handling failures as release blockers where appropriate instead of allowing strong scores in other categories to offset them. For cost, estimate:
Estimated run cost = (input tokens × input rate) + (output tokens × output rate) + retrieval, tool, or infrastructure costs
Use current provider or infrastructure rates as configurable inputs, not hard-coded article assumptions. The accompanying LLM cost optimization guide can help separate token budgets, caching, routing, and model-selection decisions.
Inputs and assumptions
A useful evaluation dataset represents the work the application is expected to perform. It should not contain only easy examples or questions selected because the current system answers them well. Include normal requests, ambiguous requests, boundary cases, malformed inputs, long context, missing information, and adversarial attempts relevant to the product.
Each test case should have a stable identifier and enough metadata to explain the result. A reusable template might look like this:
{
"case_id": "support-014",
"category": "missing-information",
"input": "Can I change the delivery address?",
"context_ids": ["policy-202"],
"expected_behavior": "Ask for the order identifier and avoid promising a change",
"must_include": ["request order identifier"],
"must_not_include": ["claim that the address was changed"],
"severity": "medium",
"evaluation_method": "human plus rule checks"
}
Record the assumptions that affect comparisons:
- The model and version, system prompt, user prompt, and tool definitions.
- Sampling settings, maximum output length, timeout, and retry behavior.
- The retrieval index, chunking rules, filters, reranking settings, and context limit.
- The dataset version, evaluator instructions, and scoring rubric.
- Input and output token counts, latency measurements, and failure counts.
Separate component tests from end-to-end tests. A retrieval test can measure whether relevant passages are returned, while an answer test checks whether the model uses them correctly. An agent test should inspect tool selection, arguments, permissions, and final responses. This separation makes debugging faster than assigning every failure to the model.
Version prompts and datasets alongside application code. The prompt version control guide describes a practical way to review and roll back changes. For security-focused cases, include injection and leakage scenarios from an LLM application security checklist.
Worked examples
Comparing two prompt versions
Suppose a support assistant has 120 labeled cases. Run the same cases against the current prompt and the candidate prompt using the same model, retrieval configuration, and evaluation instructions. Compare correct answers, unsupported claims, refusal behavior, structured-output failures, average token use, and latency.
Do not approve the candidate because its average score is higher if it introduces a critical failure in a high-severity category. Instead, review the changed cases and classify each difference: improvement, harmless variation, regression, or evaluator disagreement. Store the run results so the comparison can be repeated after another change.
Estimating a model change
For a candidate model, begin with a representative subset and then run the full regression set if the subset is acceptable. Estimate monthly usage using explicit assumptions:
Monthly requests × average input tokens and monthly requests × average output tokens.
Apply the relevant current rates outside the test data, then add known retrieval, tool, and hosting costs. Report a range when traffic or response length is uncertain. A lower per-request estimate is not automatically better if accuracy, safety, or retry frequency declines.
Evaluating a retrieval change
When changing chunk size, indexing, filters, or reranking, maintain separate retrieval and answer metrics. Inspect whether the expected source appears in the returned context, whether irrelevant passages crowd it out, and whether the final answer makes claims not supported by the retrieved material. This prevents a generation change from being blamed for a retrieval regression.
When to recalculate
Re-run the evaluation suite before releasing any change to the model, prompt, system instructions, retrieval pipeline, tool schema, safety rules, context strategy, or output parser. Also recalculate cost and latency assumptions when usage patterns, token budgets, provider rates, routing rules, or infrastructure change.
Review production monitoring on a regular schedule and after notable incidents. Add anonymized failure cases to the regression dataset only after checking privacy and access requirements. A growing dataset is useful when it remains labeled, traceable, and balanced; adding every unreviewed output can make the suite noisy and difficult to interpret.
Use this release checklist:
- Freeze the model, prompt, dataset, evaluator, and configuration versions.
- Run component and end-to-end tests against the baseline and candidate.
- Review critical safety, grounding, and tool-use failures manually.
- Compare quality, token usage, latency, retries, and failure rates.
- Record assumptions, unresolved disagreements, and the release decision.
- Monitor the changed workflow after deployment and schedule a follow-up review.
Evaluation is most valuable when it becomes part of the development loop: define expected behavior, run repeatable tests, inspect meaningful failures, and update the dataset when the product encounters a new case. That workflow gives developers a defensible way to compare LLM application changes without relying on demos, vague impressions, or a single benchmark score. For a broader evaluation workflow, see the guide to building an LLM evaluation dataset and the practical prompt evaluation framework.