Build an LLM evaluation dataset that reflects real user requests, measurable quality standards, safety risks, and regression-prone edge cases. This guide provides a reusable test-case schema, a comparison of evaluation approaches, an annotation rubric, and a practical workflow for running evaluations in development and CI.
Overview
An LLM evaluation dataset is a curated collection of inputs, expected behaviors, and scoring criteria used to test an AI feature consistently. It is more useful than a small set of impressive examples because it makes quality visible across common requests, ambiguous instructions, unsupported questions, unsafe inputs, and known failure modes.
The dataset should test the complete behavior of the feature, not only the prompt in isolation. For an LLM application, that may include system instructions, retrieved context, conversation history, tools, output schemas, model settings, and post-processing. A prompt change can improve one intent while damaging another, so evaluation cases need enough coverage to reveal those trade-offs.
A useful evaluation dataset normally contains four layers:
- Inputs: realistic user requests, documents, conversation turns, or tool conditions.
- Expected behavior: a reference answer, required facts, refusal condition, structure, or action boundary.
- Evaluation criteria: the dimensions on which the output will be judged.
- Metadata: intent, risk level, source, version, language, model, prompt version, and evaluation result.
Do not treat a reference answer as the only acceptable wording. For open-ended responses, define required elements and unacceptable behaviors instead. This makes the test more resilient to valid variations in phrasing.
How to compare options
There is no single best evaluation method for every LLM feature. Compare approaches against the risk and structure of the task rather than choosing the most sophisticated-looking tool. A spreadsheet may be sufficient for an early prompt experiment, while a high-volume production feature may need versioned datasets, automated scoring, and CI thresholds.
Compare by test coverage
First ask whether an approach can represent the cases that matter. A simple exact-match check works well for a strict classification label or a JSON field, but it is weak for a support answer where several responses could be correct. Human review can capture nuance, but reviewers need clear instructions and enough time to produce consistent labels.
Compare by repeatability
A test should produce comparable results when the prompt, model, or retrieval configuration changes. Deterministic checks are easiest to repeat. Rubric-based review is more flexible but should record the rubric, reviewer, and decision. If an automated judge is used, retain its evaluation prompt and model configuration so changes to the judge do not silently alter historical results.
Compare by cost and speed
Use a small smoke set for every pull request and a broader evaluation set on a scheduled or release workflow. This separates fast regression detection from deeper analysis. Keep the most important cases in the fast suite, especially cases tied to safety, schema validity, contractual claims, or previously observed failures.
Compare by diagnostic value
A score alone does not explain what broke. Prefer systems that preserve the input, output, criterion-level result, evaluator rationale, and relevant version identifiers. When a result fails, developers should be able to determine whether the cause was instruction following, retrieval quality, formatting, hallucination, refusal behavior, or an upstream data change.
For a practical starting point, compare four approaches:
| Approach | Best for | Limitation |
|---|---|---|
| Exact or rule-based checks | Labels, schemas, required fields, citations, and forbidden strings | May reject valid natural-language variations |
| Reference comparison | Short answers with a stable expected result | Can confuse different wording with incorrect content |
| Human review | Nuanced quality, tone, policy interpretation, and ambiguous cases | Slower and vulnerable to inconsistent interpretation |
| Rubric-based model evaluation | Large-scale screening of open-ended responses | Requires calibration and should not be treated as unquestionable ground truth |
Feature-by-feature breakdown
1. Create a balanced test-case taxonomy
Organize cases by behavior and risk, not only by department or feature name. A useful taxonomy includes:
- Core intents: common requests the feature is designed to handle.
- Instruction following: format, length, tone, ordering, and constraint adherence.
- Factuality: correct use of provided information and appropriate handling of missing information.
- Grounding and retrieval: answers supported by retrieved context, including cases where the context is incomplete or contradictory.
- Safety and privacy: requests that should be refused, narrowed, escalated, or answered without exposing sensitive data.
- Adversarial inputs: prompt injection, conflicting instructions, malformed content, and attempts to override system boundaries.
- Edge cases: empty input, very long input, unusual characters, misspellings, multiple intents, unsupported languages, and ambiguous wording.
- Regression cases: failures previously found in testing, monitoring, review, or customer feedback.
Include both positive and negative examples. A dataset containing only normal requests cannot tell you whether the application fails safely when it lacks evidence or encounters conflicting instructions.
2. Use a durable dataset schema
Store each case in JSON, a database table, or another version-controlled format. The exact implementation can vary, but each record should answer what was tested, what should happen, and how the result was judged.
{
"case_id": "support_014",
"category": "grounding",
"risk": "medium",
"input": "Can I change the billing date on this plan?",
"context": ["Approved billing policy excerpt"],
"expected_behavior": "Answer only from the supplied policy and state when the policy does not specify an option.",
"required_elements": ["uses supplied policy", "does not invent a date or fee"],
"failure_conditions": ["unsupported claim", "confident answer without evidence"],
"evaluation_method": "rubric",
"source": "production_review",
"dataset_version": "2026-10-01"
}
Keep sensitive production data out of the dataset unless it has been properly handled for your environment. Synthetic cases can expand coverage, but label them clearly and validate that they represent realistic user behavior.
3. Write an annotation guide
Annotators need operational definitions, not broad instructions such as “rate the answer quality.” Define what counts as a pass, partial pass, and fail for every criterion. For example, factuality may pass when every externally checkable claim is supported by the supplied context; it may fail when the response invents a policy detail. A partial result may apply when the answer is directionally correct but omits a required qualification.
Separate dimensions that are often confused. A response can be factually correct but fail the requested JSON format. It can follow the format but contain unsupported claims. Record these as separate results so prompt changes can be diagnosed accurately.
4. Score with gates and dimensions
A practical rubric can use a small scale, such as 0 for fail, 1 for partial, and 2 for pass. Score dimensions independently:
- Accuracy: are the answer's claims correct and supported?
- Instruction following: did it satisfy explicit constraints?
- Completeness: did it include the required information?
- Safety: did it avoid prohibited disclosures or unsafe assistance?
- Format: is the output valid for downstream parsing or display?
Do not rely only on an average score. Define release gates for critical failures. For example, a build may require valid structured output for every schema test and zero failures on designated safety cases, while allowing a small number of partial results on subjective tone cases. The correct thresholds depend on the feature's risk and business requirements.
5. Connect the dataset to regression testing
Run a baseline before changing the prompt, model, retrieval settings, or tool instructions. Then run the same dataset after the change and compare results by category. Store the prompt version, model identifier, retrieval configuration, temperature or equivalent settings, and dataset version with each run.
A useful CI workflow is:
- Validate that every dataset record has required fields.
- Run a small smoke set on each relevant code change.
- Check deterministic requirements such as JSON validity, required fields, and forbidden content.
- Run rubric-based or human review for cases that need semantic judgment.
- Compare category-level results with the approved baseline.
- Fail the workflow when a critical gate is breached, then retain the outputs for diagnosis.
For retrieval-augmented applications, evaluate retrieval and generation separately where possible. A correct answer may be impossible if the relevant document was not retrieved. The RAG evaluation guide and RAG tutorial provide useful context for separating these concerns.
Best fit by scenario
Early prompt development: begin with a small, hand-curated set covering core intents, obvious failures, and output constraints. A versioned JSON or spreadsheet can be enough if every case has a clear expected behavior.
Structured extraction or classification: emphasize exact checks, schema validation, field-level accuracy, and difficult boundary cases. Include missing fields, conflicting values, and malformed documents rather than testing only clean examples.
RAG and document workflows: add context quality, citation or evidence requirements, irrelevant retrieval, stale documents, and “answer not supported” cases. For document automation, test extraction, classification, summarization, and routing as separate stages as well as an end-to-end flow. See the document automation workflow guide.
AI agents and tool use: test whether the agent chooses the correct tool, passes valid arguments, respects authorization boundaries, handles tool errors, and stops when the task is complete. Include requests that attempt to make the agent ignore its operating constraints. The AI agent development guide covers related design patterns.
High-risk or customer-facing features: combine automated checks with qualified human review. Give safety and privacy cases explicit release gates, and maintain a clear escalation path for ambiguous outputs. A security-focused checklist can complement the dataset: LLM application security checklist.
When to revisit
An evaluation dataset is a living test asset. Revisit it whenever the feature, its inputs, or its operating constraints change. The most important triggers are a new model, revised system prompt, changed retrieval index, new tool, modified output schema, new supported intent, or a policy and privacy requirement update. Also review it after a production incident, a repeated user complaint, or a meaningful shift in the types of requests being received.
Use a simple maintenance cycle:
- Review failed and borderline cases from the latest evaluation run.
- Promote confirmed failures to permanent regression cases.
- Remove or revise cases whose expected behavior is no longer valid.
- Check whether every major intent and risk category still has coverage.
- Re-run the baseline after changing the rubric, evaluator, or dataset schema.
- Record the reason for each change and publish the new dataset version with the application change.
Keep a compact smoke suite, a broader release suite, and a growing archive of historical failures. This prevents every pull request from becoming unnecessarily slow while preserving lessons from earlier regressions. For prompt-specific measurement, compare this process with the prompt evaluation framework and review model and token trade-offs in the LLM cost optimization guide.
The immediate next step is to select ten to twenty real or carefully anonymized cases, label them across the taxonomy, define pass criteria, and run a baseline. Expand from failures rather than trying to predict every possible request at once. A small, versioned dataset with clear judgments is a stronger foundation for prompt engineering than a large collection of unexamined examples.