A reliable prompt is not the one that produces an impressive answer once; it is the one that performs consistently across representative inputs, models, and edge cases. This prompt evaluation framework gives you a reusable way to build test sets, define scoring criteria, compare prompt versions, identify regressions, and document decisions as your LLM application changes.
Overview
Prompt engineering becomes substantially more useful when it is treated as an evaluation workflow rather than a sequence of ad hoc edits. Changing a role description, adding an example, tightening an output format, or switching models can improve one response while creating failures elsewhere. A prompt testing framework makes those trade-offs visible.
The basic loop is straightforward:
- Collect representative inputs from the task your application performs.
- Define what a good response must contain and what it must avoid.
- Run the same test set against a named prompt version and model configuration.
- Score the outputs using consistent criteria.
- Compare results with a previous version before promoting the change.
This approach applies to classification, extraction, summarization, question answering, coding assistance, and agent workflows. It is also useful in RAG systems, where prompt quality is only one part of overall reliability. For retrieval-specific measurements, see the RAG Evaluation Guide.
Evaluation does not require a complicated platform. A spreadsheet can be enough for an early workflow, while a larger LLM application may use an evaluation dataset, automated checks, human review, and observability tools. The important part is consistency: every change should be tested against the same known expectations unless the test set itself is being deliberately revised.
Template structure
Use the following record as the smallest useful unit in a prompt evaluation set. Store one record for each test case, and keep prompt versions separate from the test data.
Test case ID: support-014
Task: Classify and summarize a support request
Input: [representative user message]
Expected behavior: [what the response should do]
Required facts or labels: [known answer elements]
Disallowed behavior: [fabrication, missing fields, unsafe action, etc.]
Prompt version: v1.3
Model configuration: [model, temperature, tools, retrieval settings]
Output: [captured response]
Scores: [criterion scores]
Reviewer notes: [specific evidence]
Decision: pass / revise / investigateFor the test set itself, include more than easy examples. A practical collection normally contains:
- Typical cases: common inputs that represent the main workflow.
- Boundary cases: short, long, incomplete, ambiguous, or unusually formatted inputs.
- Negative cases: requests that should be rejected, escalated, or marked as unsupported.
- Adversarial cases: instructions inside user content that could conflict with the task.
- Regression cases: previously failed examples that must remain fixed.
Next, define a rubric before reviewing outputs. Keep each criterion narrow enough that two reviewers can apply it in the same way. Common criteria include:
- Task correctness: Does the output reach the expected conclusion or classification?
- Completeness: Does it include all required fields, facts, or steps?
- Grounding: Does it stay within the supplied source material when evidence is required?
- Format compliance: Does it follow the required schema, labels, or structure?
- Instruction adherence: Does it follow the task rules without being distracted by irrelevant content?
- Safety and escalation: Does it avoid unsupported actions and identify cases requiring review?
Use a small, explicit scale. For example, score each criterion from 0 to 2: 0 means failed, 1 means partially acceptable, and 2 means fully acceptable. Add a pass rule, such as “format compliance must be 2 and task correctness must be at least 1.” This is more actionable than relying on one overall impression.
For structured outputs, validate the response programmatically where possible. A JSON parser, schema validator, required-field check, or regular-expression check can identify failures more consistently than visual review. The article on structured output from LLMs provides useful patterns for JSON Schema, function calling, and validation.
How to customize
Start by customizing the evaluation to the consequence of failure. A prompt that drafts internal titles may tolerate minor wording differences. A prompt that extracts invoice fields, routes a support case, or triggers an automated action needs stricter checks and clearer escalation rules. Do not give every criterion equal importance if one error is materially more serious than another.
Define the unit of evaluation first. Some tasks are evaluated per response; others require checking individual fields, sentences, retrieved passages, or tool calls. For extraction, a response can be mostly correct while one missing identifier makes the record unusable. Field-level scoring will reveal that problem more clearly than a single response score.
Separate deterministic checks from subjective review. Deterministic checks are appropriate for valid JSON, required keys, allowed labels, character limits, numerical ranges, and exact identifiers. Human review is better suited to tone, usefulness, nuance, and whether a summary preserves the important meaning. An LLM-based judge can help with scale, but treat its score as another signal to calibrate against human-reviewed examples, not as unquestionable ground truth.
Record the complete configuration for every run. The prompt alone may not explain a change in results. Capture the model identifier, system and user messages, temperature or equivalent sampling settings, available tools, retrieval context, output schema, and any preprocessing or postprocessing. This is especially important when comparing models or integrating prompt evaluation into LLM orchestration.
Use separate data for development and final checking when the workflow is large enough to justify it. A development set helps you iterate; a holdout set gives you a less familiar check before release. If you repeatedly tune a prompt against the same examples, it can become optimized for those examples without improving the broader task.
Set thresholds and review rules before comparing versions. A useful policy might say that a new prompt can ship only if it improves the primary score, does not reduce format compliance, and introduces no new critical failures. If scores conflict, require a human review of the affected cases instead of averaging away an important regression.
For production systems, connect evaluation results to application traces and user feedback. Observability makes it easier to determine whether a failure came from the prompt, retrieval, tool selection, input transformation, or the model itself. See LLM Observability Tools Compared for a broader view of tracing, evals, and prompt analytics.
Examples
Example 1: Document extraction
Suppose a prompt extracts a purchase order into structured JSON. The test set should include ordinary documents, missing fields, conflicting values, scanned-text artifacts, and documents with multiple dates. Score each expected field separately, then check whether the final JSON is valid and whether uncertain fields are left blank or flagged according to your application rule. A response that invents a value should fail the grounding or uncertainty criterion even if the rest of the record is correct. The guide to building a document extraction workflow with LLMs can help connect these checks to validation rules.
Example 2: Support classification and summary
For a support workflow, create cases for billing, technical faults, account access, unclear requests, and requests outside the supported scope. The expected output might contain a category, urgency level, short summary, and recommended next step. Score category accuracy, summary fidelity, required-field completeness, and escalation behavior separately. Include messages containing quoted text or copied email threads so the model must distinguish the customer's request from surrounding content.
Example 3: RAG question answering
For a retrieval-augmented application, evaluate both the evidence and the answer. A test case can specify the expected source passage, whether the question is answerable from the supplied context, and the required citation or refusal behavior. Score retrieval relevance, answer correctness, and unsupported-claim avoidance as distinct dimensions. This prevents a fluent answer from receiving a high score when it used the wrong evidence.
Example 4: Comparing prompt versions
Imagine prompt v1 asks for a free-form answer, while v2 adds a structured output example and an explicit uncertainty field. Run both versions against the same cases. Review not only average scores but also individual failures: Did v2 improve schema compliance while making answers too terse? Did it reduce hallucinated fields but increase unnecessary escalations? Document the reason for choosing the version, not just the winning score.
When to update
Revisit the evaluation set whenever the task, inputs, model configuration, or success criteria change. A prompt may need reevaluation after a model migration, a new system instruction, a changed output schema, a retrieval pipeline update, a new tool, or a change in downstream validation. Product changes matter too: if users now need concise answers instead of detailed explanations, the rubric should reflect that requirement.
Add a regression case whenever a real failure is discovered. Preserve the original input, relevant context, expected behavior, output, and diagnosis. Avoid silently replacing an old case with a revised one; keeping the history shows whether the prompt actually fixed the problem and helps explain score changes over time.
Review the rubric when reviewers repeatedly disagree. Disagreement often means a criterion is too broad, the examples are insufficient, or the pass threshold is unclear. Rewrite the criterion with observable evidence and add one or two examples of passing and failing outputs. If the publishing or deployment workflow changes, update the owner, approval step, and record-keeping process as well.
To put this framework into practice, start with 20 to 50 carefully chosen cases rather than waiting for a perfect dataset. Label the expected behavior, define three to six criteria, and run your current prompt as the baseline. Save the results, revise one meaningful aspect of the prompt, and rerun the same cases. Investigate every critical failure, add confirmed failures to the regression set, and record the decision. Once the process is stable, automate deterministic checks and connect the results to your normal developer workflow using the developer tooling checklist for shipping an LLM app. The framework then becomes a living part of prompt engineering: small enough to use during iteration, and structured enough to support dependable LLM application development.
