LLM costs are shaped by more than a model’s published token rate. This guide gives you a repeatable way to estimate spend, set token budgets, compare model-routing options, evaluate prompt caching, and decide when an optimization is worth the engineering effort.
Overview
LLM cost optimization is the process of reducing the cost of each useful outcome, rather than simply choosing the lowest-cost model. A cheaper model that produces unreliable answers may increase review time, retries, support tickets, or downstream processing. Likewise, an expensive model may be appropriate for a small number of high-risk decisions but unnecessary for routine classification or extraction.
A practical cost review should therefore track four related measures:
- API spend: the direct input, output, tool, or platform charges associated with a request.
- Usage volume: requests per day or month, including retries and background jobs.
- Quality-adjusted cost: spend divided by successful outputs that meet your acceptance criteria.
- Operational cost: infrastructure, observability, storage, human review, and engineering maintenance.
Start with a baseline before changing prompts or providers. Record the model, input and output token counts, request volume, latency, error rate, retry rate, and quality score for representative workloads. This baseline lets you distinguish a genuine improvement from a shift in traffic or a reduction in answer quality.
For related implementation concerns, see the prompt evaluation framework before treating a cheaper response as an equivalent one. If your application uses retrieval, the RAG evaluation guide can help separate retrieval problems from generation costs.
How to estimate
The basic monthly estimate is straightforward:
Monthly LLM spend = request volume × average cost per request
For token-priced APIs, calculate each request as:
Cost per request = (input tokens × input rate) + (output tokens × output rate)
Use the provider’s current pricing units in your worksheet. Some services express rates per token, while others publish rates per thousand or per million tokens. Convert all rates to the same unit before multiplying. Keep input and output rates separate because their values may differ.
A more useful planning worksheet includes these fields:
| Input | What to record | Why it matters |
|---|---|---|
| Monthly requests | Expected production requests, retries, and scheduled jobs | Traffic is often the largest cost multiplier |
| Input tokens | System instructions, user content, retrieved context, and tool results | Long prompts can dominate routine workloads |
| Output tokens | Typical and maximum completion length | Unbounded answers create avoidable variance |
| Model rates | Current input and output rates for each candidate model | Pricing changes can invalidate old comparisons |
| Retry rate | Requests repeated because of errors or failed validation | Nominal request cost understates real spend |
| Quality threshold | Minimum score or acceptance rule for a successful result | Prevents optimizing for cheap but unusable output |
For scenario planning, calculate at least three cases: expected volume, a high-volume month, and a stress case that includes elevated retries or longer contexts. This is more informative than a single point estimate.
Inputs and assumptions
Token budgets
Set separate budgets for instructions, retrieved context, user content, and generated output. A token budget should be tied to the task, not an arbitrary global limit. For example, a structured extraction task may need a short output, while a technical explanation may require more room. Ask the model for a defined format and reject or truncate unnecessary verbosity where appropriate.
Measure actual token distributions rather than relying only on averages. The typical request can hide a small group of very long conversations, documents, or tool traces. Track median, upper-percentile, and maximum values so that capacity planning includes outliers.
Prompt and context design
Remove repeated instructions, unused examples, obsolete conversation history, and irrelevant retrieved passages. In a RAG application, retrieval quality affects cost twice: poor retrieval can add unnecessary tokens, and it can trigger follow-up calls or retries. A focused context window is usually easier to evaluate than a large collection of loosely related documents.
Prompt engineering should optimize clarity and reliability before compression. Shortening a prompt that contains essential constraints can increase failures. Test reduced prompts against a fixed evaluation set and compare both quality and token use.
Prompt caching
Prompt caching can be worth evaluating when the same long prefix is sent repeatedly, such as stable system instructions, policy text, schemas, or reference material. Its value depends on the provider’s caching rules, eligible token ranges, cache lifetime, read and write treatment, and workload repetition. These details change, so record them as assumptions rather than embedding them permanently in application logic.
Use this decision sequence:
- Is a meaningful portion of the prompt repeated across requests?
- Does the provider support caching for that request pattern?
- Is the cache likely to remain warm long enough to produce hits?
- Can the prompt be arranged so stable content appears before changing content?
- Does the expected saving exceed implementation, monitoring, and invalidation costs?
Model routing
Model routing assigns different tasks to different models. A smaller or faster model may handle intent detection, summarization, classification, or routine extraction. A more capable model can be reserved for ambiguous cases, complex reasoning, long-form synthesis, or low-confidence escalations.
Routing should be based on measurable signals: task type, document length, confidence, validation results, or user tier. Avoid routing solely by intuition. A useful comparison table includes:
| Optimization lever | Potential benefit | Main risk | Measure |
|---|---|---|---|
| Shorter prompts | Lower input usage | Missing constraints or context | Tokens per request and evaluation score |
| Output limits | Lower completion usage | Incomplete answers | Output tokens and completion rate |
| Prompt caching | Lower repeated-prefix cost or latency | Low cache-hit rate or stale content | Hit rate and effective cost |
| Model routing | Lower average model cost | Quality drift and routing errors | Cost by route and quality by task |
| Batch processing | Potentially better economics for non-urgent work | Delayed results and operational complexity | Cost per item and completion time |
Worked examples
Consider a document workflow that processes 20,000 items each month. Each item sends 2,000 input tokens and produces 500 output tokens. Do not insert a provider rate into this example until you have checked the current pricing page. Instead, define:
- I: current input price per token
- O: current output price per token
- R: retry multiplier, such as 1 plus the observed retry fraction
The monthly token estimate is:
Input tokens: 20,000 × 2,000 = 40,000,000
Output tokens: 20,000 × 500 = 10,000,000
The estimated API cost is therefore:
((40,000,000 × I) + (10,000,000 × O)) × R
Now test an optimization: reduce retrieved context so average input falls from 2,000 to 1,400 tokens, while output remains unchanged. Input usage becomes 28,000,000 tokens, a reduction of 12,000,000 input tokens per month. The saving is:
12,000,000 × I × R
This change is worthwhile only if evaluation confirms that extraction accuracy and escalation rates remain within their targets.
For routing, suppose 70% of requests can use a lower-cost model and 30% require the primary model. Calculate the weighted average cost:
Average request cost = (0.70 × lower-cost route) + (0.30 × primary route)
Then add the cost of routing logic, validation calls, fallback requests, and human review. A route that looks attractive at the model level may be less effective if it frequently triggers a second call.
When to recalculate
Recalculate your LLM cost model whenever a pricing input changes, traffic grows materially, or the workload changes shape. In practice, schedule a monthly review and an additional review before a major launch or model migration.
Refresh the worksheet when:
- a provider changes input, output, caching, batch, or context-limit terms;
- request volume, document size, or conversation length changes;
- you introduce RAG, tool calls, agents, or multi-step orchestration;
- retry, fallback, or human-review rates move;
- a new model passes your quality and reliability evaluation;
- latency requirements make batching or routing impractical.
End each review with three decisions: what to keep, what to test, and what to stop. Keep a versioned record of rates, token distributions, quality results, and assumptions. That record makes model selection more defensible and prevents old estimates from silently guiding new deployments.
For systems that call tools or take actions, pair cost reviews with the LLM application security checklist. For multi-step workflows, review the AI agent development guide so that cost controls do not weaken validation or safety boundaries. The goal is not the lowest token bill; it is a predictable cost per reliable outcome.