Reliable LLM API integrations do more than send a prompt and wait for text. This checklist explains how to handle timeouts, transient errors, rate limits, partial responses, provider outages, fallbacks, and duplicate requests without turning a retry into a repeated side effect. Use it when designing an integration, preparing a release, or investigating production failures.
Overview
An LLM API call can fail in several different ways. The request may never reach the provider, the provider may reject it because of throttling, the connection may time out after processing has started, or the response may arrive incomplete. These cases are not interchangeable, so a single retry rule is rarely safe.
Start by separating failures into three broad groups:
- Transient failures: temporary network errors, service unavailability, or short-lived capacity problems. A bounded retry may help.
- Permanent failures: invalid authentication, malformed requests, unsupported parameters, or a failed schema validation. Fix the request or configuration instead of retrying indefinitely.
- Ambiguous outcomes: a timeout or connection drop where the client cannot tell whether the provider completed the request. These require idempotency and side-effect controls.
Reliability work should cover the full request lifecycle: queueing, provider selection, timeout budgets, retries, response validation, logging, user messaging, and any action triggered by the model. For observability, pair this checklist with the LLM observability guide. If prompts, models, or response schemas change frequently, include these checks in the LLM application CI/CD workflow.
Checklist by scenario
1. Timeouts and slow responses
Define separate limits for connection establishment, request transmission, time to first response, and total request duration where your HTTP client and provider support them. A total timeout should fit inside the user-facing or job-level deadline, leaving time for validation, logging, and a controlled fallback.
- Set a finite timeout for every outbound request; do not rely on an unlimited default.
- Record whether the timeout occurred before a response, during streaming, or after partial output.
- Retry only when the operation is safe to repeat and the remaining deadline allows it.
- For interactive requests, return a clear status or asynchronous job option rather than keeping the connection open without a limit.
- Cancel abandoned work where possible so timed-out requests do not continue consuming application resources.
2. Rate limits and throttling
Rate limits may apply to request count, tokens, concurrency, or other provider-defined dimensions. Treat the response as a scheduling signal, not simply as an error. When available, read the provider's retry guidance and rate-limit headers, but avoid assuming that every provider uses the same names or semantics.
- Use a bounded queue or concurrency limiter before requests reach the provider.
- Honor an explicit retry-after value when it is present and reasonable.
- Use exponential backoff with jitter when no usable delay is supplied.
- Cap both the delay and the total number of attempts.
- Apply backpressure to callers instead of allowing an unlimited in-memory backlog.
- Track throttling by provider, model, tenant, and operation where those dimensions affect capacity.
3. Transient errors and retries
A retry policy should be explicit and operation-specific. A useful starting configuration is:
maxAttempts: 3
initialDelayMs: 250
backoffMultiplier: 2
maxDelayMs: 4000
jitter: full
retryOn: [network_error, timeout_before_response, temporary_unavailable, throttled]
retryBudget: bounded_by_request_deadline
This is a template, not a universal provider setting. Tune it using measured latency, error rates, workload priority, and the provider's documented behavior. Full jitter means selecting a random delay within the current backoff range, which helps prevent many workers from retrying at the same instant.
Do not retry validation errors, authentication failures, clearly invalid parameters, or policy rejections unless the request is changed. If an SDK already retries automatically, determine whether your application is adding a second retry loop. Nested retries can multiply latency and request volume.
4. Partial responses and streaming failures
Streaming introduces an important distinction: receiving some tokens does not prove that the final response is complete or valid. Treat the stream as provisional until the expected completion signal and application-level validation have succeeded.
- Persist partial output separately from the committed final result.
- Mark interrupted streams as incomplete rather than silently displaying them as finished.
- Do not automatically append a second response to the first unless your application has a deliberate continuation design.
- For structured output, validate the assembled result before storing or acting on it.
- Tell users whether they are seeing a draft, an incomplete response, or a successfully completed result.
5. Provider or model fallback
A fallback can improve availability, but it is not a free substitute. Models and providers can differ in context limits, output formats, tool support, latency, safety behavior, and cost. Define fallback eligibility before an incident occurs.
- Choose fallbacks by capability, not only by availability.
- Keep prompts, tool definitions, and structured schemas compatible or adapt them explicitly.
- Label the selected provider and model in logs and, when useful, in the response metadata.
- Run evaluation cases against the primary and fallback paths before enabling automatic switching.
- Decide whether fallback is allowed for every operation, especially actions that change data or contact external systems.
For systems that use tools or take actions, combine fallback rules with permission boundaries and approval steps described in the AI agent architecture patterns guide.
6. Duplicate requests and idempotency
The most dangerous retry is one that repeats a side effect. An LLM response may be informational, but downstream code might create a ticket, send a message, issue a refund, or update a record. A timeout can leave the client uncertain about whether that action already happened.
- Create an idempotency key for each logical operation, not each transport attempt.
- Store the key and operation state in durable storage when the action matters.
- Make the downstream operation reject or replay a completed key safely.
- Separate generation from execution: first produce and validate a proposed action, then execute it through an idempotent service.
- Require human approval for high-impact operations when automatic certainty is not possible.
A practical request state machine might use queued, started, response_received, validated, action_pending, completed, and failed. Store transition reasons so an operator can distinguish a retryable failure from an ambiguous outcome.
What to double-check
| Area | Questions to answer |
|---|---|
| Error classification | Which status codes, SDK exceptions, and network conditions are retryable in this integration? |
| Timeout budget | Can all attempts finish before the user, queue, proxy, or server deadline? |
| Rate limits | Are limits measured by requests, tokens, concurrency, tenant, model, or another dimension? |
| Fallbacks | Does the alternate model support the same schema, tools, context, and quality requirements? |
| Billing | Could a retry or fallback cause additional usage, and is that behavior visible to operators? |
| Security | Are prompts, outputs, keys, and provider error details logged without exposing sensitive data? |
| Validation | Is the response checked for completeness, schema validity, citations, and business constraints before use? |
Test these behaviors deliberately. Inject connection delays, force throttling responses, cut a stream midway, return malformed structured output, and make the primary provider unavailable. Then verify that metrics, traces, alerts, user messages, and state transitions all match the intended design. Evaluate quality and regression risk with a repeatable test set; the LLM evaluation frameworks guide provides a useful companion workflow.
Common mistakes
- Retrying everything: Permanent errors become slower, more expensive failures.
- Using no jitter: A group of workers can synchronize and create another traffic spike.
- Ignoring the overall deadline: Three individually reasonable attempts can still violate the user or job timeout.
- Assuming a timeout means failure: The provider may have completed the request even though the client did not receive the result.
- Falling back without evaluation: A different model may produce output that passes transport checks but fails application requirements.
- Logging only the final error: Without attempt count, timing, model, provider, and request state, diagnosis becomes guesswork.
- Letting model output trigger actions directly: Validate structured output and route side effects through controlled, idempotent application code.
When to revisit
Revisit this checklist before a seasonal planning cycle, a major traffic increase, or a change to prompts, models, SDKs, providers, schemas, or downstream actions. Also review it after an outage, unexpected billing event, repeated throttling, or any incident involving duplicate work.
At minimum, schedule a broader reliability review every six months. Confirm current provider error behavior, rate-limit guidance, timeout recommendations, fallback capabilities, SDK defaults, and billing implications rather than copying assumptions from an older integration. Repeat failure-injection tests after material changes.
For the next implementation or review, take these five actions:
- Write down the retryable and non-retryable failure classes for each operation.
- Add finite timeouts, bounded exponential backoff, jitter, and a total deadline.
- Introduce idempotency keys before enabling retries around side effects.
- Test primary and fallback paths with delayed, throttled, partial, and malformed responses.
- Monitor attempts, latency, token usage, fallback rate, validation failures, and ambiguous outcomes.
Reliable LLM API handling is an operational discipline, not a single SDK option. Treat the model call as one component in a larger workflow, and design every retry, fallback, and timeout around the consequences of what happens next.