Reliable LLM applications need more than an uptime check. This guide shows how to monitor prompts, model versions, token usage, latency, errors, outputs, and user feedback so your team can diagnose failures, control costs, and detect quality regressions without exposing unnecessary sensitive data.
Overview
LLM observability is the practice of collecting enough operational and quality context to understand what happened during a model request. Traditional application monitoring can tell you that an endpoint returned an error or exceeded a latency threshold. An LLM tracing system should go further: it should connect the incoming request, prompt version, retrieved context, model configuration, tool calls, output validation, user feedback, and final response.
This context is particularly important because an apparently successful request can still be a product failure. A model may return a valid HTTP response with incomplete reasoning, unsupported claims, malformed JSON, irrelevant retrieval results, or an unexpectedly expensive token pattern. Conversely, a slow request may be caused by retrieval, a tool call, queueing, provider response time, or application-side processing rather than the model alone.
A useful observability design separates three questions:
- What happened? Capture traces, events, configuration, errors, and timing.
- How often is it happening? Aggregate metrics by route, model, prompt version, tenant, region, and feature.
- Was the result useful and safe? Combine automated evaluations, validation outcomes, and user feedback.
Start with a small, consistent schema rather than collecting every possible field. The goal is to make an individual request debuggable while keeping dashboards useful and data handling defensible.
What to track
The following categories form a practical baseline for monitoring LLM applications. Store a correlation ID on every request so application logs, model calls, retrieval events, and tool executions can be joined later.
| Area | Useful signals | Why it matters |
|---|---|---|
| Request logs | Request ID, route, tenant or workspace identifier, feature name, status, timestamp | Establishes scope and connects the request to surrounding application activity. |
| LLM traces | Prompt version, model identifier, provider, parameters, retry count, tool calls, parent span | Shows the sequence of operations that produced the response. |
| Token metrics | Input tokens, output tokens, cached or retrieved context size where available | Reveals context growth, inefficient prompts, and unexpected usage changes. |
| Latency | End-to-end duration, queue time, retrieval time, tool time, model time, timeout count | Separates a slow user experience from a slow model call. |
| Errors | Timeouts, rate limits, provider failures, validation errors, parsing failures, fallback use | Identifies reliability patterns that a single error rate can hide. |
| Quality signals | Schema-validation result, citation checks, groundedness checks, evaluation scores, escalation rate | Tracks whether successful requests are producing acceptable results. |
| User feedback | Rating, correction, retry, copy action, abandonment, support escalation | Provides evidence about usefulness from the real product workflow. |
Track latency as percentiles rather than only averages. Averages can remain stable while a smaller group of requests becomes unacceptably slow. Review at least a central percentile and a high-tail percentile appropriate to your traffic volume, then segment the results by workflow and model. A chat response, batch summarizer, retrieval pipeline, and autonomous tool call usually need different expectations.
For token monitoring, record counts returned by the provider or tokenizer used by your application, but do not assume that every provider exposes identical fields. Keep the raw provider response separate from your normalized metrics so changes in telemetry can be investigated without rewriting historical dashboards.
Tracing should include meaningful spans such as request.received, retrieval.search, llm.generate, tool.execute, and output.validate. A span should explain one operation and include duration, status, and a small set of relevant attributes. Avoid placing full prompts, retrieved documents, or user messages in every production trace by default.
Use redaction, masking, sampling, access controls, and retention limits for sensitive data. In many systems, storing prompt and response hashes, truncated previews, or references to a protected data store is safer than placing unrestricted content in general-purpose logs. Observability must help with debugging without becoming an accidental copy of the application's private data.
Cadence and checkpoints
LLM monitoring works best as a recurring operating routine, not a dashboard created only during an incident. Use different checkpoints for immediate operations, regular review, and planned changes.
Per request and during incidents
Capture a correlation ID, request status, duration, model call status, token counts, validation outcome, and fallback path. Alert on conditions that require action, such as sustained provider errors, repeated timeouts, or a sharp rise in invalid structured output. Keep alerts tied to user impact; a low-level event is not automatically an incident.
Weekly or biweekly review
Review request volume, latency percentiles, error categories, token distribution, retry frequency, and quality checks by feature. Look for changes concentrated in one prompt version, model, provider, customer segment, or retrieval configuration. Sample traces from both successful and unsuccessful requests. A dashboard that shows only failures can miss gradual quality deterioration.
Monthly or quarterly checkpoint
Compare current behavior with a stable baseline and review whether the tracked fields still answer the team's debugging questions. Remove unused metrics, add fields required by new workflows, verify redaction rules, and check that retention and access settings match the application's risk profile. Revisit alert thresholds after traffic, prompt, model, or architecture changes.
Before a release or provider change
Confirm that traces preserve the same correlation structure across environments. Run representative evaluation cases and compare latency, token usage, validation failures, and quality outcomes. Verify that dashboards distinguish the old and new prompt or model versions. The related LLM App CI/CD deployment checklist can help connect these checks to release work.
How to interpret changes
When a metric moves, avoid treating the first visible correlation as the root cause. Begin with a time window and compare the affected requests with a stable period. Then segment by model, prompt version, endpoint, provider, input size, retrieval path, and tool usage.
For a slow request, follow the trace in order:
- Check whether the delay began before the model call, such as in queueing or authentication.
- Compare retrieval and tool-call durations with model generation time.
- Check input and output token counts for context expansion or unusually long responses.
- Review retries, timeouts, rate-limit events, and fallback behavior.
- Compare the affected model, region, and prompt version with unaffected requests.
For a bad response, inspect the same request from input through output. Confirm that the intended prompt version was used, retrieved context was relevant, tool results were complete, and the output passed schema or citation validation. If the workflow uses retrieval, consult a RAG retrieval quality troubleshooting guide rather than assuming the generator caused the issue.
Common patterns provide useful clues:
- Tokens rise while latency and quality remain stable: inspect context growth, conversation memory, and duplicated instructions before usage becomes a larger cost or performance issue.
- Latency rises only for tool-enabled requests: examine tool timeouts, sequential calls, and oversized tool results.
- Valid responses fall after a prompt change: compare prompt versions, output schemas, parser behavior, and evaluation cases.
- Provider errors rise with retries: measure the user-visible effect and review backoff, fallback, and idempotency behavior.
- Negative feedback rises without an error increase: inspect quality signals, retrieval results, and representative traces; application health alone may not reveal a relevance problem.
Keep operational monitoring and quality evaluation connected but distinct. A low error rate does not prove that answers are accurate, and a high evaluation score does not guarantee acceptable latency. The LLM evaluation frameworks guide provides a useful companion for formalizing regression checks.
When to revisit
Revisit this observability design at least monthly or quarterly, and immediately after a meaningful change to the application. The review should cover new model providers, prompt templates, output schemas, retrieval indexes, agent tools, memory strategies, traffic patterns, and data-handling requirements.
Use this practical checklist:
- Can an engineer trace one user request across the application, retrieval layer, model calls, tools, and validation?
- Can the team distinguish model latency from queue, retrieval, tool, and network latency?
- Are token and error metrics normalized without hiding provider-specific details?
- Can dashboards compare prompt and model versions before and after a release?
- Are quality signals based on automated checks, sampled review, and user feedback rather than one metric?
- Are sensitive prompts, responses, documents, and tool arguments redacted or access-controlled?
- Do alerts identify actionable user impact instead of generating noise?
Record the review date, baseline period, major changes, open questions, and the next checkpoint. If you are adding agent capabilities, review permissions and approval paths alongside traces using the AI agent architecture patterns guide. If token usage is the main concern, connect observability findings to a token and cost optimization workflow.
The strongest LLM observability systems are not the ones with the most fields. They are the ones that let a team move from “the response was wrong” or “the request was slow” to a specific, testable explanation, while preserving privacy and keeping the monitoring model current as the application evolves.