LLM Application Security Checklist: Prompt Injection, Data Leakage, and Tool Risks
Before deploying an LLM application, use this practical security checklist to identify prompt injection paths, protect sensitive data, constrain tool access, and establish a review cycle that keeps pace with new features, models, and integrations.
Overview
LLM application security is broader than protecting an API key or filtering a chat message. A production system may combine user prompts, system instructions, retrieved documents, conversation history, external tools, application secrets, and business data. Each connection creates another place where untrusted content can influence behavior or where sensitive information can be exposed.
The central security question is not simply, “Can the model produce an unsafe answer?” It is, “What can an attacker influence, what can the application reveal, and what actions can the system take?” This framing is useful for chatbots, RAG applications, document workflows, and secure AI agents alike.
Use the checklist below as a lightweight threat-model worksheet. Record the data entering each component, the trust level of that data, the actions available to the model, and the controls that prevent unintended outcomes. For implementation context, compare this process with the AI agent development guide and the RAG tutorial.
A simple threat-model worksheet
| Area | Questions to answer | Evidence to retain |
|---|---|---|
| Inputs | Which inputs are user-controlled, retrieved, uploaded, or generated? | Input categories and trust boundaries |
| Data | What sensitive data can enter prompts, context, logs, or outputs? | Data classification and retention rules |
| Actions | Which tools, APIs, or workflows can the model invoke? | Tool inventory and permission scopes |
| Controls | Where are validation, authorization, and human approvals enforced? | Test cases, logs, and control owners |
| Recovery | How will the team detect, contain, and investigate an incident? | Alerts, rollback steps, and incident contacts |
What to track
1. Prompt injection exposure
Prompt injection occurs when instructions hidden in a user message, document, web page, email, or other input influence the model to disregard the application’s intended task. Direct attacks are visible in a user prompt. Indirect attacks are embedded in content the application retrieves or processes, making them especially important in RAG and document automation workflows.
Track whether untrusted text is clearly separated from system instructions, whether retrieved content is treated as data rather than authority, and whether the model can be persuaded to reveal hidden instructions or take an unauthorized action. Add adversarial test cases such as requests to ignore previous instructions, expose confidential context, follow commands found in a document, or alter the expected output format.
Prompt engineering can improve reliability, but a prompt is not an authorization boundary. Enforce permissions in application code and service layers rather than relying on wording alone. The prompt evaluation framework can help organize repeatable tests for instruction-following behavior.
2. Data leakage paths
Map sensitive data across the complete request lifecycle: ingestion, retrieval, prompt construction, model calls, tool responses, logs, traces, caches, analytics systems, and user-visible output. Include information that may be copied into a prompt automatically, such as conversation history, document metadata, internal URLs, account identifiers, or content from another tenant.
Useful checks include:
- Confirm that retrieval filters enforce the requesting user’s or tenant’s permissions.
- Minimize the context sent to the model instead of forwarding entire documents by default.
- Redact or tokenize sensitive fields when the task does not require their raw values.
- Review whether logs contain prompts, completions, tool arguments, or uploaded files.
- Test whether one user can cause the application to return another user’s data.
- Define retention and deletion behavior for conversations, files, embeddings, and traces.
For document-heavy systems, pair extraction and summarization with validation rules rather than treating the model’s output as an authoritative copy of the source. See the document extraction workflow guide for a related implementation pattern.
3. Tool and agent risks
Every tool expands the application’s attack surface. A model that can search, send messages, update records, execute code, or trigger an automation can turn a misleading instruction into a real-world side effect. Track each tool’s purpose, input schema, credentials, allowed operations, and failure behavior.
Use least privilege: provide a tool only the permissions it needs, restrict destinations where practical, and separate read operations from write operations. Validate tool arguments against a strict schema, reject unexpected fields, and perform authorization checks after the model proposes an action but before the action executes. High-impact operations should require explicit user confirmation or a human approval step.
Do not expose secrets to the model merely because a connected tool needs them. Keep credentials in the application or a dedicated secret-management layer, and pass only the minimum result back into the model. Add rate limits, timeouts, idempotency controls, and audit logs for tools that can change external state.
4. Output and downstream handling
Model output can be unsafe even when the prompt was legitimate. Treat generated text, JSON, code, URLs, and structured fields as untrusted until validated. Use typed schemas, length limits, allowlists, and business-rule checks before writing output to a database, displaying it as markup, or passing it to another service.
For applications that rely on structured responses, test malformed JSON, omitted fields, unexpected types, prompt content inserted into fields, and outputs that exceed operational limits. A parser should fail safely and visibly rather than silently accepting partial data.
Cadence and checkpoints
Security review should be part of the LLM application development workflow, not a one-time launch task. A practical cadence is a lightweight monthly review and a deeper quarterly review, with an additional assessment whenever the system’s trust boundaries change.
Monthly review
- Review failed validation events, unusual tool calls, permission errors, and output-blocking events.
- Run a small regression set covering prompt injection, data access, refusal behavior, and structured output.
- Check new prompts, tools, data sources, and integrations added since the previous review.
- Confirm that logs and traces do not contain newly introduced sensitive fields.
- Verify that disabled tools, revoked credentials, and retired data sources are no longer reachable.
Quarterly review
- Repeat the full threat model for changed architecture, model providers, retrieval pipelines, and agent plans.
- Review tool permissions, service accounts, network access, rate limits, and approval gates.
- Refresh abuse cases using realistic documents, user roles, and business workflows.
- Reassess data retention, tenant isolation, incident response, and recovery procedures.
- Compare evaluation results over time rather than judging a model or prompt from a single example.
Run an immediate review after a major model or platform release, a new tool integration, a change to retrieved data, a new user population, or any security incident. A change that appears to improve quality can still alter instruction-following, context use, or tool-selection behavior.
How to interpret changes
Not every increase in blocked requests indicates a worsening security posture. A higher count may reflect better detection, increased traffic, a new test suite, or a change in user behavior. Interpret security signals alongside exposure and impact.
Track four dimensions for each finding:
- Frequency: How often does the behavior occur in tests or production?
- Reach: Which users, tenants, data sources, or tools could be affected?
- Impact: Would the result expose information, corrupt data, trigger an external action, or merely produce an incorrect answer?
- Control strength: Is prevention enforced in code, or does it depend on model behavior and prompt wording?
Prioritize issues that combine sensitive data with broad access or consequential tools. A prompt injection that produces an incorrect draft is different from one that can retrieve private records or send an external message. Likewise, a low-frequency data leak deserves urgent attention when the affected information is highly sensitive.
Maintain a small controls matrix with columns for threat, affected component, preventive control, detection signal, owner, test case, and review date. This makes it easier to distinguish a documented risk from an untested assumption. For RAG systems, combine security checks with retrieval-quality and answer-quality evaluation; the RAG evaluation guide provides a useful complementary framework.
When to revisit
Use this checklist before launch and revisit it on a monthly or quarterly cadence. Do not wait for the scheduled review when the application gains a new data source, tool, model, agent loop, plugin, automation, or user role. Also reopen the review after changes to authentication, retrieval permissions, logging, deployment boundaries, or prompt templates.
For the next review, create a short change record containing:
- What changed since the previous assessment
- Which trust boundaries or data flows changed
- Which tools and permissions were added or removed
- Which injection, leakage, and authorization tests were rerun
- Which findings remain open and who owns them
Then perform a focused pre-launch check: submit adversarial inputs, test cross-user access, inspect tool arguments, verify structured-output validation, review logs for sensitive content, and confirm that high-impact actions require the intended approval. Keep the results with the application’s normal engineering records so security remains part of everyday developer tooling rather than a separate, occasional exercise.
As the application evolves, use the production-ready RAG architecture guide and the AI developer tools comparison to evaluate supporting workflows, but apply the same principle throughout: untrusted model output should never be the sole authority for access, secrets, or consequential actions.
