Reliable AI agent development starts with disciplined orchestration, not simply giving a language model access to more tools. This guide provides a reusable architecture for planning, tool schemas, memory, permissions, retries, human approval, observability, and failure handling so you can build tool-using agents that are easier to test, operate, and adapt as frameworks change.
Overview
An AI agent is an LLM application that can interpret a goal, choose among available actions, call tools, inspect results, and continue or stop based on the outcome. A tool might retrieve a document, query a database, create a ticket, send a message, or run a calculation. The model proposes the next action, but your application remains responsible for validation, authorization, execution, and state management.
This distinction is central to dependable LLM application architecture. Treat the model as a decision-making component inside a controlled workflow rather than as an unrestricted program. The application should define what the agent can do, what information it can see, which actions require approval, and how errors are recorded.
A useful baseline architecture has six layers:
- Request and context: The user goal, identity, conversation state, and relevant data.
- Policy and routing: Rules that determine whether the request is allowed and which agent or workflow should handle it.
- Planning and reasoning: The model’s selection of a next step, expressed through a structured response or tool call.
- Tool gateway: A controlled interface that validates arguments, checks permissions, and invokes external systems.
- State and memory: The information needed for the current task, plus carefully selected durable preferences or records.
- Observability and evaluation: Logs, traces, outcomes, human feedback, and regression tests used to improve the system.
For retrieval-heavy systems, keep the agent layer separate from retrieval quality. A RAG workflow can provide evidence to an agent, but it does not remove the need to evaluate grounding, citations, and answer quality. See this RAG tutorial for a separate architecture and retrieval perspective.
Template structure
Use the following template when designing a new tool-calling agent. It works whether the implementation uses a full agent framework, a lightweight orchestration loop, or direct model API calls.
1. Define the job and the stop condition
Write the agent’s purpose in one sentence and specify what a successful result looks like. For example: “Classify an incoming support request, look up the relevant account record, and draft a response for review.” Then define explicit stop conditions, such as a completed draft, a missing required input, a failed authorization check, or a request for human approval.
Without a stop condition, an agent can continue making low-value calls or repeat an action after receiving an ambiguous result. Limit the number of model-tool cycles and make the limit visible in logs.
2. Describe tools as narrow contracts
Each tool should have one clear purpose, a precise name, a description that explains when it should be used, and a strict input schema. Prefer an operation such as get_order_status over a general-purpose manage_orders function. Narrow contracts make tool selection easier to inspect and reduce the impact of malformed arguments.
Validate every argument outside the model. Check types, required fields, allowed values, length limits, ownership, and whether the requested operation is appropriate for the current user. Tool output should also use a predictable structure, including an explicit status and an error category when execution fails.
3. Separate planning from execution
The model may propose an action, but your application should decide whether to execute it. A useful orchestration loop is:
- Receive the current state and available actions.
- Ask the model for either a final response or one structured tool call.
- Validate the tool call against the schema and policy.
- Request approval when the action has meaningful external consequences.
- Execute the tool in a controlled environment.
- Record the result and return it to the model or finish the task.
Allowing one action at a time is often easier to audit than asking the model to produce an opaque multi-step plan. Parallel calls can be useful for independent read-only operations, but they require careful handling of partial failures and ordering.
4. Make memory explicit
Distinguish short-term task state from durable memory. Short-term state may include the current tool result, user request, and intermediate decisions. Durable memory might include a confirmed preference or an approved project setting. Do not store every conversation detail by default. Define what can be saved, for how long, who can access it, and how it can be corrected or removed.
Retrieved context should also be labeled by source and purpose. This helps the agent distinguish instructions from reference material and gives operators a way to investigate incorrect decisions.
5. Add permissions and approval gates
Use least privilege for tools and credentials. Separate read operations from write operations, and treat actions such as sending messages, changing records, deleting data, or initiating transactions as higher risk. A human approval step should show the proposed action, the important arguments, the expected effect, and the evidence behind it.
Approval is not a substitute for validation. The system should still enforce authorization and input checks after approval, because state may have changed between the review and execution.
6. Design failure handling before implementation
Classify failures rather than returning one generic error. Common categories include invalid arguments, authentication or authorization failure, unavailable service, timeout, rate limitation, missing data, and ambiguous user intent. Retry only failures that are plausibly temporary, and use a bounded retry count with increasing delays where appropriate.
For write operations, idempotency matters. A retry should not create duplicate tickets, messages, or records. Use an operation identifier or another application-level mechanism to recognize repeated requests. If a tool partially succeeds, preserve the result and route the task to a recovery path instead of blindly repeating it.
How to customize
Start with the smallest workflow that solves a real task. A deterministic sequence with one or two tool calls may be more reliable than a general autonomous agent. Introduce additional planning freedom only when a fixed workflow cannot handle the expected variation.
Customize the design along four dimensions:
- Risk: Low-risk drafting can use automatic execution, while external side effects should require stronger checks or approval.
- Uncertainty: If the agent often lacks required information, add clarification states instead of encouraging guesses.
- Latency and cost: Use smaller or faster model steps for classification and routing, reserving more capable reasoning for decisions that need it. Measure the effect rather than assuming a model choice is better.
- Auditability: Preserve the request, selected tool, validated arguments, tool result, policy decision, and final outcome. Avoid storing sensitive content unnecessarily.
Prompts should describe responsibilities and boundaries, not attempt to replace application controls. Include the agent’s objective, available tools, required output format, rules for uncertainty, and conditions for escalation. Structured prompting examples can help make these instructions testable; the related prompt evaluation framework offers a way to turn them into repeatable evaluations.
For complex systems, separate agents by responsibility rather than creating one agent with unrestricted access. A router can select a specialist, a retrieval component can gather evidence, and an action agent can prepare a change for approval. This form of LLM orchestration makes permissions and evaluation boundaries clearer.
Examples
Support triage agent
The agent receives a support message and can call search_knowledge_base, get_customer_context, and draft_reply. It may read relevant records automatically, but it cannot change account data or send the reply. If the knowledge search returns weak or conflicting evidence, the agent marks the case for review instead of presenting a confident answer.
Useful evaluation cases include missing account identifiers, contradictory documentation, requests outside the user’s permissions, and a tool timeout after a successful lookup.
Document extraction workflow
The agent identifies a document type, extracts named fields, and sends the result through deterministic validation rules. A low-confidence or invalid field is returned for review with its source location. The agent should not silently invent a missing value. This pattern combines LLM flexibility with explicit validation and is often preferable to asking an agent to make an irreversible decision from unverified extraction.
The guide on document extraction workflows with LLMs provides a useful adjacent pattern for combining extraction and validation.
Internal operations assistant
An operations assistant can look up deployment information, summarize alerts, and create a proposed incident note. Read-only tools may run automatically, while changes to infrastructure require a clearly displayed plan and approval. The tool gateway should prevent the agent from constructing arbitrary commands when a narrow, parameterized operation is available.
Across all three examples, the important pattern is the same: the model proposes, the application validates, the tool executes within limits, and the outcome becomes observable.
When to update
Revisit an agent design whenever its inputs, tools, models, or operating environment change. In practice, this means reviewing the system when you add a new tool, alter a schema, change permissions, introduce a new data source, modify a prompt, switch models, or change the surrounding publishing or automation workflow.
Also update the design after meaningful failures. Look for repeated invalid arguments, unnecessary tool calls, unsupported claims, duplicate writes, approval bypass attempts, long-running loops, and cases where users cannot understand why an action was proposed. Convert each recurring failure into one of three improvements: a clearer contract, a deterministic application rule, or a new evaluation case.
Maintain a small regression set containing normal requests, ambiguous requests, permission violations, tool errors, stale data, and high-impact actions. Run it after prompt, model, schema, or orchestration changes. Track not only whether the final answer looks correct, but also whether the agent selected the right tool, used valid arguments, respected limits, and stopped at the right time. For broader observability practices, see the guide to LLM observability tools and prompt analytics.
To put this template into practice, choose one narrowly defined task, document its stop conditions, create the smallest possible tool set, and implement validation before adding autonomy. Then log every decision, test failure paths deliberately, and require approval for consequential actions. This sequence gives you a dependable foundation that can evolve with new AI development tools and agent frameworks without making the system’s safety or behavior dependent on any single vendor.
