AI agents are compelling in demos. A language model that can use tools, retrieve information, take actions, and chain multiple steps together appears to solve problems that weren’t solvable before. Then you try to ship one and discover that the demo success rate was 80%, which means 1 in 5 operations fail silently.
Production agents require the same engineering discipline as any other distributed system. The failure modes are different, but the principles — observability, error handling, testing, graceful degradation — are the same ones you apply everywhere else.
What an Agent Actually Is
Strip away the marketing: an agent is a loop. A language model is called, it decides what to do next (call a tool, ask for more information, produce a final answer), the result is fed back in, and the loop continues until the model decides it’s done.
┌─────────────────────────────────────────┐
│ Agent Loop │
│ │
│ System prompt + history + tools │
│ ↓ │
│ LLM call │
│ ↓ │
│ ┌────────────────────────┐ │
│ │ Tool call? → Execute │ │
│ │ Final answer? → Done │ │
│ │ Request input? → Wait │ │
│ └────────────────────────┘ │
│ ↓ │
│ Append result to history │
│ ↓ (loop) │
└─────────────────────────────────────────┘
This loop is where the problems live. Each iteration costs money, takes time, and can go wrong in ways that compound. An agent that runs 20 iterations on a task that should take 5 has not just consumed 4× the budget — it has also wandered far from where it should be.
Tools: Design Them Like APIs
Tools are the mechanism by which an agent interacts with the world. They’re functions with a name, description, and JSON schema for parameters. The LLM decides which tools to call and with what arguments.
The quality of your tools determines the quality of your agent. Design them with the same care you’d apply to a public API.
Make tools atomic and focused. A tool that does five things is harder for the LLM to use correctly than five tools that each do one thing. search_and_analyze_and_summarize_documents is not a good tool. search_documents, get_document, summarize_text are.
Write precise, honest descriptions. The description is the only documentation the LLM has. Vague descriptions produce incorrect tool calls. Specific descriptions including failure modes produce more reliable behavior:
@tool(
description="""Search the product catalog by keyword.
Returns up to 10 matching products with their IDs, names, and prices.
Use when the user asks about specific products or wants to find items.
Do NOT use for inventory checks — use check_inventory instead.
Returns empty list if no products match."""
)
def search_products(query: str) -> list[Product]:
...
Return errors explicitly. Don’t raise exceptions from tools — return structured error information. The agent can reason about errors in the result; it can’t reason about exceptions.
def check_order_status(order_id: str) -> dict:
try:
order = order_service.get_order(order_id)
return {"status": "found", "order": order.to_dict()}
except OrderNotFound:
return {"status": "not_found", "message": f"No order with ID {order_id}"}
except Exception as e:
return {"status": "error", "message": "Service temporarily unavailable"}
Validate inputs. The LLM will occasionally call your tools with invalid inputs. Validate and return clear errors rather than letting exceptions propagate:
def send_email(to: str, subject: str, body: str) -> dict:
if not is_valid_email(to):
return {"status": "error", "message": f"Invalid email address: {to}"}
if len(body) > 10000:
return {"status": "error", "message": "Email body too long (max 10000 chars)"}
...
Planning and Task Decomposition
Long multi-step tasks benefit from explicit planning. Rather than letting the agent figure out each step reactively, you can prompt it to produce a plan first and then execute it step by step.
This provides two benefits: you can inspect the plan before execution (human-in-the-loop), and the agent is less likely to get lost mid-task because it has an explicit goal structure to follow.
PLANNING_PROMPT = """
Before taking any actions, produce a concise numbered plan of what you will do.
Format:
PLAN:
1. [step]
2. [step]
...
EXECUTE:
Then execute each step in order.
"""
ReAct (Reasoning and Acting) is a prompting pattern that has the agent explicitly reason before each tool call:
Thought: The user wants to book a flight. I need to first check available flights,
then check pricing, then confirm the booking.
Action: search_flights(from="AMS", to="LHR", date="2025-03-15")
Observation: Found 8 flights. Cheapest is KL1023 at €189 departing 07:30.
Thought: I have flight options. I'll present the best option and confirm.
...
The explicit reasoning step helps with complex tasks but costs tokens and time. For simple, well-defined tasks, it’s overhead.
State Management
Agents accumulate state as they work: the conversation history, tool call results, intermediate data. This state needs to be managed carefully.
Context window limits are real. A long agent run produces a long history. At some point the history exceeds the context window. You need a strategy before you hit that limit, not after.
Options:
- Summarization: periodically summarize earlier parts of the history
- Rolling window: keep only the last N exchanges plus a summary of earlier
- Memory extraction: extract key facts from the history and store them separately
Don’t trust the model’s memory. If the agent computed something in step 3 and needs it in step 15, store it explicitly in structured state rather than relying on the model to recall it from the conversation history.
class AgentState:
conversation: list[Message]
working_memory: dict[str, Any] # Explicit key-value store
completed_steps: list[str]
current_goal: str
Guardrails and Scope Limits
Without explicit limits, an agent will eventually do something you didn’t intend. This is not a hypothetical.
Define what actions the agent is allowed to take. If an agent is helping with customer support, it should not be able to issue refunds over a certain amount without human approval. If an agent is helping with code generation, it should not be able to run arbitrary shell commands.
Implement pre-execution checks. Before calling tools that have side effects (sending emails, executing database writes, calling external APIs), validate the action makes sense:
def execute_tool(tool_name: str, args: dict) -> dict:
tool = self.tools[tool_name]
# Check against allowed actions
if tool.requires_approval:
approval = self.request_human_approval(tool_name, args)
if not approval.granted:
return {"status": "blocked", "reason": approval.reason}
# Budget check
if self.tokens_used > self.token_budget:
return {"status": "budget_exceeded", "message": "Task budget reached"}
return tool.execute(args)
Set iteration limits. An agent that loops indefinitely is not solving your problem — it’s stuck. Hard iteration limits prevent runaway agents:
MAX_ITERATIONS = 25
for iteration in range(MAX_ITERATIONS):
response = llm.complete(messages)
if response.is_final_answer:
return response.content
# Handle tool calls...
raise AgentMaxIterationsError(f"Agent exceeded {MAX_ITERATIONS} iterations")
Human-in-the-Loop
Not all agent decisions should be fully autonomous. Design for appropriate human checkpoints:
- Before irreversible actions: deleting records, sending communications, making purchases
- When confidence is low: if the agent expresses uncertainty, pause for human input
- At defined cost thresholds: “this task will cost approximately $2 — proceed?”
- For exceptions: when the agent encounters a situation it wasn’t designed for
Human-in-the-loop makes agents more reliable and more trustworthy. It also makes them slower. The right balance depends on the risk profile of the actions involved.
Evaluation
An agent that works 80% of the time is not production-ready for most use cases. But how do you know what your success rate is?
Traditional unit tests don’t capture agent failures — the test may pass while the agent produces a wrong answer or takes an unnecessary path. You need task-level evaluation:
Build a labeled evaluation dataset: 50–200 representative tasks with known correct outcomes. Run the agent against them. Measure:
- Task completion rate (did it produce any answer?)
- Task success rate (was the answer correct?)
- Tool call efficiency (did it use the right tools in the right order?)
- Token consumption (did it complete the task within budget?)
Regression testing: run the eval suite before any prompt change, model upgrade, or tool modification. Agent quality is easy to regress silently.
LLM-as-judge: for tasks where the correct answer isn’t a single string, use a separate LLM to evaluate whether the agent’s output is acceptable:
def evaluate_task(task: Task, agent_output: str) -> EvalResult:
prompt = f"""
Task: {task.description}
Expected outcome: {task.expected_outcome}
Agent output: {agent_output}
Did the agent accomplish the task correctly?
Score 1-5 and explain your reasoning.
"""
return eval_llm.complete(prompt)
Observability
Agent runs are opaque by default. You need explicit observability to understand what’s happening.
Log every step of the agent loop: which tool was called, with what arguments, what was returned, what the model reasoned. This is essential for debugging failures.
Track metrics:
- Task completion rate
- Average iterations per task
- Token consumption per task
- Tool call error rate by tool
- Latency per task
Trace agent runs with a unique task ID that flows through all log entries for that run. When a user reports that the agent did something wrong, you need to be able to reconstruct exactly what happened.
Cost Control
LLM API calls are priced per token. An agent that runs many iterations on many requests compounds costs rapidly.
Strategies:
- Token budgets per task: fail explicitly when exceeded rather than continuing
- Model selection by task: use cheaper models for simple subtasks (planning, validation), expensive models for complex reasoning
- Caching: cache tool call results that don’t change frequently
- Summarization: compress long histories before they consume context unnecessarily
Calculate the expected token cost of a typical task and verify that the agent actually stays within that expectation. Surprises here are expensive.
The Engineering Mindset
AI agents are not magic black boxes. They’re software systems with specific failure modes: incorrect tool calls, reasoning errors, context overflow, runaway loops, hallucinated parameters. Every one of these is a software engineering problem with engineering solutions.
The teams that ship reliable agents treat them like any other distributed system: observable, testable, with explicit error handling and well-defined operational bounds. The teams that treat them as magic find that the magic stops working in production.