There’s a tendency to treat AI features as different from other software. Different enough that normal engineering practices don’t apply, or apply differently, or can be deferred. This tendency produces AI systems that are unreliable, unobservable, expensive to operate, and difficult to improve.
AI systems built with the same engineering discipline as other distributed systems are more reliable, cheaper to operate, and easier to improve. The disciplines aren’t different. The application is.
Testing AI Features
Unit tests don’t work for AI outputs. assertEquals("Paris", llm.answer("What is the capital of France?")) is fragile — model updates, prompt changes, and temperature variations all produce differently-worded correct answers.
What testing AI requires:
Evaluation datasets: a set of representative inputs with expected outputs (or expected properties of outputs). Run the system against them. Measure whether it meets your quality criteria.
evaluation_cases = [
{
"input": "Summarize this customer complaint: [text]",
"criteria": ["mentions issue category", "includes sentiment", "under 50 words"]
},
...
]
results = run_eval(system, evaluation_cases)
assert results.pass_rate > 0.90, f"System quality below threshold: {results.pass_rate}"
LLM-as-judge: use a separate, powerful LLM to evaluate whether outputs meet quality criteria. Imperfect but scalable. Necessary for subjective qualities (tone, helpfulness, correctness for complex questions).
Regression testing: run the eval suite before every prompt change, model update, or retrieval parameter change. AI quality regressions are silent without systematic measurement.
Behavioral testing: test specific properties you care about — does the system refuse to answer questions outside its scope? Does it cite sources? Does it handle edge cases gracefully?
The critical insight: AI system quality is not binary. It’s a distribution over many inputs. Testing means measuring that distribution against your quality requirements, not verifying specific outputs.
Observability for AI Systems
Standard application observability isn’t sufficient. AI systems have failure modes that don’t show up in error rates or latency metrics.
What to instrument:
Token usage: prompt tokens, completion tokens, cost per request. Token usage that grows unexpectedly indicates a bug (history not being truncated) or abuse.
Quality metrics: task success rate (where measurable), user feedback signals, hallucination rate from automated evaluation.
Retrieval metrics (for RAG): recall at K, reranker scores, retrieval latency by source.
Agent-specific metrics: iterations per task, tool call success rate by tool, abandonment rate.
LLM latency: by model, by prompt template, by output length. P50 is not sufficient — track P95 and P99. LLM latency variance is high.
@observe(name="rag-query")
def answer_question(query: str) -> str:
with trace("retrieval"):
chunks = retriever.retrieve(query)
metrics.histogram("retrieval.chunk_count", len(chunks))
metrics.histogram("retrieval.top_score", chunks[0].score if chunks else 0)
with trace("generation"):
response = llm.complete(build_prompt(query, chunks))
metrics.counter("tokens.prompt", response.usage.prompt_tokens)
metrics.counter("tokens.completion", response.usage.completion_tokens)
return response.content
Logging full interactions: for debugging and evaluation, log the full prompt and response (with PII scrubbing). These are expensive to store but invaluable when something goes wrong.
Versioning and Deployment
AI systems have multiple versioned components: the model, the prompt templates, the retrieval pipeline configuration, the evaluation thresholds.
Model versioning: when you upgrade from GPT-4o to GPT-4o-mini, or from Claude 3.5 to Claude 3.7, behavior changes. Track which model version is deployed to production. Run your eval suite against the new model before promoting.
Prompt versioning: treat prompts like code. Store them in version control. When a prompt changes, the change is reviewed, tested, and deployed like a code change. “Just tweaking the prompt” is a deployment with observable effects.
# Prompts as versioned artifacts
PROMPTS = {
"v1.0.0": "You are a helpful assistant...",
"v1.1.0": "You are a precise, factual assistant...",
}
active_prompt_version = config.get("prompt_version", "v1.1.0")
prompt = PROMPTS[active_prompt_version]
Canary deployments: route a small percentage of traffic to a new model/prompt version. Measure quality metrics. Expand gradually.
Rollback: when a model or prompt update degrades quality, you need to roll back quickly. This is possible only if versions are tracked and deployments are automated.
Cost Management
LLM API costs compound. A system that handles 1,000 requests per day at $0.05 per request costs $1,800 per month. At 100,000 requests per day, it’s $180,000. Cost is not an afterthought.
Token budgets: define the maximum token budget per request. Enforce it. A request that generates 10,000 tokens because the history wasn’t truncated is a bug.
Model selection by task: use expensive models for complex reasoning, cheaper models for simple tasks.
def select_model(task_complexity: float) -> str:
if task_complexity > 0.8:
return "gpt-4o" # Complex reasoning
elif task_complexity > 0.4:
return "gpt-4o-mini" # Standard tasks
else:
return "gpt-3.5-turbo" # Simple classification/routing
Caching: cache responses for identical or near-identical requests. Semantic caching (cache based on query similarity) can be effective for FAQ-type systems.
Cost attribution: attribute costs to teams, features, or customers. This makes over-spending visible and creates accountability for optimization.
Reliability Design
Fallbacks: when the primary LLM is unavailable, what happens?
- Fall back to a secondary provider
- Fall back to a cached response
- Return a degraded experience rather than an error
- Queue the request for async processing
Circuit breakers: if the LLM API is slow or failing, don’t let every request wait. Open the circuit, fail fast, and let the application degrade gracefully.
Retry with backoff: LLM rate limit errors (429) require backoff, not immediate retry. Respect Retry-After headers.
Timeout handling: LLM calls can take 10-60 seconds for complex tasks. Define timeouts. Handle them as explicit failures, not hangs.
try:
response = await asyncio.wait_for(
llm.complete_async(prompt),
timeout=30.0 # 30 second hard timeout
)
except asyncio.TimeoutError:
metrics.counter("llm.timeout")
return FallbackResponse("Request took too long. Please try again.")
except RateLimitError as e:
metrics.counter("llm.rate_limit")
raise RetryableError(retry_after=e.retry_after)
Security Is Not Optional
AI-specific security concerns on top of standard application security:
- Prompt injection through user input or retrieved content
- Data leakage through context (sensitive data in prompts goes to the LLM provider)
- Excessive agency (agents with too many permissions)
- Output validation (don’t trust LLM output directly for security decisions)
These aren’t theoretical. They’re attack vectors that have been demonstrated in production systems.
The standard approach: treat all external inputs as untrusted, apply least privilege to agents and tool permissions, validate outputs before acting on them, and audit AI interactions.
The Cultural Shift
The most common failure mode in AI engineering is treating AI features as prototypes indefinitely. “It’s AI, so it’s probabilistic — we can’t really test it” is a rationalization for not doing the work.
You can measure quality. You can track regressions. You can monitor costs. You can design for reliability. The tooling exists, the patterns exist, and teams that apply them ship AI features that work in production.
The teams that don’t apply these disciplines ship demos.