Most teams evaluate their LLM applications by asking them a few questions and deciding whether the answers look right. This is not evaluation — it’s a vibe check. It doesn’t scale, doesn’t catch regressions, and doesn’t provide any basis for measuring improvement over time.
Systematic LLM evaluation is harder than evaluating deterministic software. The outputs are probabilistic, quality is multidimensional, and the correct answer often isn’t a single string. These are difficulties, not reasons to skip evaluation.
Why Traditional Testing Fails for LLMs
Unit tests verify exact outputs: assertEquals("Paris", answer). LLM outputs for the same input vary between runs. “Paris is the capital of France” and “The capital of France is Paris” are both correct answers to the same question.
More importantly, a test that passes for “what is the capital of France?” doesn’t tell you whether the system handles “what’s the best restaurant in Paris?” correctly, or whether it admits uncertainty when it should, or whether it avoids hallucinating when the context doesn’t contain relevant information.
LLM testing requires evaluation datasets — collections of inputs with expected properties — and metrics that measure those properties systematically across the dataset.
Building Evaluation Datasets
An evaluation dataset contains:
- Representative input queries
- Expected outputs (or expected properties of outputs)
- Ground truth context (for RAG systems)
How to build one:
- Sample real user queries from your application logs (with consent)
- Supplement with synthetic queries that cover edge cases
- Have domain experts label correct answers (or correct source documents for RAG)
- Include adversarial examples: questions outside scope, ambiguous queries, trick questions
The dataset should reflect the actual distribution of queries your system receives, not just the easy ones.
A minimal dataset structure for a RAG system:
eval_dataset = [
{
"question": "What is the refund policy for digital products?",
"ground_truth": "Digital products are refundable within 14 days if unused.",
"ground_truth_context": "refund-policy.md#digital-products"
},
{
"question": "How do I cancel my subscription?",
"ground_truth": "Log into your account, go to Billing, click Cancel Subscription.",
"ground_truth_context": "account-management.md#cancel-subscription"
},
{
"question": "What is the weather like today?", # Out of scope
"ground_truth": None,
"expected_behavior": "politely_decline"
}
]
Start with 50–100 examples. This is enough to detect meaningful quality differences. Expand as you discover important categories of queries.
Metrics for RAG Systems
RAGAS (Retrieval Augmented Generation Assessment) provides four core metrics for RAG systems:
Context Recall: does the retrieved context contain the information needed to answer the question?
- Measures: fraction of ground truth statements that are present in the retrieved context
- Formula: statements in ground truth present in context / total ground truth statements
- What a low score means: your retrieval pipeline is missing relevant documents
Faithfulness: is the answer supported by the retrieved context?
- Measures: fraction of answer statements that are supported by the context
- Formula: answer statements supported by context / total answer statements
- What a low score means: the model is hallucinating, adding information not in the context
Answer Relevance: does the answer actually address the question?
- Measures: whether the answer responds to the specific question asked
- What a low score means: the model is answering a different question, or giving a generic response
Context Precision: is the retrieved context relevant? (reduces noise)
- Measures: fraction of retrieved context chunks that are relevant to the question
- What a low score means: you’re retrieving irrelevant content and adding noise to the prompt
from ragas import evaluate
from ragas.metrics import (
context_recall,
faithfulness,
answer_relevancy,
context_precision
)
from datasets import Dataset
# Prepare the dataset
eval_data = {
"question": [case["question"] for case in eval_dataset],
"contexts": [case["retrieved_contexts"] for case in eval_dataset], # Retrieved chunks
"answer": [case["generated_answer"] for case in eval_dataset], # LLM output
"ground_truth": [case["ground_truth"] for case in eval_dataset]
}
dataset = Dataset.from_dict(eval_data)
results = evaluate(
dataset=dataset,
metrics=[context_recall, faithfulness, answer_relevancy, context_precision]
)
print(results)
# {'context_recall': 0.82, 'faithfulness': 0.91, 'answer_relevancy': 0.87, 'context_precision': 0.75}
LLM-as-Judge
For subjective qualities that metrics don’t capture — tone, helpfulness, completeness, correctness for complex multi-step questions — use a powerful LLM to evaluate your system’s outputs.
def evaluate_response(question: str, response: str, criteria: list[str]) -> EvalResult:
prompt = f"""Evaluate this AI assistant response on the following criteria.
Question: {question}
Response: {response}
Criteria to evaluate (score 1-5 for each):
{chr(10).join(f"{i+1}. {c}" for i, c in enumerate(criteria))}
Respond in JSON: {{"scores": {{}}, "reasoning": {{}}, "overall": 1-5}}"""
result = eval_llm.complete(prompt, response_format={"type": "json_object"})
return EvalResult.from_json(result.content)
# Example
result = evaluate_response(
question="Explain how neural networks learn",
response=system_response,
criteria=[
"Technical accuracy",
"Appropriate complexity for a software engineer audience",
"Clear and well-structured explanation",
"Correct use of analogies without oversimplification"
]
)
LLM-as-judge has well-documented biases (preference for verbose answers, recency bias in pairwise comparison). Use it for directional guidance and relative comparison (is v2 better than v1?) rather than absolute quality measurement.
Pairwise Comparison
Instead of scoring individual responses, compare two responses and ask which is better. This is often more reliable than absolute scoring:
def compare_responses(question: str, response_a: str, response_b: str) -> str:
prompt = f"""Which response better answers this question?
Question: {question}
Response A: {response_a}
Response B: {response_b}
Reply with just "A" or "B" and a one-sentence reason."""
result = eval_llm.complete(prompt)
return result.content # "A" or "B"
# Run A/B comparison between old and new prompt versions
wins_a = sum(1 for case in eval_cases
if compare_responses(case.q, run_v1(case.q), run_v2(case.q)).startswith("A"))
wins_b = len(eval_cases) - wins_a
print(f"v1 wins: {wins_a}/{len(eval_cases)}, v2 wins: {wins_b}/{len(eval_cases)}")
Regression Testing
Every change to the system — prompt update, model upgrade, retrieval parameter change, embedding model swap — should be evaluated against your baseline.
class EvalBaseline:
def __init__(self, path: str):
self.metrics = json.loads(Path(path).read_text())
def check_regression(self, new_metrics: dict, tolerance: float = 0.02) -> list[str]:
regressions = []
for metric, baseline_score in self.metrics.items():
new_score = new_metrics.get(metric, 0)
if new_score < baseline_score - tolerance:
regressions.append(
f"{metric}: {baseline_score:.3f} → {new_score:.3f} "
f"(regression: {baseline_score - new_score:.3f})"
)
return regressions
# In CI/CD
baseline = EvalBaseline("eval/baseline.json")
current_metrics = run_evaluation(system, eval_dataset)
regressions = baseline.check_regression(current_metrics)
if regressions:
print("QUALITY REGRESSION DETECTED:")
for r in regressions:
print(f" - {r}")
sys.exit(1)
This fails the CI pipeline when quality regresses below the baseline. Teams that implement this catch quality regressions before deployment rather than after.
Behavioral Testing
Beyond quality metrics, test specific behavioral requirements:
behavioral_tests = [
{
"category": "scope_enforcement",
"input": "What is 2+2?",
"expected": lambda r: "cannot" in r.lower() or "outside" in r.lower(),
"description": "Should decline out-of-scope questions"
},
{
"category": "uncertainty",
"input": "What will the stock market do tomorrow?",
"expected": lambda r: any(w in r.lower() for w in ["don't know", "uncertain", "cannot predict"]),
"description": "Should express uncertainty about unknowable facts"
},
{
"category": "source_citation",
"input": "What is our return policy?",
"expected": lambda r: "[Source:" in r or "According to" in r,
"description": "Should cite sources for factual claims"
}
]
for test in behavioral_tests:
response = system.answer(test["input"])
passed = test["expected"](response)
if not passed:
print(f"BEHAVIORAL TEST FAILED: {test['description']}")
print(f" Input: {test['input']}")
print(f" Response: {response}")
The Minimum Viable Evaluation Setup
You don’t need a sophisticated evaluation framework to start. The minimum:
- A spreadsheet with 30–50 representative questions and expected behaviors
- A script that runs your system against all questions and records outputs
- A human review of the outputs, looking for failures
This takes a day to set up. Run it before every significant change to the system. The discipline of systematic review will catch regressions that ad hoc testing misses.
Automate the mechanical parts as you go. But don’t let the perfect be the enemy of the good — even manual review of a fixed test set is vastly better than vibe checks.