“Should we use RAG or fine-tuning?” is a question teams ask when they need an LLM to work better for their specific use case. It’s often the wrong question — they’re different tools that solve different problems and can be combined.
Getting this wrong is expensive. Fine-tuning a model to “know” information that should be retrieved leads to a model that’s expensive to update and confidently answers with stale data. Implementing RAG when the problem is actually about model behavior leads to a system that retrieves correctly but still behaves wrong.
What Fine-Tuning Actually Does
Fine-tuning trains an existing model on additional data. The model’s weights are updated to reflect the new training examples. The result is a model that has internalized patterns from your training data.
Fine-tuning is effective for:
- Changing how the model responds: teaching it to use a specific format, adopt a specific tone, or follow specific instructions it doesn’t follow reliably out-of-the-box
- Domain-specific language and terminology: if your domain uses specialized vocabulary or conventions that a general model doesn’t handle well, fine-tuning can teach these patterns
- Consistent instruction following: when the base model doesn’t reliably follow formatting or output structure requirements even with explicit prompting
Fine-tuning is not effective for:
- Making the model “know” current information: a fine-tuned model’s knowledge is frozen at the training data cutoff. Fine-tuning on your documents doesn’t give the model access to new documents — it bakes the content of those specific documents into the weights
- Updating information frequently: fine-tuning a model is expensive (time, compute, money) and can’t be done continuously as your data changes
- Factual accuracy on specific data: a fine-tuned model will confidently answer based on its training, even if that training data was wrong or is now outdated
The classic fine-tuning mistake: a company fine-tunes GPT-4 on their product documentation. Three months later, the product changes. The fine-tuned model confidently describes the old product to customers. The company has to fine-tune again — expensive, slow.
What RAG Does
RAG provides relevant external information to the model at inference time, without changing the model’s weights. The model’s capabilities stay the same; it just has more relevant context when answering.
RAG is effective for:
- Providing access to current information: retrieve today’s documentation, today’s database records, today’s internal documents
- Grounding answers in specific sources: verifiable, citable answers based on retrieved content
- Reducing hallucination for factual questions: with explicit context, the model can answer correctly rather than confabulating
- Large, frequently-updated knowledge bases: add new documents and they’re immediately available, no retraining needed
RAG is not effective for:
- Teaching the model to behave differently: if the model doesn’t follow your format requirements, providing relevant retrieved content won’t fix that
- Replacing domain expertise the model lacks: if the base model doesn’t understand your specialized domain terminology, retrieving documents written in that terminology won’t help much
- Very large context requirements: some tasks require more context than any feasible retrieval approach can provide
The Decision Framework
Ask these questions in order:
1. Is this a knowledge problem or a behavior problem?
Knowledge problem: “The model answers incorrectly because it doesn’t have access to our specific information (internal documentation, product details, current data).” → RAG
Behavior problem: “The model has the knowledge but responds in the wrong format, wrong tone, or doesn’t follow our specific conventions.” → Fine-tuning or better prompting
2. How frequently does the information change?
Changes daily or weekly → RAG (fine-tuning is too slow and expensive for frequent updates) Changes rarely or never → both RAG and fine-tuning are viable
3. How large is the knowledge base?
Fits in a context window → prompt engineering (just include it directly, no retrieval needed) Larger than a context window, smaller than hundreds of thousands of documents → RAG Massive, stable, domain-specific → possibly fine-tuning (for the domain patterns, not the facts)
4. What’s the cost sensitivity?
Fine-tuning: upfront training cost + inference cost for a larger model RAG: infrastructure cost (vector database, embedding API) + potentially lower inference cost (can use a smaller model with good context)
Prompt Engineering: Often Underestimated
Before considering RAG or fine-tuning, try prompt engineering thoroughly. Many behavior problems can be solved by:
- More specific system prompts
- Few-shot examples (2–5 examples of the desired behavior)
- Explicit output format specifications
- Chain-of-thought instructions
System: You are a customer support assistant for Acme Corp.
Always respond in the following JSON format:
{
"category": "<billing|technical|account|other>",
"urgency": "<high|medium|low>",
"response": "<your response here>",
"escalate": <true|false>
}
Examples:
User: My payment failed
Response: {"category": "billing", "urgency": "high",
"response": "I can help...", "escalate": false}
If prompt engineering gets you 90% of the way there, it’s probably good enough. Don’t over-engineer.
Structured Outputs and Tool Calling
Two other mechanisms that solve specific problems:
Structured outputs: modern LLM APIs (OpenAI, Anthropic) support constrained generation where the model is guaranteed to produce valid JSON matching your schema. No parsing required, no format failures.
response = openai.chat.completions.create(
model="gpt-4o",
messages=[...],
response_format={
"type": "json_schema",
"json_schema": {
"schema": {
"type": "object",
"properties": {
"category": {"type": "string"},
"confidence": {"type": "number"}
},
"required": ["category", "confidence"]
}
}
}
)
Tool calling: the model calls predefined functions to retrieve data, perform calculations, or take actions. For data access problems that don’t need semantic search, direct function calls are simpler and more reliable than RAG.
# Tool calling: exact lookup, no semantic search
tools = [{
"name": "get_order_status",
"description": "Get the current status of an order",
"parameters": {
"type": "object",
"properties": {
"order_id": {"type": "string"}
}
}
}]
Tool calling is often better than RAG for structured data retrieval (database queries, API calls). RAG is better for unstructured document retrieval (documentation, long-form content).
Combining Approaches
The options aren’t mutually exclusive. Common combinations:
Fine-tuning + RAG: fine-tune for behavior and domain language; use RAG for current factual content. The fine-tuned model knows how to talk about your domain; RAG gives it the current facts.
Prompt engineering + RAG: system prompt defines behavior; RAG provides relevant context. This is the default approach and often sufficient.
Tool calling + RAG: use tool calling for exact lookups (get order by ID); use RAG for semantic search over documents.
The Practical Starting Point
Start with prompt engineering and tool calling for structured data. These are cheap, fast to iterate, and often sufficient.
Add RAG when you need semantic search over unstructured documents and the knowledge base doesn’t fit in a prompt.
Consider fine-tuning only when:
- Prompt engineering doesn’t reliably produce the desired behavior
- Your use case justifies the cost and complexity
- The knowledge base is stable enough that frozen weights aren’t a problem
Fine-tuning with the right data on a smaller base model can produce something better than prompting a larger general model — at lower inference cost. But this optimization makes sense when you’re at scale, not when you’re figuring out whether the use case works at all.