Retrieval-Augmented Generation has become the default answer to “how do we make an LLM answer questions about our data.” The concept is simple enough to explain in one slide: embed your documents, store them in a vector database, retrieve the relevant ones at query time, pass them to the LLM.
The problem is that the one-slide version produces one-slide-quality results. Good RAG systems are significantly harder to build than the demos suggest, and the failures are often subtle — the system produces confident, plausible-sounding answers that are wrong in ways that are hard to detect without systematic evaluation.
This article explains what actually makes RAG hard and how to build systems that work reliably in production.
What RAG Is Actually Doing
Before discussing failure modes, let’s be precise about what RAG does.
The ingestion pipeline processes your documents:
- Documents are split into chunks
- Each chunk is embedded using an embedding model, producing a high-dimensional vector
- Vectors are stored in a vector database with the original text
The retrieval pipeline answers a query:
- The query is embedded using the same model
- Vectors similar to the query vector are retrieved (nearest neighbor search)
- Retrieved chunks are assembled into context
- The LLM generates an answer given the context
This sounds like a database query. It is not. A database query is deterministic and exact. RAG retrieval is probabilistic and approximate. The retrieved context is not necessarily the correct context — it’s the context that is semantically similar to the query according to the embedding model.
That difference is where most RAG failures originate.
The Chunking Problem
Chunking is the first place where RAG goes wrong, and it is consistently underestimated.
The naive approach is fixed-size chunking: split every N characters, step M characters. This is fast to implement and reliably mediocre.
Why fixed-size chunking fails:
A paragraph discussing a single concept gets split in half. The two resulting chunks each contain half the semantic information. When either chunk is retrieved, the LLM lacks the context to answer correctly. Worse, both halves may be retrieved — consuming context window budget — while the combined information they contain is still less useful than the original paragraph.
Better chunking strategies:
Semantic chunking: split on natural boundaries — sentences, paragraphs, sections. Respect document structure. This requires understanding the document format (HTML, Markdown, PDF, plain text are all different).
Hierarchical chunking: maintain two levels — small chunks for precise retrieval, large chunks for context richness. Retrieve at the small level, but expand to the large level before sending to the LLM.
Small chunk (for retrieval):
"Virtual threads are scheduled by the JVM, not the OS."
Parent chunk (for context, sent to LLM):
"Project Loom introduces virtual threads as a lightweight concurrency
mechanism. Virtual threads are scheduled by the JVM, not the OS.
This means you can have millions of virtual threads without the memory
overhead of platform threads. Each virtual thread consumes roughly 1KB
of stack space compared to 1MB for platform threads..."
Document-aware chunking: code blocks, tables, and lists have different semantic density than prose. Split them differently. A code example should rarely be split in the middle of a function.
Overlap: include a window of content from the surrounding chunks to catch concepts that span boundaries.
The right chunking strategy depends on your content. Evaluate it empirically: measure whether the correct information appears in the retrieved context for a representative set of test queries.
Embeddings Are Not Universal
The choice of embedding model matters more than most implementations acknowledge.
General-purpose models (OpenAI text-embedding-3-large, Cohere embed-v3, etc.) are good starting points. They perform well on general questions. They may perform poorly on domain-specific content.
A legal contract uses vocabulary and semantic patterns that don’t appear in the web crawl data these models were trained on. A technical security document uses abbreviations and jargon with specific meanings. A customer support knowledge base has question-answer patterns that differ from general prose.
Evaluate before committing. Run your embedding model against a test set of queries with known correct documents. Measure recall@K — for a given K retrieved chunks, what fraction of the correct answers are in those K chunks? A number below 0.8 for top-5 retrieval is a problem that better chunking or a different model can often fix.
Models to consider beyond the defaults: Cohere’s domain-specific models, bge-m3 for multilingual, and fine-tuned sentence transformers for specialized domains.
Hybrid Search: Combine Semantic and Keyword
Pure vector search has a systematic weakness: exact term matching. When a user asks about a specific product SKU, a person’s name, a technical error code, or any other specific identifier, vector search may miss it entirely — the embedding for “SKU-4421-B” is not obviously close to the document that contains it.
BM25 (keyword search) handles exact matches well. Vector search handles semantic similarity well. Production RAG systems need both.
Most vector databases now support hybrid search. The typical implementation:
- Run vector search for top-K semantically similar chunks
- Run BM25 keyword search for top-K lexically similar chunks
- Merge the two result sets using Reciprocal Rank Fusion (RRF) or a weighted linear combination
def hybrid_search(query: str, k: int = 20) -> list[Chunk]:
vector_results = vector_db.search(embed(query), k=k)
keyword_results = bm25_index.search(query, k=k)
return reciprocal_rank_fusion([vector_results, keyword_results], k=k)
The improvement from hybrid search is often substantial — typically 10-25% improvement in retrieval recall compared to pure vector search, especially on domain-specific queries.
Reranking: The Second Pass That Matters
Initial retrieval (vector + BM25) optimizes for speed. It uses approximate methods and simple scoring. The result is a candidate set that likely contains the relevant chunks but also likely contains noise.
A cross-encoder reranker provides a much more accurate relevance score at the cost of higher compute. Unlike bi-encoders (which embed query and document separately), cross-encoders process the query and document together and output a single relevance score.
def retrieve(query: str, final_k: int = 5) -> list[Chunk]:
# Retrieve more candidates than we need
candidates = hybrid_search(query, k=20)
# Rerank with a cross-encoder
scored = reranker.score(query, candidates)
scored.sort(key=lambda x: x.score, reverse=True)
return [chunk for chunk, score in scored[:final_k]]
Common reranker options: Cohere Rerank, cross-encoder/ms-marco-MiniLM-L-6-v2 (open source, fast), bge-reranker-large (strong quality).
The compute cost is manageable because reranking is only applied to a small candidate set (20–50 chunks), not the entire corpus.
Query Understanding
The query the user types is often not the best query for retrieval. Common problems:
Conversational context: “What did you mean by that?” has no retrieval value without the previous message.
Vocabulary mismatch: a user asks “how do I make my app faster” when the relevant document says “performance optimization strategies”.
Multi-part questions: “What is the refund policy and how long does it take?” should probably be decomposed into two retrieval queries.
Query expansion: generate alternative phrasings of the query and retrieve for each. Merge the results.
def expand_query(query: str) -> list[str]:
prompt = f"""Generate 3 alternative ways to phrase this search query
that might match different relevant documents.
Original query: {query}
Return only the 3 alternatives, one per line."""
alternatives = llm.complete(prompt).strip().split('\n')
return [query] + alternatives[:3]
HyDE (Hypothetical Document Embeddings): instead of embedding the query, ask the LLM to generate a hypothetical document that would answer the query, then embed that. This often retrieves more relevant results because the hypothetical document uses the vocabulary of the corpus.
Context Construction
Retrieved chunks are not ready to send directly to the LLM. They need to be assembled thoughtfully.
Ordering matters: LLMs pay more attention to content at the beginning and end of the context window. Put the most relevant chunks first.
Deduplication: if two retrieved chunks are near-duplicates (common with overlap), include only one.
Metadata: include source information (document title, section, date) so the LLM can reason about recency and provenance.
Context budget: most of your context window budget should go to retrieved context, not the system prompt. A common mistake is a 2,000-token system prompt that leaves 1,000 tokens for context.
def build_context(chunks: list[Chunk], max_tokens: int = 4000) -> str:
context_parts = []
token_count = 0
for chunk in chunks:
chunk_tokens = count_tokens(chunk.text)
if token_count + chunk_tokens > max_tokens:
break
context_parts.append(
f"[Source: {chunk.document_title}, {chunk.section}]\n{chunk.text}"
)
token_count += chunk_tokens
return "\n\n---\n\n".join(context_parts)
Evaluation: The Discipline You Can’t Skip
A RAG system without systematic evaluation is guesswork. The output looks plausible whether it’s working or not.
The evaluation framework has two distinct parts:
Retrieval evaluation: does the system retrieve the correct information?
- Recall@K: fraction of test queries where the correct chunk appears in the top K results
- MRR (Mean Reciprocal Rank): average of 1/rank of the first correct result
- Build a labeled test set: 100–500 queries with known correct source documents
Answer evaluation: given the retrieved context, does the LLM answer correctly?
- Faithfulness: is the answer supported by the retrieved context? (detects hallucination)
- Answer relevance: does the answer address the question?
- Factual correctness: against a ground-truth answer set
- LLM-as-judge: use a powerful LLM to evaluate answers — imperfect but scalable
Frameworks like RAGAS provide ready-made metrics:
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_recall
results = evaluate(
dataset=test_dataset,
metrics=[faithfulness, answer_relevancy, context_recall]
)
Run evaluation continuously. Every change to chunking, embedding model, retrieval parameters, or prompt should be evaluated against the same test set. Regression is common and silent without systematic measurement.
The HNSW Index
Vector databases use Approximate Nearest Neighbor (ANN) algorithms to find similar vectors efficiently. The dominant algorithm today is HNSW (Hierarchical Navigable Small World).
HNSW builds a hierarchical graph structure that allows navigation from a coarse level to a precise level during search. It achieves query times that are sub-linear in the corpus size, with a tunable trade-off between accuracy and speed.
Parameters that matter:
m: number of bidirectional links per node — higher values improve recall but increase memoryef_construction: quality of the index build — higher is more accurate but slower to indexef_search: quality of the search — higher is more accurate but slower per query
For most applications you don’t need to tune these manually. The defaults in most vector databases are reasonable. When you’re operating at tens of millions of documents and need tight latency SLAs, then tuning matters.
What Good RAG Looks Like
A production RAG system with properly implemented retrieval:
- Hybrid search (vector + BM25) with reranking
- Document-aware chunking with hierarchical context
- Query expansion or reformulation
- Systematic evaluation on a labeled test set
- Fallback behaviour when confidence is low
It is substantially more complex than the one-slide demo. It’s also substantially more reliable.
The teams that implement the minimal version and then spend months tweaking the LLM prompt trying to fix problems that are actually retrieval problems are paying the cost twice. Build the retrieval layer properly first.