Rag on dplabs — Software Engineering & Technology Consultancy

Evaluating LLM Applications: Beyond Vibe Checks

Mon, 26 Jan 2026 00:00:00 +0000

Most teams evaluate their LLM applications by asking them a few questions and deciding whether the answers look right. This is not evaluation — it’s a vibe check. It doesn’t scale, doesn’t catch regressions, and doesn’t provide any basis for measuring improvement over time.

Systematic LLM evaluation is harder than evaluating deterministic software. The outputs are probabilistic, quality is multidimensional, and the correct answer often isn’t a single string. These are difficulties, not reasons to skip evaluation.

RAG vs Fine-Tuning: Choosing the Right Tool for LLM Customization

Mon, 12 Jan 2026 00:00:00 +0000

“Should we use RAG or fine-tuning?” is a question teams ask when they need an LLM to work better for their specific use case. It’s often the wrong question — they’re different tools that solve different problems and can be combined.

Getting this wrong is expensive. Fine-tuning a model to “know” information that should be retrieved leads to a model that’s expensive to update and confidently answers with stale data. Implementing RAG when the problem is actually about model behavior leads to a system that retrieves correctly but still behaves wrong.

RAG Is Not a Database Query

Mon, 21 Oct 2024 00:00:00 +0000

Retrieval-Augmented Generation has become the default answer to “how do we make an LLM answer questions about our data.” The concept is simple enough to explain in one slide: embed your documents, store them in a vector database, retrieve the relevant ones at query time, pass them to the LLM.

The problem is that the one-slide version produces one-slide-quality results. Good RAG systems are significantly harder to build than the demos suggest, and the failures are often subtle — the system produces confident, plausible-sounding answers that are wrong in ways that are hard to detect without systematic evaluation.