RAG vs Fine-Tuning: How to Decide

The question is usually framed wrongly

Retrieval and fine-tuning are often presented as competing approaches to the same problem, which is why the decision feels harder than it is. They address genuinely different deficits. Retrieval supplies knowledge the model does not have. Fine-tuning changes behaviour the model will not reliably produce. Confusing the two leads to teams fine-tuning to inject facts, which works poorly and is expensive to keep current.

The diagnostic question is therefore not which technique to use but what is actually going wrong. If the model produces confidently wrong facts about your domain, that is a knowledge problem and retrieval addresses it. If the model knows the facts but will not follow your output format, tone or decision procedure consistently, that is a behaviour problem and fine-tuning addresses it.

A surprising proportion of cases turn out to be neither. Before committing to either architecture, it is worth establishing that a carefully specified prompt with good examples genuinely cannot do the job, because both alternatives add permanent operational cost that prompting does not.

Which technique fits which deficit

What each approach actually costs to run

Retrieval's cost is operational and continuous. You are running an indexing pipeline, keeping it fresh, monitoring recall, and paying for the retrieval infrastructure and the additional tokens that retrieved context consumes on every call. The upside is that updating knowledge means updating documents, which is cheap and can be done by people who are not engineers.

Fine-tuning's cost is front-loaded and periodic. Building a training set is genuine work, and it has to be rebuilt whenever the desired behaviour changes or the base model is upgraded. The upside is inference efficiency: a fine-tuned smaller model can be substantially cheaper and faster per call than prompting a large one, which matters at volume.

The pattern that catches teams out is fine-tuning for knowledge. It appears to work initially, because the model does absorb facts from the training data. Then the facts change, and updating them requires retraining rather than editing a document, at which point the operational burden becomes obvious.

Why the two are often combined

In mature systems the answer is frequently both, applied to the deficits they each address. Retrieval supplies current, citable domain knowledge; a fine-tuned model handles the output structure and decision procedure reliably and cheaply. Neither is doing the other's job.

The sequencing matters. Build retrieval first if knowledge is the primary deficit, because it is faster to stand up and easier to evaluate. Add fine-tuning later, once you have enough production traffic to construct a training set from real interactions rather than from invented examples, which is where fine-tuning quality mostly comes from.

Combining them prematurely is a common way to make a system hard to debug. When both are in play from day one and quality is poor, isolating whether the problem is retrieval recall or model behaviour is considerably harder than it would have been if they had been introduced in sequence.

How to evaluate the decision rather than argue about it

Both approaches are testable cheaply if you build the evaluation set before choosing. Collect fifty real queries where the current system fails, label what the correct output would have been, and score whichever approach you try against that set. This turns an architecture debate into a measurement, and it usually settles in a day what would otherwise consume several meetings.

The specific signal to look for differs by technique. For retrieval, measure whether the correct source document appears in the retrieved set at all — recall at the top few results. If recall is high and answers are still wrong, the problem is not retrieval and adding a better vector database will not help. If recall is low, you have a retrieval problem worth investing in.

For fine-tuning, the signal is consistency rather than correctness. Run the same twenty inputs several times and measure variance in format and decision. High variance that persists despite explicit instructions is the case fine-tuning genuinely addresses; low variance with wrong content is a knowledge problem wearing a behaviour problem's clothing.

Part of the AI Engineering Practice cluster · Read the pillar page