Every few weeks a founder asks us some version of the same question: "Should we fine-tune a model on our data?" Almost always, the honest answer is no — and the reason is that RAG vs fine-tuning isn't really a technical debate. It's a question about what's broken. Fine-tuning is the answer people reach for because it sounds like ownership: our model, trained on our data. But when you trace the actual problem back to its root, it's usually that the model doesn't know something — and fine-tuning is a surprisingly bad way to make a model know things.
This guide gives you the one question that settles the choice, what each approach genuinely costs, and the specific cases where fine-tuning really is the right call.
What each approach actually does
The two get compared as if they're alternatives on the same axis. They aren't.
RAG: the model looks it up
Retrieval-augmented generation leaves the model completely untouched. You take your documents, split them into chunks, convert each chunk into a vector, and store them. When a user asks something, you search that store for the most relevant chunks and paste them into the prompt before the model answers.
The model isn't smarter. It just got handed the right page of the manual a half-second before it had to answer.
Fine-tuning: the model changes
Fine-tuning continues training a base model on your examples, updating its weights. You're not giving it facts — you're shifting its behavior. The output of fine-tuning is a new model artifact you own and host or serve through a provider.
The distinction matters because it predicts what each one is good at. RAG changes what the model can see. Fine-tuning changes how the model responds.
The one question that settles it
Before comparing costs or architectures, answer this:
Is the model failing because it doesn't know something, or because it doesn't behave the way you want?
If your support bot gives a confidently wrong answer about your refund policy, that's a knowledge problem. It has never seen your refund policy. No amount of fine-tuning reliably fixes that — and OpenAI's own model optimization guidance frames this as context optimization: the model lacks knowledge because it wasn't in the training set, is out of date, or is proprietary to you. That's RAG's job.
If your bot knows the policy but explains it in six rambling paragraphs when you need two crisp sentences in your brand voice, that's a behavior problem. That's where fine-tuning earns its keep.
Most founders arrive convinced they have a behavior problem. Most of them have a knowledge problem.
When RAG is the right answer
AWS's Prescriptive Guidance is unusually direct about this. Their recommendation: if you need to build a question-answering solution that references your custom documents, start from a RAG-based approach (AWS, Comparing RAG and fine-tuning). Four properties drive that.
Your content changes
AWS notes fine-tuning can take hours to days depending on model size, which makes it a poor fit if your documents change frequently — while RAG can incorporate the latest documents in minutes. If your pricing page changes on Tuesday, a RAG system is correct on Tuesday. A fine-tuned model is correct after the next training run, whenever you get around to it.
This alone disqualifies fine-tuning for most startup use cases. Startups change everything constantly.
You need citations
A RAG system can point at the chunk it used. A fine-tuned model cannot — AWS lists "fine-tuned models do not provide a reference to the source in their responses" as a straight disadvantage.
For anything a customer will act on — a legal answer, a medical instruction, a financial number, a policy — citation isn't a nice-to-have. It's the difference between a tool people trust and a tool people quietly stop using after it burns them once.
You want fewer hallucinations, not more
This is the part that surprises people. AWS flags an increased risk of hallucination when using a fine-tuned model to answer questions, and a reduced risk with RAG, because RAG grounds the answer in retrieved context. Fine-tuning on your documents can make a model more fluent about your domain without making it more correct — which is the worst possible combination. It sounds like an expert and is wrong.
You don't have a data scientist
AWS's framing again: fine-tuning requires understanding techniques like LoRA and parameter-efficient fine-tuning, and might require a data scientist. RAG has fully managed options — AWS points to its own — that don't. If you're a five-person team, that's not a small consideration — it's the whole consideration.
We built a RAG-powered knowledge base for exactly this shape of problem: natural-language search across internal documentation, with citations back to the source doc. The gain was in retrieval quality and time saved, not in a custom model — there wasn't one.
When fine-tuning is genuinely the right answer
It's a real tool. It's just narrower than the hype suggests.
Consistent format or structure. If you need every output in a rigid schema and prompt instructions keep drifting, examples beat instructions. This is the cleanest fine-tuning win.
Voice and style at scale. AWS notes that a model fine-tuned using the unsupervised approach can produce content that more closely matches your organization's style. If your product's entire value is that it writes like you across thousands of outputs, that's weights, not prompts.
Tasks beyond question-answering. AWS's own carve-out: use fine-tuning if you need the model to perform additional tasks, such as summarization — and separately notes RAG "does not work well when summarizing information from entire documents," since retrieval fetches fragments, not wholes.
Squeezing a smaller model. A fine-tuned small model can sometimes match a much larger general model on one narrow task, at lower cost and latency. This is a real optimization — but it's an optimization, which means you should have a working product and a cost problem first.
Notice what's absent from that list: teaching the model new facts. That's the use case people want and the one fine-tuning is worst at.
What this actually costs
Be skeptical of anyone quoting precise dollar figures here — provider pricing changes often, and the real cost is usually engineering time, not inference. Directionally, and worth verifying against current pricing pages before you budget:
RAG's costs are ongoing and mostly boring. Embedding your corpus once, re-embedding as it changes, vector storage, and a slightly larger prompt on every request because you're injecting retrieved context. That last one is the sleeper cost — RAG makes every single call more expensive, forever. It's usually still cheaper than the alternative.
Fine-tuning's costs are lumpy and front-loaded. The training run itself, and — the part nobody budgets for — building the training set. Hundreds to thousands of high-quality examples, which someone has to write, review, and clean. Then you do a meaningful fraction of it again every time the base model you built on gets deprecated.
The pattern to internalize: RAG costs money per query. Fine-tuning costs people per iteration. For a startup, people are the scarcer resource. That asymmetry, not the API bill, is why we default to RAG.
The hybrid case
You can do both, and AWS explicitly describes it: keep the RAG architecture unchanged, but also fine-tune the LLM that generates the answer. Berkeley researchers formalized a version of this in RAFT: Adapting Language Model to Domain Specific RAG — training a model specifically to reason well over retrieved documents, including irrelevant ones.
Microsoft's guidance lands in the same place, framing RAG and fine-tuning as fitting different needs rather than competing (Microsoft Learn, Augment LLMs with RAG or Fine-Tuning).
This is a legitimate production architecture. It is also almost never where you should start. The hybrid is what you build in year two, when RAG works, you've measured exactly where it falls short, and you have the evaluation harness to prove the fine-tune helped. Starting hybrid means paying for both and being unable to tell which one is failing.
A decision checklist
Work down this list and stop at the first yes:
- Can a better prompt fix it? Try this first. It's free and it works more often than anyone admits. The same logic as picking a boring tech stack — exhaust the cheap option before buying the expensive one.
- Does the answer live in documents, records, or data that changes? → RAG.
- Do users need to see where the answer came from? → RAG.
- Is the model's knowledge the gap? → RAG. Fine-tuning will not fix this.
- Does the same output defect appear across hundreds of examples, and prompting can't fix it? → Fine-tuning.
- Do you need summarization over whole documents, not fragments? → Fine-tuning.
- Is RAG working, measured, and provably not enough? → Consider hybrid.
If you're in the first four, you're in RAG territory — which is most products. For the wider question of which AI features are worth building at all, we covered that in AI features to add to your MVP.
FAQ
Can fine-tuning teach a model new facts?
Not reliably, and it's the most common misconception in this whole area. Fine-tuning shifts behavior — tone, format, task performance. Facts injected this way tend to surface inconsistently, and can't be cited or updated. If you need the model to know something, retrieve it.
Is RAG cheaper than fine-tuning?
Usually, but the framing is off. RAG spreads cost across every query; fine-tuning concentrates it into training runs and dataset creation. For most startups RAG is cheaper in total because the dominant cost is engineering time, and RAG needs far less of it. Verify current inference and training prices with your provider before budgeting.
How long does a RAG system take to build?
For a scoped internal knowledge base, weeks rather than months — the pipeline (chunk, embed, store, retrieve, inject) is well-trodden and there are managed options. The hard part is never the pipeline. It's retrieval quality: chunking strategy, embedding choice, and whether your source documents are any good. Bad documents produce a bad RAG system, quickly.
Do we need a vector database?
For anything non-trivial, yes — but "database" oversells it. Several managed options and Postgres extensions handle this without a new piece of infrastructure to operate. Don't let the vector store become the interesting part of your architecture.
What if our data is sensitive?
This is an argument for careful architecture, not for fine-tuning. RAG generally keeps proprietary content in a store you control and sends only relevant fragments at query time. Fine-tuning bakes your data into model weights — which is harder to audit and, with current tooling, impractical to selectively delete. If anything, sensitivity argues for RAG.
Should we fine-tune an open-source model instead?
Same question, one layer down. Self-hosting changes your cost structure and data residency story, but it doesn't change whether your problem is knowledge or behavior. Answer that first, then decide where the model runs.
The short version
RAG vs fine-tuning is decided by one question, and most teams answer it wrong because fine-tuning sounds more serious. If the model doesn't know something, retrieve it. If the model doesn't behave right and prompting has genuinely failed, fine-tune it. If neither is measured yet, you're not ready to build either.
We build AI features for startups that ship — retrieval systems with citations, document processing, support agents. Have a look at our AI integration work or see what we've built. If you're staring at this decision and want a straight answer about which one your product needs, tell us what you're building — we'll tell you honestly, including when the answer is "neither yet."