Fine-tuning vs RAG
When to Use Which, and Why Most Teams Get This Wrong
The meeting had been going on for forty minutes and Siddharth was starting to feel uneasy.
His company was building an internal legal assistant. Lawyers would ask it questions about case law, contract clauses, and regulatory filings. The product team had done their research. The design was solid. The retrieval pipeline was working in staging. Everything looked good.
And then the VP of Engineering said: “Before we go further, I want to understand why we are not just fine-tuning a model on our legal corpus. We have ten years of case files. That is a lot of data. Shouldn’t a model trained on that data be smarter than one that just searches it?”
The room went quiet. Three engineers exchanged glances. It was a reasonable question. It sounded technically sophisticated. And Siddharth realized, with a creeping discomfort, that he did not have a clean answer ready.
He knew RAG worked. He had built it. He had tested it. But he had never sat down and really worked through why fine-tuning was or was not the better choice for this specific use case. He had just assumed RAG was correct and moved forward.
The VP was still looking at him.
“The most expensive AI mistake you can make is choosing the wrong approach before you understand the trade-offs. The second most expensive is not being able to explain why your choice is right.”
This week we fix that gap. We are going to go deep on both fine-tuning and RAG: what each one actually does under the hood, what each one costs, where each one genuinely wins, and how to make the call confidently the next time someone in a meeting room asks you exactly what that VP asked Siddharth.
What each approach actually does
Before you can compare them, you need a clear mental model of what is happening inside each approach. They are more different than most people realize.
What RAG does
RAG does not touch the model at all. The model weights stay exactly as they came from OpenAI, Anthropic, or wherever you got them. What RAG does is change what information goes into the prompt at the time of each query.
When a user asks a question, your application retrieves relevant chunks from your knowledge base and hands them to the model as context. The model reads that context and generates an answer. The model’s knowledge does not change. Its behavior for this specific query does, because you gave it the right information to work with.
Think of it like giving a very knowledgeable consultant a folder of documents and saying: answer this question using what is in here. The consultant is the same person regardless of which folder you hand them. The folder is what changes.
What fine-tuning does
Fine-tuning actually changes the model. You take a pre-trained model and continue training it on a new dataset of your own: question-answer pairs, formatted documents, task-specific examples. Through this additional training, the model’s weights shift to incorporate patterns from your data.
The result is a new version of the model that has internalized certain knowledge or behaviors. It does not need to be given that information in the prompt anymore because it has, in a sense, learned it.
Going back to the consultant analogy: fine-tuning is like hiring that consultant and having them spend six months immersed in your company’s domain before they ever talk to a client. They come out transformed. They think in your domain. They speak your language automatically.
That sounds better. And in some cases it is. The question is whether the transformation is worth what it costs, and whether it actually solves the problem you have.
The real cost of fine-tuning
When developers hear “fine-tuning,” they often imagine a process similar to training a traditional ML model: write some code, run it overnight, done. The reality is more involved, and the costs are higher than most teams anticipate the first time they go through it.
Data preparation
Fine-tuning requires high-quality, labeled training data. For an instruction-following model, this typically means thousands of input-output pairs where the input is a prompt and the output is the ideal response.
Creating this data is the hardest part of fine-tuning for most teams. You need subject matter experts to review and validate the outputs. The data needs to be diverse enough to cover the range of queries your system will face. And it needs to be formatted correctly for the specific model you are fine-tuning. For a legal assistant, this might mean months of lawyers reviewing and annotating examples. That is not a small investment.
Compute cost
Full fine-tuning of a large model is expensive. Fine-tuning GPT-3.5 Turbo via the OpenAI API costs roughly 8 dollars per million training tokens. A modest dataset of 100,000 examples with average prompt and completion lengths might cost a few hundred dollars per fine-tuning run. But you will not get it right the first time. Expect multiple iterations as you adjust your data and hyperparameters.
If you are fine-tuning open-source models yourself on your own hardware, the compute cost becomes a cloud infrastructure cost: A100 or H100 GPU instances running for hours or days. For a 7B parameter model using LoRA (a more efficient fine-tuning technique), a single run might take 4 to 8 hours on an A100. At roughly 3 to 4 dollars per hour, that is manageable. For larger models, the cost scales significantly.
Maintenance overhead
A fine-tuned model is a versioned artifact. When your underlying data changes, your fine-tuned model is stale. You need to re-collect data, re-train, re-evaluate, and re-deploy. For a legal assistant where case law and regulations change constantly, this is a genuine operational burden.
Compare this to RAG: when your documents change, you re-index them. That is a pipeline operation, not a model training job. It takes minutes to hours, not days. And it does not require GPU infrastructure.
Evaluation
How do you know if your fine-tuned model is better? You need a rigorous evaluation suite: a set of held-out test cases with known correct answers that you run against the fine-tuned model and the baseline model to measure the improvement. Building and maintaining that evaluation suite is its own project.
The honest cost summary
A serious fine-tuning project typically requires 2 to 6 months of data preparation, multiple GPU training runs, a rigorous evaluation framework, and ongoing maintenance as your data evolves. For most teams, this is a much larger commitment than initially anticipated. RAG can often be implemented and deployed in days to weeks with dramatically lower ongoing costs.
What fine-tuning is actually good at
Fine-tuning gets a bad reputation in some circles because it is overused. But there are genuine use cases where it is the right tool and RAG cannot match it.
Teaching the model a new style or format
If you need the model to consistently output in a very specific format, follow a particular writing style, or use domain-specific terminology in a precise way, fine-tuning is more reliable than prompting.
For example: a medical documentation assistant that needs to write in SOAP note format (Subjective, Objective, Assessment, Plan) for every single output. You can prompt the model to do this, but a fine-tuned model that has seen thousands of SOAP notes will do it more consistently and with less prompt engineering overhead.
Consistent tone and persona at scale
If you are building a product where the AI has a specific persona, fine-tuning that persona into the model means you do not need to include a long system prompt describing it on every single API call. For applications making millions of API calls per day, the token savings from a shorter system prompt can add up to significant cost reductions.
Classification and structured extraction tasks
For tasks like sentiment classification, named entity extraction, or intent classification where the model needs to output in a very constrained format and you have thousands of labeled examples, a fine-tuned smaller model often outperforms a larger general model on your specific task while being cheaper and faster per request.
A fine-tuned GPT-3.5 Turbo can frequently match or exceed GPT-4 on a narrow, well-defined classification task where you have good training data. That is a meaningful cost and latency advantage at scale.
Teaching genuinely new knowledge the model cannot learn from context
There is a category of knowledge that is difficult to convey through retrieval: procedural knowledge, implicit reasoning patterns, and domain-specific logic that spans multiple documents in complex ways.
For example: training a model to reason about electrical circuit diagrams in a way that requires deep domain knowledge baked into its weights, not just retrieved text. Or training a code completion model on your company’s proprietary codebase so it understands your internal APIs and conventions natively.
What RAG is actually good at
RAG wins in the scenarios that describe the majority of enterprise AI applications being built today.
Dynamic, frequently changing data
If your knowledge base changes regularly, RAG is almost always the right choice. Re-indexing documents is fast and cheap. Re-training a model is slow and expensive. A legal assistant whose case law database updates weekly, a customer support bot whose product documentation changes with every release, a financial research tool that needs yesterday’s earnings reports: all of these are RAG use cases.
When you need to cite your sources
RAG knows exactly which chunks it retrieved to answer a question. It can tell the user: this answer came from page 4 of document X and section 3 of document Y. A fine-tuned model has no idea where its knowledge came from. Its knowledge is baked into its weights in a way that cannot be audited or cited. For any application where traceability matters, RAG is the only viable approach.
Preventing hallucination in factual domains
When you give a model a context document and tell it to answer only from that document, it is significantly harder for the model to hallucinate. It has the answer right in front of it. A fine-tuned model, by contrast, has absorbed knowledge into its weights in a lossy, imprecise way. It can still confuse details, blend similar facts, or confidently produce slightly wrong versions of things it learned during training.
Smaller datasets
Fine-tuning requires substantial labeled training data to work well. If you have hundreds of documents but not thousands of labeled examples, RAG will outperform a fine-tuned model. You can build a RAG system with ten documents. You cannot meaningfully fine-tune a model with ten documents.
Privacy and data security
Fine-tuning on the OpenAI API means sending your training data to OpenAI’s servers. For many enterprise use cases, this is not acceptable. RAG keeps your documents on your own infrastructure. The LLM only ever sees retrieved text snippets at query time, not your entire document corpus.
Side by side: the complete comparison
Here is the full picture in one place:
Data freshness
RAG: Excellent: re-index in minutes
FT: Poor: requires re-training run
Source attribution
RAG: Built in: you know exactly which chunk was used
FT: Not possible: knowledge is baked into weights
Training data required
RAG: None: works with raw documents
FT: Thousands of labeled examples minimum
Hallucination risk
RAG: Lower: model reads from retrieved context
FT: Higher: knowledge is stored lossy in weights
Setup time
RAG: Days to weeks
FT: Months including data preparation
Ongoing maintenance
RAG: Low: update index when docs change
FT: High: retrain when knowledge changes
Cost per update
RAG: Low: indexing is cheap
FT: High: GPU training runs
Output format consistency
RAG: Relies on prompt engineering
FT: Excellent after fine-tuning on examples
Works with private data
RAG: Yes, data stays on your infrastructure
FT: Depends: API fine-tuning sends data to vendor
Inference cost
RAG: Higher: larger prompts with context
FT: Lower: shorter prompts, smaller model possible
Best for
RAG: Dynamic data, factual Q&A, document search
FT: Style consistency, narrow tasks, format adherence
The answer Siddharth should have given
Let us go back to the legal assistant. The VP asked: why not fine-tune on ten years of case files?
Here is the clear, confident answer:
“Fine-tuning on our case files would teach the model the general patterns of legal
reasoning in our domain. That is valuable, but it does not solve our core problem.
Our lawyers need answers to specific questions about specific cases,
clauses, and filings. They need to know which document the answer came from.
They need the information to be current, including cases filed last week.
Fine-tuning cannot give us any of those things. A fine-tuned model cannot
cite its sources. It cannot update when new cases are filed without retraining.
And it will still hallucinate on specific factual queries because that knowledge
is baked into weights in a lossy way.
RAG solves all three of those problems. The model reads from retrieved documents,
so it can cite exactly which filing it used. The index updates in minutes when
new documents arrive. And retrieval grounds the model in actual text,
dramatically reducing hallucination on factual queries.
Fine-tuning would be the right choice if we needed the model to consistently
write in a specific legal document format, or to classify contract clauses
at high throughput. For general legal Q&A over a dynamic corpus,
RAG is the correct architecture.”
That answer is specific, structured, and shows genuine understanding of both approaches. That is what separates a developer who has just heard the terminology from one who actually understands the trade-offs.
Can you use both together?
Yes, and this is actually a common pattern in mature AI products. The two approaches are not mutually exclusive.
A common combination looks like this:
- Fine-tune the model on your domain’s tone, terminology, and output format. This gives you a base model that speaks your language and follows your conventions.
- Use RAG on top of the fine-tuned model to give it access to specific, current, citable information at query time.
For example, a fine-tuned model trained on your company’s communication style combined with a RAG pipeline over your product documentation gives you a chatbot that sounds like it belongs to your brand and knows your product accurately.
This combination is more expensive to set up, but for high-stakes applications where both consistency and accuracy matter, it is the most robust architecture.
A decision framework you can use today
When you are evaluating which approach to use for a new AI feature, go through these questions in order:
- Does the model need access to information that changes over time, or is specific to your organization’s documents? If yes: start with RAG.
- Does the user need to know where the answer came from? If yes: RAG. Fine-tuning cannot provide source attribution.
- Do you have thousands of high-quality labeled examples to train on? If no: RAG. Fine-tuning without sufficient data produces worse results than a well-prompted base model.
- Is the task narrow and consistent, like formatting, classification, or style? And do you have both the data and the budget? If yes: fine-tuning is worth exploring.
- Are you under time pressure to ship? RAG can be built and deployed in days. Fine-tuning takes months. Default to RAG unless you have a specific reason not to.
The default recommendation
If you are building a new AI feature and you are not sure which to choose, build RAG first. It is faster, cheaper, more maintainable, and solves the majority of enterprise AI problems correctly. Only invest in fine-tuning when you have a specific, validated need that RAG cannot meet and the data and budget to do it properly.
A quick word on LoRA and parameter-efficient fine-tuning
If you do decide to fine-tune, you will quickly encounter the terms LoRA, QLoRA, and PEFT. These are techniques that make fine-tuning dramatically cheaper without sacrificing much quality.
LoRA stands for Low-Rank Adaptation. Instead of updating all of a model’s billions of parameters during fine-tuning, LoRA adds a small set of additional parameters and trains only those. The original model weights stay frozen. This reduces the memory required for fine-tuning by a factor of 10 or more, making it possible to fine-tune large models on a single consumer GPU.
QLoRA takes this further by quantizing the frozen base model to 4-bit precision, reducing memory requirements even more. With QLoRA, a 7B parameter model can be fine-tuned on a single 24GB GPU. A 13B model becomes feasible on a 40GB A100.
If you are working with open-source models like Llama 3, Mistral, or Phi, LoRA and QLoRA are the standard approach to fine-tuning. Libraries like Hugging Face’s PEFT and the Unsloth framework make the implementation much more accessible than it was even a year ago.
That said, even with LoRA, the data preparation challenge remains. Efficient training does not solve the problem of needing high-quality labeled examples. That is still the hardest part.
What is coming in Week 8?
Next week is the final post in the series and it is one of the most practically useful. We cover ReAct agents: the architecture that lets an LLM reason through a multi-step problem, decide which tools to use, act on the results, and keep going until it has a complete answer.
This is the architecture behind most of the AI agents you see being built today: coding assistants that run code and fix their own errors, research agents that search the web and synthesize findings, and workflow automation agents that take a goal and figure out the steps themselves.
We will explain how the ReAct loop works, build a simple agent from scratch to make it concrete, and talk about where agents are genuinely useful versus where they add complexity without adding value. It is a fitting end to a series that started with “what is Gen AI” and ends with AI that can think, plan, and act on its own.
Key Takeaway
Fine-tuning and RAG solve different problems. RAG gives the model access to specific, current, citable information at query time. Fine-tuning changes how the model thinks and communicates. Most enterprise AI applications need the first, not the second. When in doubt, build RAG. Add fine-tuning only when you have a validated need, sufficient data, and the time to do it properly.
Follow the full series at codekerdos.in
New post every week. Practical Gen-AI content built for Java and Spring Boot developers. Join our WhatsApp community for early access, Q&A, and hands-on exercises every week.