RAG vs Fine-Tuning

Last reviewed: 2026-07-08 · By Victor Humenhuk (AIGP certified)

Retrieval-augmented generation leaves a model's weights untouched and instead retrieves relevant documents at query time and places them in the prompt. Fine-tuning changes the model itself, training a pre-trained model further on a smaller, task-specific dataset so that the behaviour is baked into the weights. RAG is the better default where knowledge changes often, must be cited, or contains personal data you may have to correct or erase, while fine-tuning is the better choice where you need a consistent style, format or domain behaviour that prompting cannot reliably achieve. From a governance perspective the decisive difference is reversibility, because you can delete a document from a retrieval index but you cannot cleanly delete a fact from a set of weights.

RAG vs fine-tuning at a glance

	RAG	Fine-tuning
What changes	The prompt, through context retrieved at query time	The model weights, or adapter layers
Knowledge freshness	Immediate, by updating the index	Requires a further training run
Source attribution	Can return citations to retrieved documents	Cannot reliably attribute an output to a source
Removing a fact	Delete it from the index	Retrain or attempt unlearning, and never cleanly
Cost profile	Retrieval infrastructure, added latency and larger per-query token counts	Compute up front, then cheaper and shorter prompts per query
Best for	Changing facts, private document corpora, traceability, sourced answers	Tone, format, structured output, domain-specific behaviour, latency
Main failure modes	Retrieval misses, stale or poisoned index, prompt injection through retrieved content, access-control leakage	Memorisation and extraction, overfitting to a small set, catastrophic forgetting, drift when the base model updates
Data protection profile	Source data stays in a governed store, so erasure and rectification are workable	Data is absorbed into weights, so erasure and rectification are hard

When should you choose which?

Work up the ladder rather than starting at the top.

Prompt engineering first. It is cheap, instantly reversible, and solves more problems than teams expect. Exhaust it before adding infrastructure.
Choose RAG when the answer depends on facts that change, on internal documents the base model has never seen, or on being able to show where an answer came from. Regulatory question-and-answer tools, customer support over a product catalogue and internal policy assistants are all RAG problems.
Choose fine-tuning when the problem is behaviour rather than knowledge: a house style, a rigid output schema, a classification task with plenty of labelled examples, or a domain vocabulary the base model handles badly. Fine-tuning also shortens prompts, which can matter at scale.
Combine them when you need both: fine-tune for the behaviour, retrieve for the facts. This is a common production pattern, and it means you inherit both risk profiles.

See GenAI choices and the pre-launch checklist for how this decision sits inside a release process.

What changes for privacy and data protection?

The choice has direct consequences under the GDPR.

Erasure and rectification. With RAG, a deletion request is a database operation on the index and the source store. With fine-tuning, the personal data has been used to shape parameters, and machine unlearning is an active research area rather than a control you can promise a regulator.
Lawful basis and purpose limitation. Fine-tuning on customer data is a fresh processing purpose that needs its own basis and a compatibility assessment. Retrieval over documents you already hold for a defined purpose is usually easier to justify, though not automatically compatible.
Model status. The EDPB's Opinion 28/2024 takes the position that a model trained on personal data is not automatically anonymous. A fine-tuned model can therefore itself be in scope, and a retrieval index plainly is.
Access control. The most common RAG incident is not exotic. It is a retrieval layer that ignores the permissions on the underlying documents and surfaces HR or legal files to whoever asks. Enforce authorisation at retrieval time, per user, not at the corpus level.
Retention. Prompts, retrieved chunks and outputs are all logs that can contain personal data. Set retention for them deliberately rather than inheriting a vendor default.

What changes under the EU AI Act?

Two provisions matter most.

Article 25. If you modify the intended purpose of an AI system, including a general-purpose AI system, so that it becomes high-risk, you become the provider of that high-risk system with the full Chapter III load. Fine-tuning a general model to screen job applications is the textbook example, and so is building a RAG assistant that materially drives eligibility decisions for a public benefit. Substantial modification of an existing high-risk system has the same effect.

General-purpose AI models. A downstream actor that modifies a general-purpose AI model may become the provider of a modified model, with the Article 53 duties attached: technical documentation, information for downstream providers, a copyright policy and a sufficiently detailed summary of the content used for training. The Commission's July 2025 guidelines on general-purpose AI models set out an indicative, compute-based criterion for when a downstream modification crosses that line. Treat that as guidance on a moving target and check the current position rather than relying on a remembered figure.

Beyond those, Article 50 transparency duties apply to the deployed system regardless of which technique you used, Article 12 logging and Article 13 instructions for use have to reflect what you actually built, and your technical documentation must describe the retrieval corpus or the fine-tuning dataset, because a reviewer cannot assess data provenance without it.

What controls should you put around each?

For RAG:

Enforce user-level authorisation at retrieval, and test it with an account that should see nothing.
Treat retrieved content as untrusted input, because it is the main prompt-injection vector, and constrain what the model is permitted to do with it.
Monitor for index poisoning and stale content, and log which chunks informed which answer so you can reconstruct an output later.
Measure retrieval quality separately from generation quality. Many reported hallucination incidents are retrieval failures.

For fine-tuning:

Document the dataset: sources, licences, consent or legal basis, and lineage. Record it in a model card.
Test for memorisation and extraction before release, and re-test after each training run.
Guard against data poisoning in the training set, particularly where examples come from users or third parties.
Version and pin the base model. A silent upstream update can undo your evaluation results.
Keep a rollback path. If the fine-tune degrades robustness or fairness, you need to be able to revert quickly.

Related study notes

Frequently asked questions

Which is cheaper, RAG or fine-tuning?

It depends on where the cost lands. RAG avoids training compute but adds retrieval infrastructure, latency and longer prompts, so per-query cost is higher. Fine-tuning costs compute up front and then reduces prompt length and per-query cost, which usually wins at high volume with stable requirements.

Can you use RAG and fine-tuning together?

Yes, and it is a common production pattern: fine-tune the model for tone, format and task behaviour, then retrieve for the facts. Just note that you take on both risk profiles and both sets of documentation duties.

Does fine-tuning make me a provider under the EU AI Act?

It can. Under Article 25 you become the provider where you modify the intended purpose of a system so that it becomes high-risk, or make a substantial modification to an existing high-risk system. Separately, modifying a general-purpose AI model can make you the provider of a modified model with its own Article 53 obligations. The answer turns on what you changed and what you use it for, not on the technique alone.

Does RAG stop hallucinations?

It reduces them by grounding answers in retrieved text, but it does not eliminate them. The model can still misread a retrieved passage, blend sources, or answer confidently when retrieval returned nothing useful. Grounding checks, citation verification and an explicit abstain path are all still needed.

Test yourself

Try the free AIGP practice questions, or read the full AIGP study guide - free.