🎛️

Advanced

Fine-tuning vs RAG vs prompting

The decision most people get wrong — you probably do not need to fine-tune

What fine-tuning is

Fine-tuning takes a pre-trained model and trains it further on your own data. This adjusts the model's weights so it behaves differently — it might adopt a specific writing style, learn domain terminology, follow a particular output format, or perform a specialised task better than the base model.

Fine-tuning is powerful but expensive, slow and hard to maintain. Before you fine-tune, you should exhaust prompting and RAG first — they solve most problems without touching the model.

When to use prompting, RAG or fine-tuning

This is the most important decision in applied AI engineering, and most people get it wrong by jumping to fine-tuning too early.

Prompting: when you need to change the model's behaviour or output format. Try this first — always
RAG: when you need the model to answer from specific knowledge it was not trained on
Fine-tuning: when you need to change the model's fundamental capabilities, style or domain expertise, and prompting cannot achieve it
Rule of thumb: if you can show the model what you want through examples in the prompt, you do not need fine-tuning

When fine-tuning makes sense

Fine-tuning makes sense in a few specific situations. When you need a very specific output style that few-shot examples cannot capture consistently. When you need to reduce latency by moving instructions from a long prompt into the model's weights. When you have a classification or extraction task with thousands of labelled examples and need near-perfect accuracy. When you need a smaller, cheaper model to perform like a larger one on a narrow task.

In practice, most teams that fine-tune do it for style/format consistency and latency, not for knowledge injection (that is what RAG is for).

How fine-tuning works

You prepare a dataset of input-output examples in the format you want (typically hundreds to thousands of examples). You upload this to the model provider's fine-tuning API (OpenAI, Anthropic and Google all offer this). The provider runs training for a few epochs and gives you a custom model endpoint.

Full fine-tuning adjusts all model weights and is very expensive. LoRA (Low-Rank Adaptation) adjusts a small subset of weights and is much cheaper — this is what most people use. QLoRA adds quantisation to make it even more memory-efficient.

Full fine-tuning: adjusts all weights, expensive, needs many GPUs
LoRA: adjusts a small adapter layer, much cheaper
QLoRA: LoRA with quantised weights, runs on consumer GPUs
Instruction tuning: fine-tuning on instruction-following examples

The real cost of fine-tuning

Fine-tuning is not a one-time cost. Every time the base model updates, you need to re-evaluate whether your fine-tune still works. Your training data needs to be maintained and updated. You need an eval pipeline to detect regressions. And you are now maintaining a custom model instead of using a standard API.

This operational burden is why most startups and mid-size companies should prefer prompting and RAG. Fine-tuning is justified when the performance gain is large enough to outweigh the maintenance cost.

Key concepts

LoRA

Efficient fine-tuning by adjusting a small adapter layer

QLoRA

LoRA with quantisation — fine-tune on consumer hardware

Instruction tuning

Teaching a model to follow specific instructions

Data preparation

Creating high-quality training datasets

Transfer learning

Leveraging pre-trained knowledge for new tasks

Catastrophic forgetting

When fine-tuning makes the model forget general abilities

Tools & resources

🛠

OpenAI Fine-tuning API

Fine-tune GPT models via API

🛠

Hugging Face PEFT

Library for parameter-efficient fine-tuning

🛠

Axolotl

Easy-to-use fine-tuning framework for open models

🛠

Unsloth

Faster LoRA fine-tuning with lower memory usage

Learning path

1
First, exhaust prompting and RAG for your use case
2
If those are not enough, define exactly what fine-tuning would improve
3
Prepare a high-quality dataset (at least 200 examples)
4
Start with LoRA on a small model to validate the approach
5
Build an eval set before fine-tuning so you can measure the improvement
6
Fine-tune, evaluate, iterate on the dataset, repeat

📝 Interview questions for this topic 🧭 Which roles use this?

What fine-tuning is

Fine-tuning is powerful but expensive, slow and hard to maintain. Before you fine-tune, you should exhaust prompting and RAG first — they solve most problems without touching the model.

When to use prompting, RAG or fine-tuning

This is the most important decision in applied AI engineering, and most people get it wrong by jumping to fine-tuning too early.

Prompting: when you need to change the model's behaviour or output format. Try this first — always

RAG: when you need the model to answer from specific knowledge it was not trained on

Fine-tuning: when you need to change the model's fundamental capabilities, style or domain expertise, and prompting cannot achieve it

Rule of thumb: if you can show the model what you want through examples in the prompt, you do not need fine-tuning

When fine-tuning makes sense

In practice, most teams that fine-tune do it for style/format consistency and latency, not for knowledge injection (that is what RAG is for).

How fine-tuning works

Full fine-tuning: adjusts all weights, expensive, needs many GPUs

LoRA: adjusts a small adapter layer, much cheaper

QLoRA: LoRA with quantised weights, runs on consumer GPUs

Instruction tuning: fine-tuning on instruction-following examples