Fine-tuning vs RAG vs prompting

Pre-training involves training a model on massive unlabelled text data with a self-supervised objective like next-token prediction. This gives the model general language understanding. Fine-tuning then adapts this pre-trained model to specific downstream tasks using smaller labelled datasets, transferring the knowledge learned during pre-training.

Fine-tuning adapts a pre-trained model to a specific task by continuing training on task-specific data. Use it when prompt engineering alone is insufficient, you need consistent output formatting, domain-specific knowledge integration, or significant behaviour changes. It is more expensive than prompting but cheaper than training from scratch.

Transfer learning involves taking a model pre-trained on a large dataset and fine-tuning it on a smaller, task-specific dataset. It reduces training time and data requirements dramatically. Models like BERT and ResNet are commonly used as pre-trained bases for NLP and computer vision tasks.

Prompt engineering crafts inputs to guide a model's existing capabilities without changing weights. Instruction tuning fine-tunes the model on instruction-following data to make it inherently better at following prompts. Instruction-tuned models (ChatGPT, Claude) respond better to natural language prompts compared to base models that require more carefully engineered prompts.

Use RAG when you need access to frequently updated data, require citations, or have a large knowledge base. Use fine-tuning when you need to change the model's behaviour, style, or reasoning patterns, or when knowledge is stable and fits in training data. RAG is cheaper and more flexible; fine-tuning gives better performance for specific formats. Many production systems combine both approaches.

Full fine-tuning updates all model parameters, requiring significant GPU memory and risking catastrophic forgetting. PEFT methods like LoRA, QLoRA, and adapters only modify a small subset of parameters, reducing compute and memory needs by 10-100x while achieving comparable performance. PEFT is preferred for most practical use cases due to its efficiency.

LoRA freezes the pre-trained weights and injects trainable low-rank decomposition matrices into each layer. Instead of updating a full weight matrix W (d x d), it learns two smaller matrices A (d x r) and B (r x d) where r is much smaller than d. This reduces trainable parameters by 100-1000x while maintaining performance. Multiple LoRA adapters can be swapped for different tasks.

Create instruction-response pairs in the model's expected format. Ensure diversity in instructions, response lengths, and complexity. Include edge cases and negative examples. Clean data for quality: remove duplicates, fix formatting, validate factual accuracy. Aim for 1K-100K high-quality examples. Use consistent formatting with clear instruction, input, and output fields.

Catastrophic forgetting occurs when fine-tuning causes the model to lose previously learned general capabilities. Prevent it by using a low learning rate, keeping training epochs low, mixing in general-purpose data, using PEFT methods like LoRA which preserve base weights, applying elastic weight consolidation, and validating on both the target task and general benchmarks.

Key hyperparameters include learning rate (typically 1e-5 to 5e-5), batch size, number of epochs (1-3 for LLMs), warmup ratio, weight decay, and for LoRA: rank (4-64), alpha (typically 2x rank), and target modules. Start with recommended defaults, use a validation set to monitor overfitting, and adjust learning rate first as it has the largest impact.

SFT trains the model to mimic demonstration data by minimising the difference between its outputs and the training examples. RLHF adds a preference optimisation step where a reward model learns human preferences and the LLM is optimised to maximise that reward using reinforcement learning. SFT teaches what to generate; RLHF teaches what humans prefer among options.

Compare against the base model on your task-specific evaluation set using relevant metrics. Check for regression on general capabilities with standard benchmarks. Compute perplexity on a held-out set. Use human evaluation for subjective quality. A/B test in production. Verify the model handles edge cases in your domain. Monitor both task performance and safety metrics.

High-quality data consistently outperforms large quantities of noisy data for fine-tuning. A few thousand expert-curated examples often beat tens of thousands of auto-generated ones. Focus on diverse, representative, and accurately labelled examples. Clean data aggressively. However, some quantity is needed for generalisation. The sweet spot is typically 1K-10K high-quality examples for most tasks.

Create training examples with consistent JSON output format. Include diverse examples covering all possible output structures. Add examples with edge cases (null values, empty arrays, special characters). Validate that training data contains only valid JSON. Fine-tune with a focus on format compliance. Evaluate using both format validity (parse success rate) and content accuracy metrics.

Continual pre-training extends the model's knowledge by training on domain-specific corpora using the same objective as pre-training (next-token prediction on raw text). Fine-tuning adapts the model's behaviour using instruction-response pairs. Continual pre-training adds knowledge; fine-tuning teaches how to use it. For domain adaptation, do continual pre-training first, then instruction fine-tuning.

Load the base model with appropriate quantisation config (BitsAndBytesConfig for QLoRA). Apply LoRA config specifying target modules, rank, and alpha. Prepare the dataset in the instruction format using a tokeniser. Use SFTTrainer from trl library with training arguments (learning rate, epochs, batch size). Monitor training loss and validation metrics. Save and merge LoRA weights after training.

The learning rate schedule controls how the learning rate changes during training. For fine-tuning, a warmup phase (5-10% of steps) gradually increases from zero to prevent large initial weight updates. Followed by cosine or linear decay to allow fine-grained convergence. The peak learning rate for fine-tuning is typically 10-100x lower than pre-training to preserve learned representations.

Use a stronger model (like GPT-4 or Claude) to generate instruction-response pairs. Provide diverse seed prompts covering your domain. Apply quality filters to remove low-quality generations. Validate with domain experts for a subset. Augment with paraphrasing and variation generation. Be careful about model distillation terms of service. Combine synthetic data with real examples for best results.

Overfitting is common with small fine-tuning datasets. Detect it by monitoring training loss vs validation loss divergence, checking for verbatim memorisation of training examples, and testing on held-out data. Prevent it by using fewer epochs (1-3), applying dropout, using weight decay, increasing dataset diversity, and employing early stopping based on validation metrics.

Consider: model size vs available compute, pre-existing knowledge in your domain, licensing restrictions (commercial vs research), language support (especially for Indian languages), community support and tooling, and baseline performance on your task. Start with the smallest model that meets quality requirements. Test 2-3 candidates on your evaluation set before committing to full fine-tuning.

Use the merge_and_unload() method from the PEFT library, which adds the LoRA weight deltas (A * B scaled by alpha/r) to the corresponding base model weights. The merged model has the same architecture as the base model with no inference overhead. Save the merged model for deployment. Verify that the merged model produces identical outputs to the LoRA model before deployment.

For a 7B model: QLoRA needs ~16GB VRAM (single consumer GPU), full fine-tuning needs ~112GB (2x A100). For 13B: QLoRA needs ~24GB, full needs ~208GB. For 70B: QLoRA needs ~48GB (single A100), full needs ~1TB+ (multi-node). Training time scales roughly linearly with data size. Use gradient checkpointing and DeepSpeed ZeRO to reduce memory at the cost of speed.

Fine-tuning updates model weights (all or a subset) using gradient descent on task data. Prompt tuning learns continuous prompt embeddings prepended to the input while keeping all model weights frozen. Prompt tuning uses far fewer parameters (often under 0.1% of model size) and can store multiple task adaptations cheaply, but generally underperforms LoRA for complex tasks.

Use transfer learning with a pre-trained CNN like EfficientNet fine-tuned on labelled crop disease images. The model should handle diverse Indian crops and work offline on low-end devices via model quantisation. Include data augmentation for varying lighting and camera quality. Deploy as a lightweight mobile app with results in local languages.

Reinforcement Learning from Human Feedback trains a reward model on human preferences about response quality, then uses PPO or similar algorithms to fine-tune the LLM to maximise that reward. It aligns model outputs with human values, making responses more helpful, harmless, and honest compared to pure supervised fine-tuning.

Choose a model with strong multilingual capabilities (like GPT-4, Claude, or IndicBERT for embedding). Implement language detection for routing. Use RAG with a knowledge base containing Indian language content. Fine-tune on Indian language conversation data if quality is insufficient. Handle transliteration (Hindi in Roman script) and code-mixing (Hinglish) explicitly.

Distillation trains a smaller student model to mimic the outputs of a larger teacher model, resulting in a genuinely smaller architecture. Quantisation keeps the same architecture but reduces the numerical precision of weights. Distillation requires retraining but can achieve better compression ratios; quantisation is faster to apply but limited in compression.

DPO directly optimises the LLM policy using preference pairs without training a separate reward model, simplifying the RLHF pipeline. It reformulates the reward-based objective as a classification loss on preference data. DPO is simpler to implement and more stable but may not perform as well as RLHF with a well-tuned reward model on complex alignment tasks.

Collect domain-specific query-document pairs from search logs or manually curated examples. Create training triplets (query, positive document, negative document) using hard negative mining. Fine-tune a pre-trained embedding model (like BGE or E5) using contrastive loss. Evaluate on a held-out domain-specific test set using retrieval metrics. This typically yields significant improvement over general embeddings.

QLoRA combines quantisation with LoRA. It loads the base model in 4-bit precision (NormalFloat4) and adds LoRA adapters in higher precision. This reduces memory requirements by approximately 4x compared to LoRA alone, enabling fine-tuning of 65B parameter models on a single 48GB GPU. It uses techniques like double quantisation and paged optimisers for memory efficiency.

Collect Indian legal documents (court judgments, statutes, legal opinions) and create instruction pairs for tasks like summarisation, clause extraction, and case law reference. Include Indian legal terminology and citation formats. Use QLoRA on a capable base model. Validate against expert-labelled outputs. Handle multiple Indian languages. Ensure the model correctly references Indian statutes and precedents.

Collect domain-specific query-passage pairs from search logs or expert annotation. Create training triplets with hard negatives (similar but irrelevant passages). Fine-tune using contrastive learning (InfoNCE or multiple negatives ranking loss). Evaluate with retrieval metrics on a held-out test set. Use a pre-trained model like BGE or E5 as the base. Typically 5K-50K pairs suffice.

Adapter layers insert small bottleneck modules between existing layers of a frozen model. They consist of a down-projection, non-linearity, and up-projection. Compared to LoRA, adapters add sequential computation (slightly increasing latency) while LoRA modifies existing weights in parallel. LoRA has become more popular due to zero additional inference latency and easier merging with base weights.

Combine datasets from multiple tasks with task-specific prefixes or instruction formats. Balance dataset sizes to prevent dominant tasks from overwhelming others. Use weighted sampling or data mixing ratios. Monitor performance on each task separately. Consider using task-specific LoRA adapters that can be loaded independently. Evaluate for cross-task transfer and interference.

RLAIF (RL from AI Feedback) replaces human feedback with AI-generated feedback based on a constitution (set of principles). An AI model critiques and revises outputs according to these principles, generating preference data. This preference data trains a reward model or directly optimises the policy. It is more scalable than human feedback while maintaining alignment quality.

Knowledge distillation trains a smaller student model to mimic a larger teacher model's output distributions, not just hard labels. The student learns the teacher's soft probabilities, which contain richer information. It is used to create efficient production models. Combined with fine-tuning, you can distill a large fine-tuned model into a smaller, deployable one with minimal quality loss.

Use on-premise or private cloud GPU infrastructure. Apply differential privacy during training to prevent memorisation. Anonymise or pseudonymise PII in training data. Use federated fine-tuning when data cannot leave its source. Implement access controls on model artifacts. Audit the model for data leakage using membership inference tests. Comply with India's DPDP Act requirements.

DPO simplifies preference-based fine-tuning by eliminating the need for a separate reward model. It directly optimises the policy model using pairs of preferred and rejected responses. The loss function implicitly defines a reward and trains the model to increase the probability of preferred responses while decreasing rejected ones. It is simpler and more stable than PPO-based RLHF.

Prefix tuning prepends learnable continuous vectors (soft prompts) to the input at each layer while keeping model weights frozen. These prefixes steer the model's behaviour without modifying its parameters. Compared to LoRA, prefix tuning has fewer parameters but slightly higher inference latency due to added prefix tokens. LoRA generally achieves better performance and has become the preferred PEFT method.

Collect Indian documents (Aadhaar, PAN cards, invoices in multiple scripts) with annotated fields. Use a pre-trained VLM like LLaVA or PaLI as the base. Create instruction data for OCR, field extraction, and document classification. Handle multiple Indian scripts (Devanagari, Tamil, Bengali, etc.). Fine-tune with LoRA on vision and language components. Evaluate on diverse document formats and quality levels.