Fine-tuned Models

Adapt pre-trained AI models to your specific domain, tasks, and style - from LoRA to RLHF, learn when and how to fine-tune effectively.

What Is Fine-tuning?

Fine-tuning is the process of taking a pre-trained model (one that has already learned general patterns from massive datasets) and further training it on a smaller, task-specific dataset. This allows the model to specialize in a particular domain, adopt a consistent output style, or improve performance on targeted tasks - all without the enormous cost of training from scratch.

Think of it like this: a pre-trained model is a university graduate with broad knowledge. Fine-tuning is the on-the-job training that makes them an expert in your specific field.

Fine-tuning vs Alternatives

Before committing to fine-tuning, understand how it compares to other customization approaches:

Types of Fine-tuning

1. Full Fine-tuning

Updates all parameters in the model. This produces the highest-quality results but requires substantial GPU memory (typically 2-3x the model size in VRAM) and large datasets. Full fine-tuning is generally reserved for organizations with significant compute budgets.

• Pros: Maximum quality, full control over model behavior
• Cons: Expensive, risk of catastrophic forgetting, slow training
• Use case: Building a specialized foundation model (e.g., BloombergGPT for finance)

2. Parameter-Efficient Fine-Tuning (PEFT)

PEFT methods update only a small fraction of the model's parameters while keeping the rest frozen. This dramatically reduces compute requirements while achieving results close to full fine-tuning.

LoRA (Low-Rank Adaptation)

LoRA freezes the pre-trained weights and injects small low-rank decomposition matrices into Transformer layers. Instead of updating a full weight matrix W (d × d), LoRA learns two smaller matrices A (d × r) and B (r × d) where the rank r is much smaller than d (typically 8-64).

• Updates only ~0.1-1% of parameters
• LoRA adapters are small files (10-100 MB) that can be swapped at inference time
• Multiple LoRA adapters can serve different tasks from the same base model

QLoRA (Quantized LoRA)

QLoRA combines LoRA with 4-bit quantization of the base model, enabling fine-tuning of 65B+ parameter models on a single 48GB GPU. It uses a novel NormalFloat4 data type and double quantization to minimize memory while maintaining quality.

Adapters

Adapter layers are small bottleneck modules inserted between existing Transformer layers. Each adapter typically has a down-projection, a nonlinearity, and an up-projection. They update ~1-5% of total parameters.

Prefix Tuning

Prefix tuning prepends a set of trainable continuous vectors (the "prefix") to the input at every Transformer layer. Only these prefix vectors are updated during training, leaving all model weights frozen.

3. Instruction Tuning

Instruction tuning trains models to follow natural language instructions by fine-tuning on datasets of (instruction, response) pairs. This is what transforms a raw language model into a helpful assistant.

• FLAN: Google's approach using 1,800+ tasks described via instructions
• InstructGPT: OpenAI's method combining supervised fine-tuning with human feedback
• Alpaca/Vicuna: Open-source instruction tuning using GPT-generated instruction data

4. RLHF (Reinforcement Learning from Human Feedback)

RLHF is a multi-step process that aligns models with human preferences:

1. Supervised Fine-Tuning (SFT): Train on high-quality demonstrations
2. Reward Model Training: Train a separate model to predict human preferences from comparison data
3. PPO Optimization: Use the reward model to guide RL training of the main model via Proximal Policy Optimization

RLHF is how ChatGPT, Claude, and other assistant models learn to be helpful, harmless, and honest. It requires significant expertise and infrastructure to implement well.

5. DPO (Direct Preference Optimization)

DPO simplifies RLHF by eliminating the separate reward model. Instead, it directly optimizes the language model using preference pairs (chosen vs rejected responses). DPO is mathematically equivalent to a specific form of RLHF but is much simpler to implement and more stable to train.

When to Fine-tune

Fine-tuning is the right choice when you need:

• Domain-specific terminology: Medical, legal, or financial language that the base model handles poorly
• Consistent output style: A specific tone, format, or voice across all responses
• Format compliance: Reliable JSON output, structured reports, or specific templates
• Performance improvement: Higher accuracy on specific tasks like classification or extraction
• Latency reduction: A smaller fine-tuned model can replace a larger general model for specific tasks
• Cost reduction: Fine-tuned smaller models can match larger model performance at lower inference cost

Data Requirements

Step-by-Step Fine-tuning Workflow

1. Define the objective: What specific behavior should improve? How will you measure success?
2. Collect and prepare data: Gather high-quality examples in the required format (instruction/response pairs, preference pairs, etc.)
3. Choose a base model: Select a pre-trained model appropriate for your task size and requirements
4. Select a fine-tuning method: Full, LoRA, QLoRA, or other PEFT method based on your compute budget
5. Configure hyperparameters: Learning rate (typically 1e-5 to 5e-5), batch size, epochs (1-3 for LLMs), LoRA rank
6. Train and monitor: Track loss curves, watch for overfitting, use validation set
7. Evaluate: Test on held-out data, run benchmarks, compare to base model
8. Deploy: Merge adapter weights or serve with adapter, monitor production performance

Tools and Platforms

Code Example: LoRA Fine-tuning with PEFT

import torch
from datasets import load_dataset
from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    TrainingArguments,
    BitsAndBytesConfig,
)
from peft import LoraConfig, get_peft_model, TaskType
from trl import SFTTrainer

# 1. Load base model with 4-bit quantization (QLoRA)
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

model_name = "meta-llama/Meta-Llama-3.1-8B-Instruct"
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    quantization_config=bnb_config,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained(model_name)
tokenizer.pad_token = tokenizer.eos_token

# 2. Configure LoRA
lora_config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=16,                    # Rank: lower = fewer params, higher = more capacity
    lora_alpha=32,           # Scaling factor (alpha / r = scaling)
    lora_dropout=0.05,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    bias="none",
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# Output: trainable params: 41,943,040 || all params: 8,072,204,288 || 0.52%

# 3. Prepare dataset (instruction format)
dataset = load_dataset("json", data_files="training_data.jsonl", split="train")
# Expected format: {"instruction": "...", "input": "...", "output": "..."}

def format_instruction(example):
    return f"""<|begin_of_text|><|start_header_id|>user<|end_header_id|>

{example['instruction']}
{example.get('input', '')}
<|eot_id|><|start_header_id|>assistant<|end_header_id|>

{example['output']}<|eot_id|>"""

# 4. Training arguments
training_args = TrainingArguments(
    output_dir="./lora-finetuned-model",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    learning_rate=2e-4,
    weight_decay=0.01,
    warmup_ratio=0.03,
    lr_scheduler_type="cosine",
    logging_steps=10,
    save_strategy="epoch",
    bf16=True,
    optim="paged_adamw_8bit",
)

# 5. Train with SFTTrainer
trainer = SFTTrainer(
    model=model,
    args=training_args,
    train_dataset=dataset,
    formatting_func=format_instruction,
    max_seq_length=2048,
    tokenizer=tokenizer,
)

trainer.train()

# 6. Save the LoRA adapter (small file, ~50MB)
trainer.model.save_pretrained("./lora-adapter")
# To use: load base model + adapter at inference time

Cost Considerations

Evaluation and Preventing Catastrophic Forgetting

Catastrophic forgetting occurs when fine-tuning causes the model to lose its general capabilities while learning the new task. This is one of the biggest risks of fine-tuning.

Prevention Strategies

• Use PEFT methods: LoRA and QLoRA inherently reduce forgetting by keeping most weights frozen
• Low learning rates: Use 1e-5 to 5e-5 for full fine-tuning to preserve existing knowledge
• Short training: 1-3 epochs is usually sufficient; more risks overfitting and forgetting
• Mix in general data: Include some general-purpose examples alongside your specialized data
• Evaluate broadly: Test on both your target task and general benchmarks (MMLU, HellaSwag, etc.)
• Early stopping: Monitor validation loss and stop when it begins to increase

Evaluation Checklist

# Task-specific metrics
- Accuracy / F1 on target task (held-out test set)
- Format compliance rate (does output match expected structure?)
- Domain terminology accuracy
- Human preference ratings (A/B testing vs base model)

# General capability preservation
- MMLU score (general knowledge)
- HumanEval (code generation)
- MT-Bench (conversation quality)
- Perplexity on general text

# Production metrics
- Inference latency (should not increase with LoRA merge)
- Token throughput
- Error rate in production