Best Practices Intermediate

Training ML models is as much art as science. These best practices distill years of collective experience from the ML community into actionable guidelines. Following these recipes will save you countless hours of debugging and help you achieve better results faster.

Training Recipe

1. Start simple, then scale

   Begin with a small model on a small dataset. Verify it can overfit a single batch (loss goes to near zero). This confirms your pipeline is correct.

2. Use standard defaults first

   AdamW with lr=3e-4, batch size 32-64, no fancy scheduling. Get a baseline before experimenting.

3. Monitor everything

   Track training loss, validation loss, gradient norms, learning rate, and parameter norms. Use W&B or TensorBoard.

4. Add complexity gradually

   Learning rate schedules, data augmentation, regularization - add one at a time and measure the impact.

Debugging Optimization

Essential Techniques

Quick Reference

import torch
import torch.nn as nn

# Standard training loop with best practices
model = MyModel()
optimizer = torch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=0.01)
scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=num_epochs)

best_val_loss = float('inf')
patience_counter = 0

for epoch in range(num_epochs):
    model.train()
    for batch in train_loader:
        optimizer.zero_grad()
        loss = criterion(model(batch.x), batch.y)
        loss.backward()
        torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
        optimizer.step()

    scheduler.step()

    # Validation + early stopping
    val_loss = evaluate(model, val_loader)
    if val_loss < best_val_loss:
        best_val_loss = val_loss
        torch.save(model.state_dict(), 'best_model.pt')
        patience_counter = 0
    else:
        patience_counter += 1
        if patience_counter >= 10:
            break  # Early stopping