Best Practices Intermediate

Working with probabilities in code requires careful attention to numerical precision. Probabilities are often very small numbers that can underflow to zero, and naive implementations can produce incorrect results. This lesson covers the practical wisdom for robust probabilistic ML.

Work in Log-Space

import numpy as np

# BAD: multiplying small probabilities
probs = [1e-100, 1e-200, 1e-150]
product = np.prod(probs)  # 0.0 (underflow!)

# GOOD: add log-probabilities
log_probs = np.log(probs)
log_product = np.sum(log_probs)  # -1035.0 (correct!)

# LogSumExp trick for stable softmax
def log_sum_exp(x):
    c = np.max(x)
    return c + np.log(np.sum(np.exp(x - c)))

def stable_softmax(x):
    x_shifted = x - np.max(x)  # Prevent overflow
    exp_x = np.exp(x_shifted)
    return exp_x / np.sum(exp_x)

Practical Tips

1. Add epsilon to prevent log(0)

   Always use np.log(p + 1e-8) instead of np.log(p) to avoid -inf values.

2. Use framework-provided functions

   PyTorch's F.cross_entropy and F.log_softmax are numerically stable. Do not implement your own unless necessary.

3. Set random seeds for reproducibility

   Use np.random.seed(42) and torch.manual_seed(42) for reproducible experiments.

4. Validate probability distributions

   Check that probabilities sum to 1 (discrete) or integrate to 1 (continuous). Off-by-one errors in normalization are common.

Common Mistakes

Useful Libraries