MLE & MAP Estimation Advanced

Maximum Likelihood Estimation (MLE) and Maximum A Posteriori (MAP) are the two fundamental approaches to learning model parameters from data. MLE finds parameters that make the observed data most probable, while MAP additionally incorporates prior beliefs. Understanding these connects probability theory directly to model training.

Maximum Likelihood Estimation (MLE)

MLE asks: "What parameter values make my observed data most likely?" It maximizes the likelihood function P(data | parameters).

Python

import numpy as np
from scipy.optimize import minimize

# MLE for Gaussian: estimate mean and variance from data
data = np.random.normal(loc=5.0, scale=2.0, size=100)

# Analytical MLE solution
mu_mle = np.mean(data)       # MLE for mean
sigma2_mle = np.var(data)    # MLE for variance
print(f"MLE: mu={mu_mle:.2f}, sigma^2={sigma2_mle:.2f}")

# Negative log-likelihood (what we minimize)
def neg_log_likelihood(params, data):
    mu, log_sigma = params
    sigma = np.exp(log_sigma)
    n = len(data)
    nll = n/2 * np.log(2*np.pi) + n * log_sigma + np.sum((data - mu)**2) / (2*sigma**2)
    return nll

result = minimize(neg_log_likelihood, [0, 0], args=(data,))
print("Optimized mu:", result.x[0])

Key Connection: Training a neural network with cross-entropy loss is equivalent to MLE. Minimizing cross-entropy = maximizing the log-likelihood of the data under the model's predicted distribution.

Maximum A Posteriori (MAP)

MAP adds a prior distribution to MLE. It finds parameters that maximize P(parameters | data) ∝ P(data | parameters) · P(parameters):

Python

# MAP = MLE + Prior (regularization!)
# Gaussian prior on weights = L2 regularization
# Laplace prior on weights = L1 regularization

def map_loss(w, X, y, lambda_reg=0.01):
    # Negative log-likelihood (MSE for Gaussian)
    nll = np.mean((y - X @ w) ** 2)
    # Negative log-prior (L2 = Gaussian prior)
    prior = lambda_reg * np.sum(w ** 2)
    return nll + prior  # MAP objective

MLE vs MAP

Aspect	MLE	MAP
Objective	Maximize P(data \| θ)	Maximize P(θ \| data)
Prior	No prior (or uniform prior)	Incorporates prior P(θ)
Overfitting	More prone to overfitting	Prior acts as regularization
Small data	Unreliable estimates	Prior stabilizes estimates
Equivalent to	Unregularized training	L1/L2 regularized training

Next Up: Best Practices

Learn practical tips for working with probabilities in ML systems, including numerical stability and common pitfalls.

Next: Best Practices →

← Random Variables Best Practices →

Ready to Go Deeper?

Live instructor-led courses from our partners. Affiliate disclosure.

🧠

AI & ML Courses - 30% Off

Live instructor-led AI, machine learning, data science, and cloud courses for working professionals. Use code Limited30 at checkout.

Edureka

📊

DataCamp - AI & Data Science

Hands-on Python, machine learning, and AI courses with interactive exercises and real projects.

DataCamp

🎓

edX - Top AI Courses

University-level AI courses from MIT, Harvard, Stanford. Earn certificates that employers recognize.

edX