Probability Distributions Beginner

A probability distribution describes how likely different outcomes are. In ML, distributions model everything: data noise, weight initialization, output predictions, and latent variables. Choosing the right distribution is one of the most important modeling decisions you make.

Key Distributions for ML

Distribution Type Parameters ML Use Case
Gaussian (Normal) Continuous μ (mean), σ (std dev) Weight init, noise modeling, regression
Bernoulli Discrete p (probability) Binary classification, coin flips
Categorical Discrete p1...pk Multi-class classification (softmax)
Uniform Continuous a (min), b (max) Random initialization, sampling
Poisson Discrete λ (rate) Count data, event frequency

The Gaussian Distribution

The most important distribution in ML. The Central Limit Theorem says that sums of many random variables tend toward a Gaussian, which is why it appears everywhere:

Python 
import numpy as np
from scipy import stats

# Gaussian distribution
mu, sigma = 0, 1  # Standard normal

# Sample from it
samples = np.random.normal(mu, sigma, size=1000)

# Probability density function
x = 0.5
pdf = stats.norm.pdf(x, mu, sigma)
print(f"P(X = {x}) density = {pdf:.4f}")

# Weight initialization: Xavier/Glorot
n_in, n_out = 256, 128
weights = np.random.normal(0, np.sqrt(2.0 / (n_in + n_out)), (n_in, n_out))

Discrete Distributions

Python 
# Bernoulli: binary outcomes (spam/not spam)
p = 0.7  # Probability of class 1
samples = np.random.binomial(1, p, size=100)

# Categorical: multi-class (softmax output)
probs = [0.1, 0.3, 0.6]  # 3 classes
samples = np.random.choice([0, 1, 2], size=100, p=probs)

# Multinomial: counts across categories
counts = np.random.multinomial(100, probs)
print("Class counts:", counts)  # e.g., [12, 28, 60]
Softmax Connection: When a neural network outputs class probabilities via softmax, it is defining a Categorical distribution. The cross-entropy loss measures how different this predicted distribution is from the true distribution (one-hot label).

Next Up: Bayes Theorem

Learn how to update probability estimates when new evidence arrives - the foundation of Bayesian machine learning.

Next: Bayes Theorem →
← Introduction Bayes Theorem →

Ready to Go Deeper?

Live instructor-led courses from our partners. Affiliate disclosure.

🧠

AI & ML Courses - 30% Off

Live instructor-led AI, machine learning, data science, and cloud courses for working professionals. Use code Limited30 at checkout.

Edureka
📊

DataCamp - AI & Data Science

Hands-on Python, machine learning, and AI courses with interactive exercises and real projects.

DataCamp
🎓

edX - Top AI Courses

University-level AI courses from MIT, Harvard, Stanford. Earn certificates that employers recognize.

edX