Bayes Theorem Intermediate

Bayes theorem is the fundamental rule for updating beliefs with evidence. It tells us how to compute the probability of a hypothesis given observed data. This simple formula is the foundation of spam filters, medical diagnosis systems, Bayesian neural networks, and much more.

The Formula

P(H|D) = P(D|H) · P(H) / P(D)

  • P(H|D) - Posterior: probability of hypothesis H given data D
  • P(D|H) - Likelihood: probability of observing data D if hypothesis H is true
  • P(H) - Prior: our initial belief about H before seeing data
  • P(D) - Evidence: total probability of the data (normalizing constant)
Python 
# Medical test example
# Disease prevalence: 1%
# Test sensitivity (true positive): 95%
# Test specificity (true negative): 90%

P_disease = 0.01
P_positive_given_disease = 0.95
P_positive_given_no_disease = 0.10

# P(positive) = P(pos|disease)*P(disease) + P(pos|no disease)*P(no disease)
P_positive = (P_positive_given_disease * P_disease +
              P_positive_given_no_disease * (1 - P_disease))

# Bayes theorem: P(disease | positive test)
P_disease_given_positive = (P_positive_given_disease * P_disease) / P_positive
print(f"P(disease | positive) = {P_disease_given_positive:.2%}")
# ~8.7% - surprisingly low due to the low base rate!
Base Rate Fallacy: Even with a 95% accurate test, a positive result only means ~9% chance of disease when the prevalence is 1%. This counterintuitive result highlights why Bayesian reasoning is critical in AI - prior probabilities matter.

Naive Bayes Classifier

The Naive Bayes classifier directly applies Bayes theorem for classification, assuming features are independent given the class:

Python 
from sklearn.naive_bayes import GaussianNB
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split

# Load data
X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)

# Naive Bayes: applies Bayes theorem with Gaussian likelihoods
model = GaussianNB()
model.fit(X_train, y_train)

accuracy = model.score(X_test, y_test)
print(f"Accuracy: {accuracy:.2%}")

# Predict probabilities (posterior distribution over classes)
probs = model.predict_proba(X_test[:1])
print("Class probabilities:", probs)

Bayesian vs Frequentist

Aspect Frequentist Bayesian
Parameters Fixed but unknown Random variables with distributions
Estimation MLE (single point) Posterior distribution
Uncertainty Confidence intervals Credible intervals, full posterior
Prior knowledge Not used Encoded in prior

Next Up: Random Variables

Learn about expectations, variance, and how random variables formalize the concept of uncertainty in ML.

Next: Random Variables →
← Distributions Random Variables →

Ready to Go Deeper?

Live instructor-led courses from our partners. Affiliate disclosure.

🧠

AI & ML Courses - 30% Off

Live instructor-led AI, machine learning, data science, and cloud courses for working professionals. Use code Limited30 at checkout.

Edureka
📊

DataCamp - AI & Data Science

Hands-on Python, machine learning, and AI courses with interactive exercises and real projects.

DataCamp
🎓

edX - Top AI Courses

University-level AI courses from MIT, Harvard, Stanford. Earn certificates that employers recognize.

edX