Entropy Intermediate

Entropy is the central concept of information theory. It measures the average amount of uncertainty or surprise in a random variable. In AI, entropy tells us how "spread out" or "uncertain" a probability distribution is.

Shannon Entropy Formula

For a discrete random variable X with possible outcomes x1, x2, ..., xn and probability distribution P:

H(X) = -SUM[ P(xi) * log2(P(xi)) ]  for all i

# Equivalently:
H(X) = E[-log2(P(X))]  # Expected value of self-information

Entropy Examples

Computing Entropy in Python

import numpy as np
from scipy.stats import entropy

def shannon_entropy(probs):
    """Compute Shannon entropy in bits."""
    probs = np.array(probs)
    probs = probs[probs > 0]  # Avoid log(0)
    return -np.sum(probs * np.log2(probs))

# Fair coin
print(shannon_entropy([0.5, 0.5]))      # 1.0 bits

# Biased coin
print(shannon_entropy([0.9, 0.1]))      # 0.469 bits

# Confident model prediction
print(shannon_entropy([0.95, 0.03, 0.02]))  # 0.335 bits

# Uncertain model prediction
print(shannon_entropy([0.33, 0.33, 0.34]))  # 1.585 bits

# Using scipy (uses natural log by default, pass base=2 for bits)
print(entropy([0.5, 0.5], base=2))       # 1.0 bits

Entropy in AI Applications

Model Confidence

A classifier's softmax output is a probability distribution. Its entropy tells you how confident the model is:

• Low entropy: Model is confident (probability concentrated on one class)
• High entropy: Model is uncertain (probability spread across classes)

Decision Trees

Information gain (reduction in entropy) is used to select the best feature to split on at each node of a decision tree. The feature that reduces entropy the most gives the most information.

Maximum Entropy Principle

When you have incomplete information about a distribution, the maximum entropy principle says to choose the distribution with the highest entropy that is consistent with your constraints. This avoids adding assumptions you do not have evidence for.