Eigenvalues & Eigenvectors Intermediate

Eigenvalues and eigenvectors reveal the fundamental structure of data transformations. They tell you which directions in your data carry the most information, making them essential for dimensionality reduction (PCA), spectral clustering, and understanding how neural networks transform data.

The Core Idea

When a matrix A multiplies a special vector v, the result is just a scaled version of v. That vector is an eigenvector, and the scaling factor is the eigenvalue:

A v = λ v

Python 
import numpy as np

A = np.array([[4, 1],
              [2, 3]])

# Compute eigenvalues and eigenvectors
eigenvalues, eigenvectors = np.linalg.eig(A)

print("Eigenvalues:", eigenvalues)    # [5, 2]
print("Eigenvectors:\n", eigenvectors)

# Verify: A @ v = lambda * v
v = eigenvectors[:, 0]
lam = eigenvalues[0]
print(np.allclose(A @ v, lam * v))  # True
Intuition: Think of a matrix as a transformation. Most vectors change direction when transformed. Eigenvectors are the special directions that stay the same - they only stretch or shrink by the eigenvalue factor.

PCA: The Killer Application

Principal Component Analysis (PCA) uses eigendecomposition of the covariance matrix to find the directions of maximum variance in data:

Python 
# PCA from scratch using eigendecomposition
X = np.random.randn(100, 5)  # 100 samples, 5 features

# Step 1: Center the data
X_centered = X - X.mean(axis=0)

# Step 2: Compute covariance matrix
cov = (X_centered.T @ X_centered) / (len(X) - 1)

# Step 3: Eigendecomposition
eigenvalues, eigenvectors = np.linalg.eigh(cov)

# Step 4: Sort by eigenvalue (largest first)
idx = eigenvalues.argsort()[::-1]
eigenvalues = eigenvalues[idx]
eigenvectors = eigenvectors[:, idx]

# Step 5: Project to 2D
X_2d = X_centered @ eigenvectors[:, :2]
print(X_2d.shape)  # (100, 2) - reduced from 5D to 2D

Eigenvalues in ML

Application What Eigenvalues Tell You
PCA How much variance each principal component captures
Spectral Clustering The number of clusters (via eigenvalue gaps)
PageRank The dominant eigenvector gives page importance scores
Stability Analysis Whether a system (or RNN) is stable or exploding

Next Up: SVD

Singular Value Decomposition generalizes eigendecomposition to non-square matrices, making it even more powerful for ML applications.

Next: SVD →
← Matrices SVD →

Ready to Go Deeper?

Live instructor-led courses from our partners. Affiliate disclosure.

🧠

AI & ML Courses - 30% Off

Live instructor-led AI, machine learning, data science, and cloud courses for working professionals. Use code Limited30 at checkout.

Edureka
📊

DataCamp - AI & Data Science

Hands-on Python, machine learning, and AI courses with interactive exercises and real projects.

DataCamp
🎓

edX - Top AI Courses

University-level AI courses from MIT, Harvard, Stanford. Earn certificates that employers recognize.

edX