Beginner

Linear Regression Deep Dive

The most fundamental machine learning algorithm - learn how to model the relationship between variables using a straight line, and why it remains essential in modern ML.

The Core Idea

Linear Regression finds the best-fitting straight line through your data points. Given input features, it predicts a continuous output value by computing a weighted sum of the inputs plus a bias term.

The Math: y = mx + b

For simple linear regression (one feature), the model is:

y = mx + b

Where:
  y = predicted value (dependent variable)
  x = input feature (independent variable)
  m = slope (weight/coefficient)
  b = y-intercept (bias)

For multiple linear regression (multiple features), the model generalizes to:

y = w1*x1 + w2*x2 + ... + wn*xn + b

In vector notation:
  y = W^T * X + b

Where:
  W = weight vector [w1, w2, ..., wn]
  X = feature vector [x1, x2, ..., xn]
  b = bias (intercept)

Cost Function (Mean Squared Error)

The model learns by minimizing the Mean Squared Error (MSE) - the average of the squared differences between predicted and actual values:

MSE = (1/n) * SUM[(y_actual - y_predicted)^2]
    = (1/n) * SUM[(y_i - (W^T * x_i + b))^2]

Why squared?
  1. Penalizes large errors more than small ones
  2. Makes the function differentiable (smooth)
  3. Always positive (no cancellation of errors)

Gradient Descent

Gradient descent is the optimization algorithm used to find the weights that minimize the cost function. Think of it as a ball rolling downhill to find the lowest point:

# Gradient Descent Algorithm
# Repeat until convergence:

w = w - learning_rate * dMSE/dw
b = b - learning_rate * dMSE/db

# The gradients (partial derivatives):
dMSE/dw = (-2/n) * SUM[x_i * (y_i - y_predicted_i)]
dMSE/db = (-2/n) * SUM[(y_i - y_predicted_i)]

# learning_rate (alpha): controls step size
#   Too large -> overshoots minimum
#   Too small -> very slow convergence
#   Typical values: 0.01, 0.001, 0.0001

💡

Closed-form solution: Unlike many ML algorithms, Linear Regression has an exact solution called the Normal Equation: W = (X^T * X)^(-1) * X^T * y. This is faster for small datasets but becomes impractical for large ones (matrix inversion is O(n^3)). Gradient descent scales better to large datasets.

Assumptions of Linear Regression

Linear Regression relies on several key assumptions. Violating these can lead to unreliable predictions:

Assumption	What It Means	How to Check	What If Violated
Linearity	Relationship between X and Y is linear	Scatter plot, residual plot	Use polynomial features or non-linear model
Normality	Residuals are normally distributed	Q-Q plot, Shapiro-Wilk test	Transform target variable, use robust methods
Homoscedasticity	Constant variance of residuals	Residual vs fitted plot	Use weighted least squares or log transform
Independence	Observations are independent of each other	Durbin-Watson test	Use time-series models if autocorrelated
No multicollinearity	Features aren't highly correlated	VIF (Variance Inflation Factor)	Remove correlated features or use regularization

Simple vs. Multiple Regression

Aspect	Simple Regression	Multiple Regression
Features	1 input variable	2+ input variables
Equation	y = mx + b	y = w1x1 + w2x2 + ... + b
Geometry	Line in 2D	Hyperplane in n+1 dimensions
Use Case	Single predictor analysis	Real-world problems (multiple factors)
Risk	Underfitting	Overfitting if too many features

Regularization: Ridge, Lasso, ElasticNet

When you have many features, the model can overfit - memorizing the training data instead of learning general patterns. Regularization adds a penalty to large weights:

Ridge Regression (L2)

Cost = MSE + alpha * SUM(w_i^2)

# Adds the sum of SQUARED weights
# Shrinks all coefficients toward zero
# Never sets them exactly to zero
# Good for many small/medium effects
# alpha controls regularization strength

Lasso Regression (L1)

Cost = MSE + alpha * SUM(|w_i|)

# Adds the sum of ABSOLUTE weights
# Can set coefficients exactly to zero
# Performs automatic feature selection
# Good when you suspect many irrelevant features
# Produces sparse models

ElasticNet (L1 + L2)

Cost = MSE + alpha * (l1_ratio * SUM(|w_i|) + (1-l1_ratio) * SUM(w_i^2))

# Combines both L1 and L2 penalties
# l1_ratio controls the mix (0 = Ridge, 1 = Lasso)
# Best of both worlds: feature selection + stability
# Good when features are correlated

Evaluation Metrics

Metric	Formula	Interpretation	Range
R²	1 - (SS_res / SS_tot)	Proportion of variance explained	0 to 1 (higher is better)
MSE	(1/n) * SUM(y - y_hat)²	Average squared error	0 to inf (lower is better)
RMSE	sqrt(MSE)	Error in same units as target	0 to inf (lower is better)
MAE	(1/n) * SUM(\|y - y_hat\|)	Average absolute error	0 to inf (lower is better)

✅

Which metric to use? Use R² for a quick "how good is my model" overview. Use RMSE when large errors are especially bad (it penalizes outliers). Use MAE when you want a robust measure that's less sensitive to outliers. Use MSE during optimization (it's differentiable).

Python Implementation with scikit-learn

import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression, Ridge, Lasso, ElasticNet
from sklearn.metrics import mean_squared_error, r2_score, mean_absolute_error
from sklearn.datasets import fetch_california_housing

# Load real dataset: California Housing
data = fetch_california_housing()
X, y = data.data, data.target
feature_names = data.feature_names

print(f"Dataset shape: {X.shape}")
print(f"Features: {feature_names}")
print(f"Target: Median house value (in $100,000s)")

# Split into train and test sets
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# --- Simple Linear Regression ---
model = LinearRegression()
model.fit(X_train, y_train)
y_pred = model.predict(X_test)

print(f"\n--- Linear Regression ---")
print(f"R² Score:  {r2_score(y_test, y_pred):.4f}")
print(f"RMSE:      {np.sqrt(mean_squared_error(y_test, y_pred)):.4f}")
print(f"MAE:       {mean_absolute_error(y_test, y_pred):.4f}")

# Feature coefficients
print(f"\nFeature Coefficients:")
for name, coef in zip(feature_names, model.coef_):
    print(f"  {name:<15} {coef:+.4f}")
print(f"  {'Intercept':<15} {model.intercept_:+.4f}")

# --- Ridge Regression (L2) ---
ridge = Ridge(alpha=1.0)
ridge.fit(X_train, y_train)
y_pred_ridge = ridge.predict(X_test)
print(f"\n--- Ridge Regression (alpha=1.0) ---")
print(f"R² Score:  {r2_score(y_test, y_pred_ridge):.4f}")

# --- Lasso Regression (L1) ---
lasso = Lasso(alpha=0.01)
lasso.fit(X_train, y_train)
y_pred_lasso = lasso.predict(X_test)
print(f"\n--- Lasso Regression (alpha=0.01) ---")
print(f"R² Score:  {r2_score(y_test, y_pred_lasso):.4f}")
print(f"Features with zero coefficient: {sum(lasso.coef_ == 0)}")

# --- ElasticNet ---
elastic = ElasticNet(alpha=0.01, l1_ratio=0.5)
elastic.fit(X_train, y_train)
y_pred_elastic = elastic.predict(X_test)
print(f"\n--- ElasticNet (alpha=0.01, l1_ratio=0.5) ---")
print(f"R² Score:  {r2_score(y_test, y_pred_elastic):.4f}")

# --- Visualization: Actual vs Predicted ---
plt.figure(figsize=(8, 6))
plt.scatter(y_test, y_pred, alpha=0.3, s=10)
plt.plot([0, 5], [0, 5], 'r--', linewidth=2, label='Perfect prediction')
plt.xlabel('Actual House Value ($100K)')
plt.ylabel('Predicted House Value ($100K)')
plt.title('Linear Regression: Actual vs Predicted')
plt.legend()
plt.tight_layout()
plt.show()

When to Use / When Not to Use

✓

Use When

Relationship between features and target is approximately linear
You need interpretable coefficients
You want a fast, reliable baseline model
Feature effects should be additive
Small to medium datasets
Regulated industries requiring explainability

✗

Avoid When

Relationships are highly non-linear
Complex feature interactions exist
Target is categorical (use Logistic Regression)
Assumptions are severely violated
Outliers dominate the dataset
Features vastly outnumber samples (without regularization)

⚠

Common pitfall: Always check for multicollinearity before interpreting coefficients. If two features are highly correlated (e.g., "rooms" and "area"), their individual coefficients become unreliable even if the overall model performs well. Use VIF (Variance Inflation Factor) to detect this.

← Previous Overview Next → Logistic Regression

Ready to Go Deeper?

Live instructor-led courses from our partners. Affiliate disclosure.

🧠

AI & ML Courses - 30% Off

Live instructor-led AI, machine learning, data science, and cloud courses for working professionals. Use code Limited30 at checkout.

Edureka

📊

DataCamp - AI & Data Science

Hands-on Python, machine learning, and AI courses with interactive exercises and real projects.

DataCamp

🎓

edX - Top AI Courses

University-level AI courses from MIT, Harvard, Stanford. Earn certificates that employers recognize.

edX