Intermediate

Regression Models

Regression models predict continuous outcomes based on one or more predictor variables. From simple linear regression to regularized models, this lesson covers the techniques you need to model real-world relationships.

Simple Linear Regression

Simple linear regression models the relationship between one predictor (x) and one outcome (y) as a straight line: y = β₀ + β₁x + ε

  • β₀ (intercept) - The predicted value of y when x = 0
  • β₁ (slope) - The change in y for a one-unit increase in x
  • ε (error) - The difference between predicted and actual values
Python (Scikit-learn) 
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
import numpy as np

# Prepare data
X = df[['experience_years']]  # Predictor (2D array)
y = df['salary']                # Target

# Split into train and test sets
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Fit the model
model = LinearRegression()
model.fit(X_train, y_train)

print(f"Intercept: ${model.intercept_:,.0f}")
print(f"Slope: ${model.coef_[0]:,.0f} per year")
print(f"R-squared: {model.score(X_test, y_test):.4f}")

Multiple Linear Regression

Extends simple regression to multiple predictors: y = β₀ + β₁x₁ + β₂x₂ + ... + βₙxₙ + ε

Python (Statsmodels) 
import statsmodels.api as sm

# Multiple regression with detailed statistics
X = df[['experience', 'education_years', 'age']]
y = df['salary']

# Add constant for intercept
X = sm.add_constant(X)

# Fit OLS model
model = sm.OLS(y, X).fit()
print(model.summary())

# The summary includes:
# - R-squared and Adjusted R-squared
# - Coefficients with p-values
# - Confidence intervals
# - F-statistic for overall model significance

Assumptions of Linear Regression

Linear regression relies on several assumptions. Violating them can produce misleading results.

Assumption What It Means How to Check
Linearity The relationship between X and Y is linear Scatter plot of X vs Y, residuals vs fitted
Independence Observations are independent of each other Durbin-Watson test, study design
Normality Residuals are normally distributed Q-Q plot, Shapiro-Wilk test
Homoscedasticity Residuals have constant variance across all X Residuals vs fitted plot, Breusch-Pagan test
No multicollinearity Predictors are not highly correlated with each other VIF (Variance Inflation Factor) > 10 is concerning

R-squared and Adjusted R-squared

R-squared (R²) measures the proportion of variance in the outcome explained by the model. Values range from 0 to 1, where 1 means perfect prediction.

⚠
R² always increases when you add more predictors, even useless ones. Use Adjusted R² instead, which penalizes for the number of predictors and only increases if a new variable genuinely improves the model.

Polynomial Regression

When the relationship is curved rather than straight, polynomial regression adds higher-order terms (x², x³, etc.).

Python 
from sklearn.preprocessing import PolynomialFeatures
from sklearn.pipeline import make_pipeline

# Polynomial regression (degree 2)
poly_model = make_pipeline(
    PolynomialFeatures(degree=2),
    LinearRegression()
)
poly_model.fit(X_train, y_train)

print(f"R-squared: {poly_model.score(X_test, y_test):.4f}")

Regularization

Regularization prevents overfitting by adding a penalty for large coefficients. This is especially important with many predictors or multicollinearity.

🛠

Ridge (L2)

Shrinks coefficients toward zero but never eliminates them. Good when all features are potentially relevant. Penalty: αΣβ²

✏

Lasso (L1)

Can shrink coefficients to exactly zero, performing feature selection. Good when you suspect many features are irrelevant. Penalty: αΣ|β|

🔄

ElasticNet

Combines Ridge and Lasso penalties. Balances feature selection with coefficient shrinkage. Best of both worlds.

Python 
from sklearn.linear_model import Ridge, Lasso, ElasticNet
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

# Always scale features before regularization
ridge = make_pipeline(StandardScaler(), Ridge(alpha=1.0))
lasso = make_pipeline(StandardScaler(), Lasso(alpha=0.1))
elastic = make_pipeline(StandardScaler(), ElasticNet(alpha=0.1, l1_ratio=0.5))

# Fit and compare
for name, model in [('Ridge', ridge), ('Lasso', lasso), ('ElasticNet', elastic)]:
    model.fit(X_train, y_train)
    score = model.score(X_test, y_test)
    print(f"{name}: R-squared = {score:.4f}")

# Use cross-validation to find optimal alpha
from sklearn.linear_model import RidgeCV
ridge_cv = RidgeCV(alphas=[0.01, 0.1, 1.0, 10.0, 100.0])
ridge_cv.fit(X_train, y_train)
print(f"Best alpha: {ridge_cv.alpha_}")
✅
When to use regularization: Use Ridge when you have many correlated features. Use Lasso when you want automatic feature selection. Use ElasticNet when you are unsure. Always standardize features first, since regularization is sensitive to feature scales.

Ready to Go Deeper?

Live instructor-led courses from our partners. Affiliate disclosure.

🧠

AI & ML Courses - 30% Off

Live instructor-led AI, machine learning, data science, and cloud courses for working professionals. Use code Limited30 at checkout.

Edureka
📊

DataCamp - AI & Data Science

Hands-on Python, machine learning, and AI courses with interactive exercises and real projects.

DataCamp
🎓

edX - Top AI Courses

University-level AI courses from MIT, Harvard, Stanford. Earn certificates that employers recognize.

edX