Intermediate

Classification Models

Classification models predict categorical outcomes - will a customer churn? Is an email spam? Will a loan default? Learn the key algorithms and how to evaluate their performance.

Logistic Regression

Despite its name, logistic regression is a classification algorithm. It predicts the probability that an observation belongs to a category using the sigmoid function, which maps any input to a value between 0 and 1.

Python 
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

# Prepare data
X = df[['age', 'income', 'credit_score', 'debt_ratio']]
y = df['default']  # 0 = no default, 1 = default

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Scale features
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# Fit logistic regression
model = LogisticRegression(random_state=42)
model.fit(X_train_scaled, y_train)

# Predict probabilities
probabilities = model.predict_proba(X_test_scaled)[:, 1]
predictions = model.predict(X_test_scaled)

Decision Trees

Decision trees split data into branches based on feature values, creating a tree-like structure of if-then rules. They are intuitive and easy to interpret.

Python 
from sklearn.tree import DecisionTreeClassifier, plot_tree
import matplotlib.pyplot as plt

# Fit decision tree
tree = DecisionTreeClassifier(max_depth=4, random_state=42)
tree.fit(X_train, y_train)

# Visualize the tree
plt.figure(figsize=(16, 8))
plot_tree(tree, feature_names=X.columns, class_names=['No', 'Yes'],
          filled=True, rounded=True)
plt.title('Decision Tree')
plt.show()

# Feature importance
for name, importance in zip(X.columns, tree.feature_importances_):
    print(f"{name}: {importance:.4f}")

Random Forests

Random forests build many decision trees on random subsets of data and features, then average their predictions. This reduces overfitting and usually outperforms a single decision tree.

Python 
from sklearn.ensemble import RandomForestClassifier

# Fit random forest
rf = RandomForestClassifier(
    n_estimators=100,   # Number of trees
    max_depth=10,       # Max depth per tree
    random_state=42
)
rf.fit(X_train, y_train)

print(f"Training accuracy: {rf.score(X_train, y_train):.4f}")
print(f"Test accuracy: {rf.score(X_test, y_test):.4f}")

Evaluating Classification Models

Confusion Matrix

A confusion matrix shows how many predictions were correct and incorrect for each class.

Python 
from sklearn.metrics import (confusion_matrix, classification_report,
                               accuracy_score, roc_auc_score, roc_curve)

y_pred = rf.predict(X_test)

# Confusion matrix
cm = confusion_matrix(y_test, y_pred)
print("Confusion Matrix:")
print(cm)
# [[TN, FP],
#  [FN, TP]]

# Detailed classification report
print(classification_report(y_test, y_pred))

Key Metrics

Metric Formula When to Prioritize
Accuracy (TP + TN) / Total Balanced classes, equal cost of errors
Precision TP / (TP + FP) When false positives are costly (spam detection)
Recall TP / (TP + FN) When false negatives are costly (disease detection)
F1-Score 2 × (Precision × Recall) / (Precision + Recall) When you need a balance of precision and recall
AUC-ROC Area under the ROC curve Overall model discriminative ability
⚠
Do not rely on accuracy alone. With imbalanced data (e.g., 95% negative, 5% positive), a model that always predicts "negative" gets 95% accuracy but is useless. Use precision, recall, and F1-score instead.

ROC Curves and AUC

The ROC (Receiver Operating Characteristic) curve plots the true positive rate against the false positive rate at various classification thresholds. AUC (Area Under the Curve) summarizes the overall performance - 0.5 is random guessing, 1.0 is perfect.

Python 
import matplotlib.pyplot as plt

# Get predicted probabilities
y_prob = rf.predict_proba(X_test)[:, 1]

# Calculate ROC curve
fpr, tpr, thresholds = roc_curve(y_test, y_prob)
auc = roc_auc_score(y_test, y_prob)

# Plot ROC curve
plt.figure(figsize=(8, 6))
plt.plot(fpr, tpr, label=f'ROC Curve (AUC = {auc:.3f})')
plt.plot([0, 1], [0, 1], 'k--', label='Random (AUC = 0.5)')
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('ROC Curve')
plt.legend()
plt.show()

Cross-Validation

Cross-validation gives a more reliable estimate of model performance by training and testing on different subsets of data.

Python 
from sklearn.model_selection import cross_val_score

# 5-fold cross-validation
scores = cross_val_score(rf, X, y, cv=5, scoring='accuracy')
print(f"CV Accuracy: {scores.mean():.4f} (+/- {scores.std():.4f})")

# Cross-validation with different metrics
f1_scores = cross_val_score(rf, X, y, cv=5, scoring='f1')
print(f"CV F1-Score: {f1_scores.mean():.4f} (+/- {f1_scores.std():.4f})")
✅
Model comparison tip: Use cross-validation to compare models fairly. A model with a higher mean CV score and lower standard deviation is preferable - it is both better and more consistent.

Ready to Go Deeper?

Live instructor-led courses from our partners. Affiliate disclosure.

🧠

AI & ML Courses - 30% Off

Live instructor-led AI, machine learning, data science, and cloud courses for working professionals. Use code Limited30 at checkout.

Edureka
📊

DataCamp - AI & Data Science

Hands-on Python, machine learning, and AI courses with interactive exercises and real projects.

DataCamp
🎓

edX - Top AI Courses

University-level AI courses from MIT, Harvard, Stanford. Earn certificates that employers recognize.

edX