Intermediate

Data Transformation for Machine Learning

Transform raw data into ML-ready features through cleaning, encoding, normalization, and feature engineering techniques.

The Transformation Pipeline

Data transformation for ML follows a structured sequence. Each step builds on the previous one:

• Data cleaning: Handle missing values, remove duplicates, fix data types, and correct errors.
• Feature engineering: Create new features from existing data to improve model performance.
• Encoding: Convert categorical variables to numerical representations.
• Scaling/Normalization: Standardize numerical features to comparable ranges.
• Feature selection: Remove irrelevant or redundant features to reduce dimensionality.

Handling Missing Values

Missing data is inevitable. Choose your strategy based on the data type and the missingness pattern:

import pandas as pd
import numpy as np
from sklearn.impute import SimpleImputer, KNNImputer

# Strategy 1: Drop rows with too many missing values
df = df.dropna(thresh=len(df.columns) * 0.7)

# Strategy 2: Impute numerical columns with median
num_imputer = SimpleImputer(strategy="median")
df[num_cols] = num_imputer.fit_transform(df[num_cols])

# Strategy 3: KNN imputation for correlated features
knn_imputer = KNNImputer(n_neighbors=5)
df[num_cols] = knn_imputer.fit_transform(df[num_cols])

# Strategy 4: Categorical imputation with mode
cat_imputer = SimpleImputer(strategy="most_frequent")
df[cat_cols] = cat_imputer.fit_transform(df[cat_cols])

Feature Engineering

Feature engineering is often the difference between a mediocre model and a great one:

# Time-based features
df["hour"] = df["timestamp"].dt.hour
df["day_of_week"] = df["timestamp"].dt.dayofweek
df["is_weekend"] = df["day_of_week"].isin([5, 6]).astype(int)

# Aggregation features
user_stats = df.groupby("user_id").agg(
    total_purchases=("amount", "sum"),
    avg_purchase=("amount", "mean"),
    purchase_count=("amount", "count"),
    days_since_first=("timestamp", lambda x: (x.max() - x.min()).days)
).reset_index()

# Interaction features
df["price_per_unit"] = df["total_price"] / df["quantity"]
df["discount_ratio"] = df["discount"] / df["original_price"]

# Text features
df["description_length"] = df["description"].str.len()
df["word_count"] = df["description"].str.split().str.len()

Categorical Encoding

Method	Best For	Example
Label Encoding	Ordinal categories	low=0, medium=1, high=2
One-Hot Encoding	Nominal, low cardinality	[1,0,0], [0,1,0], [0,0,1]
Target Encoding	High cardinality	Replace with mean target value
Frequency Encoding	When count matters	Replace with occurrence frequency
Embedding	Very high cardinality	Learned dense representations

Feature Scaling

Many ML algorithms are sensitive to feature scale. Choose the right scaler for your data:

from sklearn.preprocessing import StandardScaler, MinMaxScaler, RobustScaler

# StandardScaler: zero mean, unit variance (best for normally distributed)
scaler = StandardScaler()
df[num_cols] = scaler.fit_transform(df[num_cols])

# MinMaxScaler: scale to [0, 1] (good for neural networks)
scaler = MinMaxScaler()
df[num_cols] = scaler.fit_transform(df[num_cols])

# RobustScaler: uses median and IQR (robust to outliers)
scaler = RobustScaler()
df[num_cols] = scaler.fit_transform(df[num_cols])

Building a Transformation Pipeline

Use scikit-learn Pipelines to chain transformations together for reproducibility:

from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.impute import SimpleImputer

numeric_pipeline = Pipeline([
    ("imputer", SimpleImputer(strategy="median")),
    ("scaler", StandardScaler())
])

categorical_pipeline = Pipeline([
    ("imputer", SimpleImputer(strategy="most_frequent")),
    ("encoder", OneHotEncoder(handle_unknown="ignore"))
])

preprocessor = ColumnTransformer([
    ("numeric", numeric_pipeline, num_cols),
    ("categorical", categorical_pipeline, cat_cols)
])

# Fit on training data, transform both train and test
X_train_transformed = preprocessor.fit_transform(X_train)
X_test_transformed = preprocessor.transform(X_test)