Intermediate
Feature Engineering with Spark
Learn how to transform raw data into ML-ready features using Spark's built-in transformers, estimators, and feature utilities.
Transformers vs Estimators
Spark ML uses two key abstractions for feature engineering:
- Transformer: Takes a DataFrame, returns a new DataFrame with added columns. Uses
.transform(). Example: VectorAssembler. - Estimator: Takes a DataFrame, learns parameters, returns a Transformer (model). Uses
.fit()then.transform(). Example: StandardScaler.
VectorAssembler
The most essential transformer - combines multiple columns into a single feature vector.
Python - VectorAssembler
from pyspark.ml.feature import VectorAssembler
assembler = VectorAssembler(
inputCols=["age", "income", "credit_score", "num_products"],
outputCol="features"
)
df_assembled = assembler.transform(df)
df_assembled.select("features", "label").show(5, truncate=False)Encoding Categorical Features
Python - StringIndexer and OneHotEncoder
from pyspark.ml.feature import StringIndexer, OneHotEncoder
# Convert string labels to numeric indices
indexer = StringIndexer(inputCol="category", outputCol="category_index")
df_indexed = indexer.fit(df).transform(df)
# One-hot encode the indices
encoder = OneHotEncoder(inputCol="category_index", outputCol="category_vec")
df_encoded = encoder.fit(df_indexed).transform(df_indexed)
# For multiple columns at once
indexers = [
StringIndexer(inputCol=col, outputCol=f"{col}_index")
for col in ["city", "gender", "education"]
]Scaling and Normalization
Python - Feature Scaling
from pyspark.ml.feature import StandardScaler, MinMaxScaler, Normalizer
# StandardScaler (zero mean, unit variance)
scaler = StandardScaler(
inputCol="features",
outputCol="scaled_features",
withMean=True,
withStd=True
)
scaler_model = scaler.fit(df)
df_scaled = scaler_model.transform(df)
# MinMaxScaler (scale to [0, 1])
min_max = MinMaxScaler(inputCol="features", outputCol="normalized_features")
df_normalized = min_max.fit(df).transform(df)
# L2 Normalization (unit norm)
normalizer = Normalizer(inputCol="features", outputCol="norm_features", p=2.0)
df_normed = normalizer.transform(df)Text Features
Python - Text Feature Extraction
from pyspark.ml.feature import Tokenizer, HashingTF, IDF
# Tokenize text
tokenizer = Tokenizer(inputCol="text", outputCol="words")
df_words = tokenizer.transform(df)
# Term frequency
hashing_tf = HashingTF(inputCol="words", outputCol="raw_features", numFeatures=10000)
df_tf = hashing_tf.transform(df_words)
# TF-IDF
idf = IDF(inputCol="raw_features", outputCol="tfidf_features")
df_tfidf = idf.fit(df_tf).transform(df_tf)Feature Selection
Python - Feature Selection with ChiSqSelector
from pyspark.ml.feature import ChiSqSelector
selector = ChiSqSelector(
numTopFeatures=20,
featuresCol="features",
outputCol="selected_features",
labelCol="label"
)
model = selector.fit(df)
df_selected = model.transform(df)
print(f"Selected feature indices: {model.selectedFeatures}")Common Transformers Reference
| Transformer | Type | Purpose |
|---|---|---|
| VectorAssembler | Transformer | Combine columns into feature vector |
| StringIndexer | Estimator | Map strings to numeric indices |
| OneHotEncoder | Estimator | One-hot encode categorical indices |
| StandardScaler | Estimator | Standardize to zero mean, unit variance |
| Bucketizer | Transformer | Bin continuous features into buckets |
| Imputer | Estimator | Fill missing values with mean/median |
| PCA | Estimator | Dimensionality reduction |
Chain transformers in a Pipeline: Rather than applying transformers one at a time, chain them in a Pipeline for cleaner code and easier model persistence. We cover this in the next lesson.
Ready to Go Deeper?
Live instructor-led courses from our partners. Affiliate disclosure.
AI & ML Courses - 30% Off
Live instructor-led AI, machine learning, data science, and cloud courses for working professionals. Use code Limited30 at checkout.
EdurekaDataCamp - AI & Data Science
Hands-on Python, machine learning, and AI courses with interactive exercises and real projects.
DataCampedX - Top AI Courses
University-level AI courses from MIT, Harvard, Stanford. Earn certificates that employers recognize.
edX