Introduction to Apache Spark for ML

Understand what Apache Spark is, how distributed computing powers large-scale ML, and why Spark has become the industry standard for big data machine learning.

What is Apache Spark?

Apache Spark is a unified analytics engine for large-scale data processing. Originally developed at UC Berkeley's AMPLab in 2009 and later donated to the Apache Software Foundation, Spark provides an interface for programming entire clusters with implicit data parallelism and fault tolerance.

Spark is up to 100x faster than Hadoop MapReduce for in-memory processing and 10x faster on disk. It supports batch processing, streaming, SQL queries, graph processing, and machine learning - all through a single engine.

Why Spark for Machine Learning?

• Scale: Train models on datasets that don't fit in a single machine's memory - terabytes or petabytes of data.
• Speed: In-memory computing and lazy evaluation optimize execution plans automatically.
• Unified platform: Data preparation, feature engineering, model training, and evaluation in one framework.
• MLlib: Built-in library with distributed implementations of common ML algorithms.
• Integration: Works with Hadoop, S3, Kafka, Delta Lake, and all major cloud platforms.

Spark Architecture

Spark Ecosystem for ML

Getting Started

# Install PySpark via pip
pip install pyspark

# Or install with all extras
pip install pyspark[sql,ml,streaming]

# Verify installation
python -c "import pyspark; print(pyspark.__version__)"

from pyspark.sql import SparkSession

# Create a SparkSession
spark = SparkSession.builder \
    .appName("MyFirstSparkML") \
    .master("local[*]") \
    .getOrCreate()

# Read data
df = spark.read.csv("data.csv", header=True, inferSchema=True)
df.printSchema()
df.show(5)

# Basic statistics
df.describe().show()

# Stop the session when done
spark.stop()