Introduction to Hugging Face Datasets

Discover the datasets library that provides efficient access to over 100,000 datasets, backed by Apache Arrow for blazing-fast data processing with minimal memory usage.

What is the Datasets Library?

The Hugging Face datasets library is the standard way to access, process, and share ML datasets. It provides a unified API for loading datasets from the Hugging Face Hub, local files, or in-memory data structures, with built-in support for efficient processing and caching.

Quick Start

from datasets import load_dataset

# Load a dataset from the Hugging Face Hub
dataset = load_dataset("imdb")

# Explore the dataset
print(dataset)
# DatasetDict({
#     train: Dataset({features: ['text', 'label'], num_rows: 25000})
#     test:  Dataset({features: ['text', 'label'], num_rows: 25000})
# })

# Access individual examples
print(dataset["train"][0])
# {'text': 'I rented I AM CURIOUS...', 'label': 0}

# Access columns
labels = dataset["train"]["label"]  # Returns a list

Why Not Just Use Pandas?

Apache Arrow Backend

The datasets library stores data in Apache Arrow format, which provides several key advantages:

• Memory mapping - Data is read directly from disk without copying into RAM, enabling processing of datasets larger than available memory.
• Zero-copy reads - Slicing and indexing operations do not copy data, making them extremely fast.
• Columnar storage - Accessing a single column is fast because data is stored column-wise, not row-wise.
• Cross-language support - Arrow tables can be shared between Python, R, and other languages without serialization.

Installation

# Install the datasets library
pip install datasets

# With audio support
pip install datasets[audio]

# With vision support
pip install datasets[vision]

Key Concepts

1. Dataset

   A single table of data with typed columns (features). Backed by an Arrow table.

2. DatasetDict

   A dictionary of Dataset objects, typically with "train", "validation", and "test" splits.

3. Features

   The schema defining column names and types (ClassLabel, Value, Image, Audio, etc.).

4. IterableDataset

   A streaming version that yields examples one at a time without loading everything into memory.