Beginner

Text Preprocessing

Raw text is messy. Before any NLP model can work with text data, it must be cleaned, normalized, and transformed into a structured format.

Why Preprocessing Matters

Text data from the real world contains noise: HTML tags, special characters, inconsistent casing, typos, and more. Preprocessing transforms this raw text into a clean, consistent format that models can learn from effectively. Poor preprocessing leads to poor models - garbage in, garbage out.

Tokenization

Tokenization is the process of splitting text into individual units called tokens. These tokens can be words, subwords, or characters depending on the approach.

Word Tokenization

The simplest approach: split text by whitespace and punctuation.

Python - NLTK

import nltk
nltk.download('punkt')

text = "NLP isn't just about splitting words. It's more complex!"

# Word tokenization
tokens = nltk.word_tokenize(text)
print(tokens)
# ['NLP', 'is', "n't", 'just', 'about', 'splitting', 'words', '.', ...]

Subword Tokenization

Modern models like BERT and GPT use subword tokenization (BPE, WordPiece, SentencePiece) to handle unknown words by breaking them into meaningful subunits.

Python - Hugging Face Tokenizer

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
tokens = tokenizer.tokenize("unhappiness is unavoidable")
print(tokens)
# ['un', '##happi', '##ness', 'is', 'un', '##avoid', '##able']

Character Tokenization

Breaking text into individual characters. Useful for languages without clear word boundaries or for character-level models.

Stopword Removal

Stopwords are common words like "the," "is," "and," "in" that carry little semantic meaning. Removing them reduces noise and dimensionality.

Python - NLTK

from nltk.corpus import stopwords
nltk.download('stopwords')

stop_words = set(stopwords.words('english'))
tokens = ['NLP', 'is', 'an', 'exciting', 'field', 'of', 'AI']

filtered = [t for t in tokens if t.lower() not in stop_words]
print(filtered)
# ['NLP', 'exciting', 'field', 'AI']

💡

Caution: Do not blindly remove stopwords for all tasks. For sentiment analysis, words like "not" and "no" are crucial. For transformer-based models, stopword removal is usually unnecessary and can hurt performance.

Stemming and Lemmatization

Both techniques reduce words to their base form, but they differ in approach:

Technique	Method	Example	Result
Stemming	Chops off word endings using rules	"running," "runs," "ran"	"run" (but may produce "runn")
Lemmatization	Uses vocabulary and morphology	"running," "runs," "ran"	"run" (always valid word)

Python

from nltk.stem import PorterStemmer, WordNetLemmatizer
nltk.download('wordnet')

stemmer = PorterStemmer()
lemmatizer = WordNetLemmatizer()

words = ["running", "better", "studies", "happily"]

print("Stemming:", [stemmer.stem(w) for w in words])
# ['run', 'better', 'studi', 'happili']

print("Lemmatization:", [lemmatizer.lemmatize(w) for w in words])
# ['running', 'better', 'study', 'happily']

Regular Expressions for Text Cleaning

Regular expressions (regex) are powerful tools for finding and replacing patterns in text:

Python

import re

text = "Contact us at info@example.com or visit https://example.com! Price: $29.99"

# Remove emails
clean = re.sub(r'\S+@\S+', '', text)

# Remove URLs
clean = re.sub(r'https?://\S+', '', clean)

# Remove special characters (keep alphanumeric and spaces)
clean = re.sub(r'[^a-zA-Z0-9\s]', '', clean)

# Remove extra whitespace
clean = re.sub(r'\s+', ' ', clean).strip()

print(clean)
# "Contact us at  or visit  Price 2999"

Cleaning HTML and Special Characters

Web-scraped text often contains HTML tags and entities that need removal:

Python

from bs4 import BeautifulSoup
import html

raw = "<p>Hello &amp; welcome to <b>NLP</b>!</p>"

# Remove HTML tags
soup = BeautifulSoup(raw, "html.parser")
clean = soup.get_text()

# Decode HTML entities
clean = html.unescape(clean)
print(clean)
# "Hello & welcome to NLP!"

Sentence Segmentation

Splitting text into individual sentences is important for tasks like summarization and translation:

Python - spaCy

import spacy

nlp = spacy.load("en_core_web_sm")
doc = nlp("Dr. Smith went to Washington. He arrived at 3 p.m. It was raining.")

for sent in doc.sents:
    print(sent.text)
# "Dr. Smith went to Washington."
# "He arrived at 3 p.m."
# "It was raining."

Complete Preprocessing Pipeline with spaCy

Python - spaCy Pipeline

import spacy

nlp = spacy.load("en_core_web_sm")

def preprocess(text):
    doc = nlp(text.lower())
    tokens = [
        token.lemma_
        for token in doc
        if not token.is_stop
        and not token.is_punct
        and not token.is_space
        and len(token.text) > 1
    ]
    return tokens

result = preprocess("The cats are running quickly through the beautiful garden!")
print(result)
# ['cat', 'run', 'quickly', 'beautiful', 'garden']

✅

Key takeaway: The preprocessing steps you choose depend on your task. Traditional ML models (Naive Bayes, SVM) benefit from extensive preprocessing, while transformer models like BERT handle raw text with their own tokenizers and often perform better without manual preprocessing.

← Previous Introduction Next → Text Representation

Ready to Go Deeper?

Live instructor-led courses from our partners. Affiliate disclosure.

🧠

AI & ML Courses - 30% Off

Live instructor-led AI, machine learning, data science, and cloud courses for working professionals. Use code Limited30 at checkout.

Edureka

📊

DataCamp - AI & Data Science

Hands-on Python, machine learning, and AI courses with interactive exercises and real projects.

DataCamp

🎓

edX - Top AI Courses

University-level AI courses from MIT, Harvard, Stanford. Earn certificates that employers recognize.

edX