Intermediate

Text Representation

Machines do not understand words - they understand numbers. Text representation converts words and documents into numerical vectors that ML models can process.

Bag of Words (BoW)

The simplest text representation. Each document is represented as a vector of word counts, ignoring word order and grammar.

Python - sklearn

from sklearn.feature_extraction.text import CountVectorizer

corpus = [
    "I love NLP",
    "NLP is amazing",
    "I love machine learning"
]

vectorizer = CountVectorizer()
X = vectorizer.fit_transform(corpus)

print(vectorizer.get_feature_names_out())
# ['amazing', 'is', 'learning', 'love', 'machine', 'nlp']

print(X.toarray())
# [[0, 0, 0, 1, 0, 1],   <-- "I love NLP"
#  [1, 1, 0, 0, 0, 1],   <-- "NLP is amazing"
#  [0, 0, 1, 1, 1, 0]]   <-- "I love machine learning"

💡

Limitation: BoW loses all word order information. "Dog bites man" and "Man bites dog" would have the same representation despite having very different meanings.

TF-IDF (Term Frequency-Inverse Document Frequency)

TF-IDF improves on BoW by weighing words based on how important they are to a document relative to the entire corpus. Common words get lower weights; rare but meaningful words get higher weights.

Python - sklearn

from sklearn.feature_extraction.text import TfidfVectorizer

corpus = [
    "NLP processes natural language",
    "Machine learning powers NLP",
    "Deep learning is a subset of machine learning"
]

tfidf = TfidfVectorizer()
X = tfidf.fit_transform(corpus)

print(tfidf.get_feature_names_out())
print(X.toarray().round(2))
# Words unique to a document get higher TF-IDF scores
# Words appearing in many documents get lower scores

Word Embeddings

Word embeddings represent words as dense, low-dimensional vectors where semantically similar words are close together in vector space. This was a major breakthrough in NLP.

Word2Vec

Introduced by Google in 2013, Word2Vec learns word representations from large text corpora using two architectures:

CBOW (Continuous Bag of Words): Predicts a target word from surrounding context words. Faster to train, works well with frequent words.
Skip-gram: Predicts surrounding context words from a target word. Works better with rare words and smaller datasets.

Python - Gensim

from gensim.models import Word2Vec

# Training sentences (each is a list of tokens)
sentences = [
    ["king", "is", "a", "ruler"],
    ["queen", "is", "a", "ruler"],
    ["man", "is", "strong"],
    ["woman", "is", "strong"],
]

model = Word2Vec(sentences, vector_size=50, window=5, min_count=1)

# Get word vector
vector = model.wv["king"]

# Find similar words
similar = model.wv.most_similar("king")
print(similar)

GloVe (Global Vectors)

Developed at Stanford, GloVe learns word vectors by factorizing the word co-occurrence matrix of a corpus. It captures both local and global statistical information, producing high-quality embeddings.

FastText

Created by Facebook, FastText extends Word2Vec by representing each word as a bag of character n-grams. This means it can generate vectors for out-of-vocabulary words by composing their character n-grams.

Contextual Embeddings

Unlike static embeddings (Word2Vec, GloVe) where each word has one fixed vector, contextual embeddings generate different vectors for the same word depending on its context:

Type	Model	"Bank" in different contexts
Static	Word2Vec / GloVe	Same vector for "river bank" and "bank account"
Contextual	BERT / GPT	Different vectors for "river bank" and "bank account"

Python - BERT Embeddings

from transformers import AutoTokenizer, AutoModel
import torch

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
model = AutoModel.from_pretrained("bert-base-uncased")

text = "NLP is transforming the world"
inputs = tokenizer(text, return_tensors="pt")

with torch.no_grad():
    outputs = model(**inputs)

# Last hidden state: contextual embeddings for each token
embeddings = outputs.last_hidden_state
print(embeddings.shape)
# torch.Size([1, 8, 768]) -- 8 tokens, 768-dim each

Sentence Embeddings

Sometimes you need a single vector for an entire sentence or paragraph, not just individual words. Sentence-BERT (SBERT) and similar models produce fixed-size sentence embeddings:

Python - Sentence Transformers

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")

sentences = [
    "I love natural language processing",
    "NLP is my favorite field",
    "The weather is nice today"
]

embeddings = model.encode(sentences)
print(embeddings.shape)  # (3, 384)

Vector Similarity (Cosine Similarity)

Once text is represented as vectors, we can measure how similar two texts are using cosine similarity, which measures the angle between two vectors:

Python

from sklearn.metrics.pairwise import cosine_similarity
import numpy as np

# Using sentence embeddings from above
sim_matrix = cosine_similarity(embeddings)

print("Similarity: 'I love NLP' vs 'NLP is my favorite':", sim_matrix[0][1].round(3))
# High similarity (e.g., 0.82)

print("Similarity: 'I love NLP' vs 'Weather is nice':", sim_matrix[0][2].round(3))
# Low similarity (e.g., 0.15)

✅

Key takeaway: Text representation has evolved from sparse, high-dimensional vectors (BoW, TF-IDF) to dense, meaningful embeddings (Word2Vec, BERT). Modern contextual embeddings capture nuanced meanings and are the foundation of today's state-of-the-art NLP systems.

← Previous Text Preprocessing Next → NLP Tasks

Ready to Go Deeper?

Live instructor-led courses from our partners. Affiliate disclosure.

🧠

AI & ML Courses - 30% Off

Live instructor-led AI, machine learning, data science, and cloud courses for working professionals. Use code Limited30 at checkout.

Edureka

📊

DataCamp - AI & Data Science

Hands-on Python, machine learning, and AI courses with interactive exercises and real projects.

DataCamp

🎓

edX - Top AI Courses

University-level AI courses from MIT, Harvard, Stanford. Earn certificates that employers recognize.

edX