Introduction to Data Poisoning

Data poisoning is one of the most insidious threats to AI systems - attackers compromise the training data itself, causing the model to learn wrong behaviors that persist through deployment.

What Is Data Poisoning?

Data poisoning is an attack on the training pipeline of a machine learning model. Instead of attacking the deployed model directly, the attacker manipulates the data used to train it. This causes the model to learn incorrect patterns, biased behaviors, or hidden backdoors that activate under specific conditions.

Attack Surface in the ML Pipeline

The ML training pipeline has multiple points where data can be poisoned:

Data Collection  →  Web scraping, user contributions, third-party datasets
     [ATTACK POINT]     Inject malicious samples into source data

Data Storage      →  Databases, data lakes, file systems
     [ATTACK POINT]     Modify stored data directly

Data Processing   →  Cleaning, normalization, augmentation
     [ATTACK POINT]     Compromise processing scripts

Labeling          →  Human annotation, automated labeling
     [ATTACK POINT]     Corrupt labels through malicious annotators

Training          →  Model fitting, fine-tuning
     [ATTACK POINT]     Supply poisoned pre-trained weights

Deployment        →  Model serving, continuous learning
     [ATTACK POINT]     Poison feedback loops

Types of Data Poisoning

Real-World Incidents

Why Data Poisoning Is Hard to Defend Against

• Scale: Modern models train on billions of samples - manual review is impossible
• Subtlety: Clean-label attacks look legitimate even to expert reviewers
• Persistence: Once trained, poisoned behaviors are baked into model weights
• Supply chain complexity: Training data comes from many sources, each a potential attack vector
• Delayed detection: Effects may only appear in production with specific inputs