Data Poisoning Detection
Learn techniques to identify poisoned training data and backdoored models using statistical analysis, spectral methods, activation clustering, and reverse-engineering approaches.
Detection Approaches Overview
| Method | Detects | When to Use |
|---|---|---|
| Statistical Analysis | Data anomalies | Before training - screen incoming data |
| Spectral Signatures | Poisoned samples | After training - analyze learned representations |
| Activation Clustering | Backdoors | After training - examine model internals |
| Neural Cleanse | Trigger patterns | After training - reverse-engineer triggers |
| STRIP | Triggered inputs | At inference - real-time detection |
Spectral Signature Detection
Poisoned samples often leave a detectable signature in the model's learned representation space:
import numpy as np from sklearn.decomposition import PCA def detect_spectral_signatures(representations, labels, epsilon=1.5): """Detect poisoned samples via spectral signatures.""" suspicious = [] for target_class in np.unique(labels): # Get representations for this class class_reps = representations[labels == target_class] centered = class_reps - class_reps.mean(axis=0) # Compute top singular vector _, _, Vt = np.linalg.svd(centered, full_matrices=False) top_vector = Vt[0] # Score each sample by projection onto top vector scores = np.abs(centered @ top_vector) # Flag outliers as potentially poisoned threshold = np.mean(scores) + epsilon * np.std(scores) poisoned_idx = np.where(scores > threshold)[0] suspicious.extend(poisoned_idx.tolist()) return suspicious
Neural Cleanse
Neural Cleanse reverse-engineers potential trigger patterns by finding the smallest input perturbation that causes all samples of one class to be classified as another:
# For each target class t: For each source class s != t: # Find minimum trigger that flips s -> t trigger = optimize( minimize: |trigger_mask| + |trigger_pattern| subject_to: model(x + trigger) == t for all x in class s ) record trigger size # Backdoored class has anomalously small trigger If any class has trigger_size << median(all trigger_sizes): ALERT: Likely backdoor detected for that class The recovered trigger approximates the actual backdoor trigger
Activation Clustering
Backdoor samples create a distinct cluster in the model's activation space:
- Extract activations from an intermediate layer for all samples of each class
- Apply dimensionality reduction (PCA, t-SNE)
- Run clustering (k-means with k=2) on each class
- If a class has two well-separated clusters, the smaller cluster likely contains poisoned samples
STRIP: Runtime Detection
STRIP (STRong Intentional Perturbation) detects triggered inputs at inference time:
- Overlay the input with random clean images/text
- If the prediction stays constant despite perturbation, the input likely contains a trigger
- Clean inputs change prediction when perturbed; triggered inputs maintain the backdoor label
Ready to Go Deeper?
Live instructor-led courses from our partners. Affiliate disclosure.
AI & ML Courses - 30% Off
Live instructor-led AI, machine learning, data science, and cloud courses for working professionals. Use code Limited30 at checkout.
EdurekaDataCamp - AI & Data Science
Hands-on Python, machine learning, and AI courses with interactive exercises and real projects.
DataCampedX - Top AI Courses
University-level AI courses from MIT, Harvard, Stanford. Earn certificates that employers recognize.
edX