Advanced

Data Poisoning Detection

Learn techniques to identify poisoned training data and backdoored models using statistical analysis, spectral methods, activation clustering, and reverse-engineering approaches.

Detection Approaches Overview

MethodDetectsWhen to Use
Statistical AnalysisData anomaliesBefore training - screen incoming data
Spectral SignaturesPoisoned samplesAfter training - analyze learned representations
Activation ClusteringBackdoorsAfter training - examine model internals
Neural CleanseTrigger patternsAfter training - reverse-engineer triggers
STRIPTriggered inputsAt inference - real-time detection

Spectral Signature Detection

Poisoned samples often leave a detectable signature in the model's learned representation space:

Python - Spectral Signature Detection
import numpy as np
from sklearn.decomposition import PCA

def detect_spectral_signatures(representations, labels, epsilon=1.5):
    """Detect poisoned samples via spectral signatures."""
    suspicious = []

    for target_class in np.unique(labels):
        # Get representations for this class
        class_reps = representations[labels == target_class]
        centered = class_reps - class_reps.mean(axis=0)

        # Compute top singular vector
        _, _, Vt = np.linalg.svd(centered, full_matrices=False)
        top_vector = Vt[0]

        # Score each sample by projection onto top vector
        scores = np.abs(centered @ top_vector)

        # Flag outliers as potentially poisoned
        threshold = np.mean(scores) + epsilon * np.std(scores)
        poisoned_idx = np.where(scores > threshold)[0]
        suspicious.extend(poisoned_idx.tolist())

    return suspicious

Neural Cleanse

Neural Cleanse reverse-engineers potential trigger patterns by finding the smallest input perturbation that causes all samples of one class to be classified as another:

Neural Cleanse Algorithm
# For each target class t:
For each source class s != t:
    # Find minimum trigger that flips s -> t
    trigger = optimize(
        minimize: |trigger_mask| + |trigger_pattern|
        subject_to: model(x + trigger) == t for all x in class s
    )
    record trigger size

# Backdoored class has anomalously small trigger
If any class has trigger_size << median(all trigger_sizes):
    ALERT: Likely backdoor detected for that class
    The recovered trigger approximates the actual backdoor trigger

Activation Clustering

Backdoor samples create a distinct cluster in the model's activation space:

  • Extract activations from an intermediate layer for all samples of each class
  • Apply dimensionality reduction (PCA, t-SNE)
  • Run clustering (k-means with k=2) on each class
  • If a class has two well-separated clusters, the smaller cluster likely contains poisoned samples

STRIP: Runtime Detection

STRIP (STRong Intentional Perturbation) detects triggered inputs at inference time:

  • Overlay the input with random clean images/text
  • If the prediction stays constant despite perturbation, the input likely contains a trigger
  • Clean inputs change prediction when perturbed; triggered inputs maintain the backdoor label
Layer your defenses: No single detection method catches all poisoning attacks. Use pre-training data screening, post-training model analysis, and runtime input validation together for comprehensive coverage.

Ready to Go Deeper?

Live instructor-led courses from our partners. Affiliate disclosure.