The AI Threat Landscape Beginner

Understanding the threat landscape is the foundation of effective threat modeling. This lesson maps the categories of attacks targeting AI systems, examines standardized threat taxonomies, and reviews real-world incidents that demonstrate the practical risks of deploying AI without adequate security measures.

Categories of AI Threats

AI threats can be organized into several broad categories based on the attacker's goal and the stage of the ML lifecycle they target:

Category	Attack Stage	Attacker Goal	Example
Evasion	Inference	Cause misclassification	Adversarial perturbations on images
Poisoning	Training	Corrupt model behavior	Injecting malicious samples into training data
Privacy	Inference	Extract sensitive information	Model inversion, membership inference
Theft	Inference	Steal model IP	Model extraction via API queries
Abuse	Deployment	Misuse AI capabilities	Generating deepfakes, automated attacks
Supply Chain	Development	Compromise dependencies	Backdoored pretrained models, poisoned datasets

OWASP Machine Learning Top 10

The OWASP ML Top 10 provides a standardized ranking of the most critical security risks for machine learning applications:

Rank	Risk	Description
ML01	Input Manipulation	Adversarial inputs crafted to cause incorrect predictions or bypass security controls
ML02	Data Poisoning	Corrupting training data to embed backdoors or degrade model accuracy
ML03	Model Inversion	Reconstructing training data or sensitive features from model outputs
ML04	Membership Inference	Determining whether specific records were used to train the model
ML05	Model Theft	Extracting or replicating a proprietary model via query access
ML06	AI Supply Chain	Compromised pretrained models, libraries, or datasets from third parties
ML07	Transfer Learning Attack	Exploiting vulnerabilities inherited from pretrained foundation models
ML08	Model Skewing	Manipulating data distribution to shift model behavior over time
ML09	Output Integrity	Tampering with model outputs or confidence scores after inference
ML10	Model Poisoning	Direct modification of model weights or architecture during deployment

MITRE ATLAS Framework

MITRE ATLAS (Adversarial Threat Landscape for AI Systems) extends the ATT&CK framework with AI-specific tactics and techniques. It organizes threats across the adversary lifecycle:

Key ATLAS Tactics

• Reconnaissance - Gathering information about the target AI system, its architecture, and training data
• Resource Development - Building adversarial tools, datasets, and surrogate models
• Initial Access - Gaining access to training pipelines, model registries, or inference APIs
• ML Attack Staging - Preparing adversarial inputs, poisoned data, or extraction queries
• ML Model Access - Querying the model to extract information or test adversarial inputs
• Exfiltration - Extracting model weights, training data, or sensitive predictions
• Impact - Degrading model performance, causing denial of service, or manipulating outputs

Real-World Attack Case Studies

Autonomous Vehicle Evasion

Researchers demonstrated that small stickers placed on stop signs could cause autonomous vehicle vision systems to misclassify them as speed limit signs. The perturbations were imperceptible to human drivers but consistently fooled deep learning models, highlighting the gap between human and machine perception.

Chatbot Manipulation

Microsoft's Tay chatbot was deployed on Twitter and within 24 hours was manipulated by coordinated users into generating offensive content. This demonstrated how adversarial users can exploit the learning mechanisms of AI systems through targeted interactions.

Model Extraction via API

Researchers showed they could replicate the functionality of cloud-based ML models by systematically querying their APIs and using the responses to train substitute models. The extracted models achieved near-identical accuracy at a fraction of the training cost.

Building AI Threat Intelligence

Effective threat modeling requires ongoing threat intelligence gathering. Key sources include:

• Academic research - ML security conferences (NeurIPS, ICML, IEEE S&P) publish new attack techniques regularly
• MITRE ATLAS case studies - Documented real-world incidents with tactics and techniques mapped
• CVE databases - Vulnerabilities in ML frameworks (TensorFlow, PyTorch, scikit-learn)
• Industry reports - Gartner, NIST, and ENISA publish AI security guidelines and threat assessments
• Bug bounty programs - Organizations running AI-specific bug bounties reveal common vulnerability patterns