Introduction to AI Threat Modeling Beginner

AI systems face a unique set of security challenges that traditional threat modeling does not address. From adversarial inputs that fool classifiers to poisoned training data that corrupts models, the attack surface of AI extends far beyond conventional software. This lesson introduces the foundational concepts, motivations, and frameworks for modeling threats against AI systems.

Why AI Systems Need Specialized Threat Modeling

Traditional software threat modeling focuses on well-understood attack vectors: SQL injection, buffer overflows, authentication bypasses. AI systems inherit all of these risks and introduce entirely new categories of threats rooted in the statistical nature of machine learning.

Consider a fraud detection model deployed at a bank. Beyond the standard web application threats, attackers can:

• Craft adversarial transactions that appear normal to the model but are actually fraudulent
• Poison the training data by injecting subtle patterns that cause the model to misclassify future fraud
• Extract the model by querying the API repeatedly and building a replica
• Infer private data by probing the model to reveal information about its training set

What is AI Threat Modeling?

AI threat modeling is the systematic process of identifying potential threats to AI and ML systems, evaluating their likelihood and impact, and determining appropriate countermeasures. It extends traditional threat modeling with AI-specific considerations:

Core Threat Modeling Frameworks

Several frameworks have been adapted or created for AI threat modeling. Here are the most widely used:

1. STRIDE for AI

Microsoft's STRIDE framework - Spoofing, Tampering, Repudiation, Information Disclosure, Denial of Service, Elevation of Privilege - can be extended to cover AI-specific threats. We will explore this in depth in Lesson 3.

2. OWASP Machine Learning Top 10

The OWASP ML Top 10 identifies the most critical security risks for machine learning systems:

1. ML01 - Input Manipulation

   Adversarial inputs designed to cause misclassification or unexpected behavior.

2. ML02 - Data Poisoning

   Corrupting training data to introduce backdoors or degrade model performance.

3. ML03 - Model Inversion

   Extracting sensitive information about training data from model outputs.

4. ML04 - Membership Inference

   Determining whether specific data points were used in training.

5. ML05 - Model Theft

   Stealing or replicating a model through API queries or direct access.

3. MITRE ATLAS

MITRE's Adversarial Threat Landscape for AI Systems (ATLAS) provides a knowledge base of adversary tactics and techniques based on real-world attack observations. It follows the same structure as the MITRE ATT&CK framework but focuses on AI-specific threats.

The AI Threat Modeling Process

A structured AI threat modeling process typically follows these phases:

1. Define the System

   Document the AI system architecture: data sources, training pipelines, model types, serving infrastructure, APIs, and downstream consumers.

2. Identify Assets

   Catalog what needs protection: trained models, training data, feature engineering code, API keys, model weights, and intellectual property.

3. Enumerate Threats

   Use frameworks like STRIDE or ATLAS to systematically identify potential threats for each component and data flow.

4. Assess Risk

   Evaluate each threat using a risk matrix that considers likelihood, impact, exploitability, and detectability.

5. Plan Mitigations

   Design countermeasures for high-priority threats. Balance security controls with model performance and usability.

# Threat Model Entry Template
Threat ID:    TM-AI-001
Category:     Input Manipulation (STRIDE: Tampering)
Asset:        Image Classification Model (Production)
Description:  Adversarial perturbations to input images cause
              misclassification while appearing normal to humans
Likelihood:   High (public tools available, API accessible)
Impact:       High (safety-critical application)
Risk Level:   Critical
Mitigation:   Input preprocessing, adversarial training,
              confidence thresholds, human review for edge cases
Status:       In Progress