STRIDE for AI Systems Intermediate

STRIDE is one of the most widely adopted threat modeling frameworks in software security. Originally developed by Microsoft, it categorizes threats into six types. This lesson extends each STRIDE category with AI-specific threats, providing a systematic approach to identifying vulnerabilities in machine learning systems.

STRIDE Categories Applied to AI

Spoofing in AI Systems

In AI contexts, spoofing extends beyond identity impersonation to include input spoofing - crafting inputs that appear to be one thing but are interpreted differently by the model.

• Adversarial examples that look like one class to humans but are classified as another by the model
• Deepfake audio/video used to bypass biometric authentication systems
• Synthetic data injection that mimics legitimate training data to manipulate model behavior

Tampering with AI Systems

Tampering is arguably the broadest STRIDE category for AI, encompassing any unauthorized modification to data, models, or pipelines:

• Data poisoning - Modifying training data to embed backdoors or bias the model
• Model tampering - Directly altering model weights, architecture, or configuration files
• Pipeline tampering - Compromising data preprocessing, feature engineering, or augmentation steps
• Label flipping - Changing ground truth labels in the training set to degrade or redirect model behavior

Repudiation in AI Systems

AI systems often lack the audit trails needed to trace decisions back to their causes:

• Untraceable predictions - No record linking a specific input to the model version, features, and weights that produced the output
• Training data provenance - Inability to prove which data was used to train a model version
• Model versioning gaps - Insufficient tracking of model changes over time

Information Disclosure in AI

AI models can inadvertently leak sensitive information about their training data:

• Model inversion - Reconstructing training data from model outputs (e.g., reconstructing faces from a facial recognition API)
• Membership inference - Determining whether a specific record was in the training set
• Training data extraction - Prompting LLMs to reveal memorized training data including PII, API keys, or proprietary content
• Gradient leakage - In federated learning, gradients shared during training can reveal individual data points

Denial of Service for AI

AI-specific DoS attacks exploit the computational intensity of ML inference:

• Sponge examples - Inputs designed to maximize inference time and resource consumption
• Model degradation - Gradually poisoning a continually-learning model until it becomes useless
• Resource exhaustion - Triggering expensive model operations like beam search with pathological inputs

Elevation of Privilege in AI

In AI systems, privilege escalation takes on new forms:

• Prompt injection - Crafting inputs that cause an LLM to ignore its system instructions and follow attacker commands
• Jailbreaking - Bypassing safety filters and content policies in AI assistants
• Tool abuse - Manipulating AI agents into using their tools (file access, code execution, API calls) for unauthorized purposes

STRIDE Analysis Worksheet

For each AI component in your system, work through this analysis:

Component:     [Name of AI component]
Description:   [What it does, what data it processes]

SPOOFING:
  - Can inputs be crafted to appear as something they are not?
  - Can the model's identity or provenance be faked?

TAMPERING:
  - Can training data be modified without detection?
  - Can model weights or config be altered?

REPUDIATION:
  - Are all predictions logged with full context?
  - Can model version and data lineage be verified?

INFORMATION DISCLOSURE:
  - Can sensitive training data be extracted from outputs?
  - Do confidence scores leak information?

DENIAL OF SERVICE:
  - Can adversarial inputs cause excessive computation?
  - Can the model be degraded through poisoned feedback?

ELEVATION OF PRIVILEGE:
  - Can inputs bypass safety filters or content policies?
  - Can the model be manipulated to access unauthorized resources?