The Drift Taxonomy

Data drift, concept drift, and prediction drift: what each one is, what triggers it, and what early signals betray it before users notice.

Why Naming the Type Matters

Not all drift is the same, and treating it as one thing leads to expensive mistakes. Applying a retrain when you actually have data pipeline drift wastes weeks and changes nothing. Missing concept drift because you were only watching input distributions lets quality degrade indefinitely. The right response depends entirely on which type you have - so naming it precisely is the first diagnostic step.

The standard taxonomy has three primary categories, each with subtypes. Most real-world drift incidents involve a combination, but understanding each in isolation first makes the combination much easier to diagnose.

Data Drift (Covariate Shift)

Data drift occurs when the statistical distribution of input features changes, but the underlying relationship between those inputs and the correct output stays the same. In formal terms: P(X) changes, but P(Y|X) - the conditional probability of the label given the input - remains stable. The model's learned mapping is still valid; the problem is that it is now rarely called on to make the predictions it learned best.

Subtypes of Data Drift

• Feature drift: Individual input features shift. A numerical feature that ranged 0-100 during training now regularly sees values up to 250 because a sensor was recalibrated.
• Categorical shift: New category values appear, or the proportion of existing categories changes. A product category that was 5% of training data is now 40% of live traffic after a marketing campaign.
• Temporal drift: Seasonal patterns, weekly cycles, or time-zone shifts create distributional differences that were not adequately represented in training data.

Detection signal:

Input feature distributions diverge from the training baseline. Population Stability Index (PSI) and Kolmogorov-Smirnov tests flag this directly. Lesson 3 covers both.

Concept Drift

Concept drift occurs when the underlying relationship between inputs and correct outputs changes - P(Y|X) shifts - even if the inputs themselves look the same. The world has changed, and what was the right answer before is no longer the right answer now. This is the most dangerous form of drift because it can be invisible to input-monitoring alone.

Subtypes of Concept Drift

• Sudden concept drift: An external event changes the correct answer abruptly. A regulatory change, a major product announcement, or a market event can all instantly invalidate what the model learned. Sudden drift shows up fast in outcome metrics if you have them.
• Gradual concept drift: Consumer preferences shift slowly. Language evolves. What users consider "good" or "relevant" changes over months. The most common form - and the hardest to detect without labeled ground truth.
• Recurring concept drift: Seasonal patterns that return. A model trained on holiday-season data over-serves seasonal buying behavior in January. This type is predictable and can be mitigated with seasonal recalibration cycles.

Detection signal:

Outcome metrics decline - accuracy, precision, recall, NDCG, user satisfaction scores. Concept drift requires either labeled ground truth or a strong proxy. For LLMs, evaluation harnesses with held-out examples can serve this role.

Prediction Drift (Output Drift)

Prediction drift is a shift in the model's output distribution, regardless of why. It is an effect, not a cause - it can result from data drift, concept drift, model updates, or interaction effects between all three. But it is often the first observable signal that something has changed.

Why monitor it even if it is just a symptom?

Because outcome labels take time to collect. Labeled ground truth for a document ranking model might only be available weeks after the predictions were made. But prediction distributions are available immediately, at every inference. Prediction drift monitoring is the fastest early-warning signal you can set up.

What prediction drift looks like:

• Classification: the proportion of class-1 predictions rises from 23% to 47% of all outputs
• Regression: mean predicted values shift upward; variance increases
• LLMs: output token count per request rises 40%; JSON outputs start including unexpected fields; sentiment of responses shifts toward more hedging language

Prior Probability Shift (Label Drift)

A fourth type worth knowing: the prevalence of outcomes in the real world changes, even if the model and the data relationship are both fine. A fraud detection model trained when fraud rates were 2% of transactions will become miscalibrated if fraud rates climb to 8% - not because the model forgot how to detect fraud, but because the base rate it was calibrated to has changed. Recalibration rather than retraining is usually the correct response.

The Combined View

LLM-Specific Drift Types (Preview)

LLMs add several drift modes that do not map cleanly to the classical taxonomy above. Prompt sensitivity drift occurs when the same prompt text produces structurally different outputs because the underlying model checkpoint changed. Knowledge staleness drift occurs when the model's training cutoff means it lacks facts that are now central to correct answers. Embedding drift occurs when retrieval systems built on top of one embedding model start returning degraded results after the embedding model is updated. Lesson 5 covers each of these in detail.