Beginner

The Silent Failure

What model drift is, why it hides from conventional monitoring, and the real cost of catching it late in production AI systems.

The Model That Forgot How to Work

The team celebrated launch day. The model hit 94% accuracy in testing, latency was under 200ms, and the infrastructure scaled cleanly. Six months later, user satisfaction scores quietly dropped from 4.2 to 3.1 out of 5. Support tickets about "wrong answers" tripled. Nobody touched the model or the code. What happened?

Model drift happened. And it is the most underestimated reliability risk in production AI - not because it is hard to fix, but because it is hard to notice.

What Model Drift Actually Is

Model drift is the degradation of a deployed model's performance over time, caused by changes in the real world that the model was not trained to handle. The fundamental issue is a mismatch: the model learned patterns from historical data, but the present is different from that history in ways that matter for the task.

Three things can change to cause this mismatch:

• The inputs change. Users start phrasing queries differently, a data pipeline shifts format, or a new customer segment starts using the product. The model sees inputs it has never been optimized for.
• The world changes. The relationship between inputs and correct outputs shifts. A pricing model trained on 2023 economic conditions makes different recommendations than users now need. A sentiment model trained before a major cultural event misreads the language that emerged from it.
• The outputs drift. Even with stable inputs and a stable world, the model's output distribution can shift - especially in LLMs, where the same prompt can produce structurally different responses as the underlying model is updated by the provider.

Why Conventional Monitoring Misses It

Standard observability tools are built to detect infrastructure failures: service is up or down, latency is within threshold or it is not, error rates are acceptable or they spike. These checks are binary and fast. Drift is neither.

Drift is a gradual, statistical phenomenon. The model does not break - it just becomes progressively less right. A classification model that was 94% accurate drifts to 88%, then 81%, then 74%. At every step, it still returns a response. The HTTP status is 200. The latency is fine. The error rate is zero.

The only way to detect this is to compare what the model outputs to what it should output - and that requires either labeled ground truth (expensive, slow) or statistical proxies for output quality (learnable, automatable). This course teaches both.

The Real Cost of Late Detection

Drift caught in week one is a configuration tweak. Drift caught in month six is a business problem. The costs compound along three dimensions:

Drift Age	Business Impact	Fix Complexity
Days 1-7	Performance gap detectable in metrics, few users affected	Low - often a prompt update or data refresh
Weeks 2-4	User complaints start; NPS signals drop	Medium - may need recalibration or fine-tuning
Months 2-3	Support ticket volume rises; product reputation affected	High - full retrain often required
Months 4+	Trust damage sets in; feature usage drops measurably	Very high - retrain plus trust-recovery program

LLMs Drift Differently

Classical ML models drift because the data distribution shifts relative to training. LLMs do all of that, and then some. Because LLM providers regularly update underlying models - sometimes silently - a production LLM application faces a form of drift that has no equivalent in traditional ML: the model itself changes under you, even if your code does not.

A system prompt that reliably produced structured JSON six months ago may now produce trailing commentary. A few-shot example that anchored tone precisely may now be interpreted differently by a newer model checkpoint. Lesson 5 covers LLM-specific drift in detail.

The Drift Detection Gap in Most Organizations

Teams that have robust monitoring for classical software - CPU, memory, latency, error rate - often have almost no monitoring for model quality. The most common "monitoring" is user feedback, which catches drift only after users have already suffered its effects.

A 2025 industry survey (illustrative - check current vendor research for up-to-date statistics) found that the median time between drift onset and detection was 47 days for teams without automated drift monitoring, versus 3 days for teams with statistical baselines in place. That 44-day gap is the cost of not taking this course.

What This Course Covers

This course is the operational layer for any team running AI in production. You will learn:

• The three categories of drift - data, concept, and prediction - and what distinguishes them (Lesson 2)
• The four statistical methods used to detect drift reliably: PSI, KL divergence, KS test, and Jensen-Shannon (Lesson 3)
• How to build a four-layer monitoring stack and set alert thresholds that minimize both false positives and detection lag (Lesson 4)
• How LLMs drift in ways classical ML does not - and how to monitor each mode specifically (Lesson 5)
• A five-step root cause framework that distinguishes data drift from concept drift from evaluation drift (Lesson 6)
• The retrain/recalibrate/rollback decision matrix and when to use each path (Lesson 7)
• The 40-point drift prevention checklist and the five-level drift maturity model (Lesson 8)