Root Cause Analysis
A five-step framework for diagnosing whether drift came from your data, your model, your evaluation, or the world - so you pick the right fix on the first attempt.
Why the Wrong Diagnosis Is Expensive
A retrain takes days or weeks, consumes significant compute, and still requires a deployment cycle. If the actual problem was a broken data pipeline that started sending malformed inputs, a retrain changes nothing - the pipeline is still broken. The model learns from bad data and performs the same way. You spent two weeks for zero improvement.
Root cause analysis before remediation is not optional overhead. It is the only thing that ensures the fix you pick actually addresses the problem. The five steps below are structured to rule out cheaper causes first, escalating to more expensive ones only when necessary.
The Five-Step Drift RCA Framework
Before treating any performance change as model drift, verify the data pipeline is intact. Infrastructure problems frequently masquerade as drift because they change the distribution of inputs arriving at the model. Check: Have any upstream data sources changed schemas? Are there new NULL proportions or encoding changes? Has any ETL job failed silently and started sending default or stale values? Has the serving infrastructure been updated (new container image, new runtime version)?
Resolution time if positive: Hours to a day. Fix the pipeline. Recompute the drift metrics on clean data - most apparent drift disappears.
For LLM applications: did your provider update the model version, change default parameters, or modify safety filters during the period when drift appeared? For any model served by a third-party API: check the provider's changelog and status page. For internally served models: check the model registry for any recent deployments that may have gone out without proper announcement.
Resolution time if positive: Hours. Update the prompt to compensate, pin to a specific model version, or roll back to the previous version if one is available.
Run your PSI and KS detection metrics broken down by feature. Identify which specific features have shifted. Then segment your production inputs and examine whether the drift is concentrated in a specific user cohort, product category, geography, or time-of-day window. Localization tells you whether you have a broad distributional shift (entire population) or a segment-specific shift (e.g., mobile users only, a new enterprise customer, a recently launched geography).
This step tells you: Is the drift input-driven (data drift from a specific source) or broad (potentially concept drift)?
For the segment identified in Step 3, collect ground-truth labels for a sample of 100-500 recent predictions. Compare the model's predictions against these labels. Calculate accuracy (or your relevant metric) on this fresh labeled sample and compare it to the baseline accuracy at deployment.
What the result tells you: If accuracy on the new segment's inputs matches deployment-time accuracy, you have data drift without quality impact - the model is still correct, just being called on a different input distribution. If accuracy has dropped, you have either concept drift (the relationship changed) or the data drift is severe enough that the model cannot generalize to it.
Run the drifted model against your original, unchanged holdout test set from deployment time. If performance on this held-out set has not degraded, your model is unchanged - the drift is entirely environmental (data or concept drift in the input population). If performance has degraded on this fixed test set, you have a model regression - likely caused by a silent provider update, an infrastructure change, or (for internally served models) a deployment error.
The Diagnosis Decision Matrix
After running all five steps, map your findings to a root cause and recommended action:
| Input PSI | New-segment accuracy | Original test set accuracy | Root Cause | Recommended Fix |
|---|---|---|---|---|
| High (drift) | Stable | Stable | Data drift - input shift, model still correct | Monitor; no fix needed unless segment grows large enough to warrant retraining |
| High (drift) | Degraded | Stable | Data drift - inputs outside training distribution | Retrain on recent data including new segment |
| Stable | Degraded | Stable | Concept drift - world changed, inputs same | Retrain on recent labeled data; may need new features |
| Stable | Degraded | Degraded | Model regression - provider or deployment change | Roll back model version; investigate pipeline change |
| Stable | Stable | Stable | Evaluation drift - metric or label drift | Review evaluation methodology; check label collection process |
| Stable | Stable | Stable, but user complaints up | Proxy metric divergence - evaluation does not capture what users care about | Revise evaluation metrics; run user research |
Worked Example: Embedding Drift in a RAG System
A customer service team noticed that their LLM-powered FAQ assistant was producing less relevant answers starting approximately 6 weeks after a document index refresh. User satisfaction with responses dropped from 4.4 to 3.8 out of 5.
- Step 1: Data pipeline clean. The document ingestion pipeline was intact. New documents were being indexed correctly.
- Step 2: Checked provider changelog. The embedding model provider had silently updated the embedding model version during the 6-week window.
- Step 3: Retrieval recall@5 on the held-out eval suite had dropped from 88% to 71% across all query types - not segment-specific.
- Step 4: Fresh labels confirmed that queries were returning topically correct but not best-match documents.
- Step 5: Original test set (indexed documents only) performed at 91% - slightly above baseline. The problem was that old documents were indexed with the old embedding model and new documents with the new one, creating a mixed-model index where cosine similarity comparisons were not meaningful across the split.
- Root cause: Embedding model drift due to provider update + mixed-model index.
- Fix: Re-index all documents with the new embedding model. Implement version pinning for the embedding model to prevent silent future updates. Full re-index took 4 hours; recall@5 returned to 87%.
Ready to Go Deeper?
Live instructor-led courses from our partners. Affiliate disclosure.