Intermediate

The Drift Prevention Checklist

40 checkpoints across pre-deploy baseline setup, monitoring configuration, alert response, and prevention hygiene - plus the five-level drift maturity model to know where your team stands.

✍️ AI School Editorial Team · Lilly Tech Systems 📅 Published Jul 30, 2026 · Reviewed Jul 30, 2026
✅
How to use this checklist: Run Part A at every new model deployment. Run Part B monthly to audit your monitoring configuration. Run Part C at the start of every drift incident. Run Part D quarterly as a prevention hygiene review. The maturity model at the end tells you which parts to prioritize based on where your team currently stands.

Part A: Pre-Deploy Baseline Setup (10 Checks)

These tasks must be completed before every model deployment. A model without a baseline cannot be monitored for drift.

📋
Pre-Deploy Checklist
  1. Snapshot input distributions. For every numerical input feature, compute and store: mean, std, min, max, p5, p10, p25, p50, p75, p90, p95, p99. For categorical features: value counts and proportions. Persist in your monitoring store with a deployment timestamp.
  2. Snapshot output distributions. Store the distribution of model outputs (score distributions for classifiers; value distributions for regressors; output token counts, format compliance rate, and mean embedding cosine similarity for LLMs) at deployment time.
  3. Save the evaluation set and its results. Archive the exact holdout evaluation set used to certify this deployment, along with the per-sample predictions. This is the anchor for Step 5 of the RCA framework and cannot be reconstructed later.
  4. Build or update the golden test suite. For LLM applications: maintain 50-200 representative prompts with reference outputs. Run the suite against the new deployment before going live. Store the baseline similarity scores.
  5. Document the model version explicitly. Record the exact model identifier, checkpoint hash, provider version string (for LLMs), training data date range, and any fine-tuning history. This is the artifact you roll back to if Step 2 of RCA finds a model regression.
  6. Set PSI alert thresholds on each critical feature. Using the baseline variance methodology from Lesson 4, compute calibrated warn (PSI > 0.1) and critical (PSI > 0.2) thresholds for each monitored feature. Document which features are business-critical (tighter thresholds) vs. informational (wider thresholds).
  7. Configure output distribution alerts. Set JS divergence alert thresholds for the prediction distribution. For classifiers: alert if any class proportion shifts by more than an absolute threshold. For LLMs: alert if format compliance rate drops below 99% or mean output token count shifts by more than 20%.
  8. Confirm rollback artifacts are in place. Before deploying the new model, verify that the previous model version is in the registry and can be redeployed in under 2 hours. For LLMs: confirm the previous provider model version string is recorded and pinnable. Rollback readiness = the most important 15 minutes you will spend before going live.
  9. Assign a drift owner. Name a specific person responsible for monitoring this deployment's drift metrics, receiving drift alerts, and initiating RCA when thresholds are breached. If there is no named owner, alerts will be ignored.
  10. Schedule the first drift review. Book a calendar slot 30 days after deployment for a first drift review: check PSI trends, confirm monitoring is working, and identify any early signals worth investigating. This meeting is also the time to collect and label a sample of recent predictions if ground truth is available.

Part B: Monthly Monitoring Configuration Audit (10 Checks)

Run these monthly to ensure your monitoring stack is still correctly configured and functioning.

📋
Monthly Monitoring Audit
  1. Verify that all drift metrics are actually computing. Check logs to confirm daily PSI, KS, and JS divergence jobs ran every day this month. Silent job failures are the most common reason teams discover monitoring was not working only when a drift incident occurs.
  2. Check that alert routing is current. Confirm that drift alerts are routed to the current on-call team, not a stale Slack channel or a person who left the organization three months ago.
  3. Audit baseline staleness. How old is the current baseline? If the model has been retrained and redeployed since baseline setup, confirm the baseline was also updated. A stale baseline will show perpetual "drift" against the new model's output distribution.
  4. Review the PSI trend for each critical feature. Plot PSI over the past 30 days. Are any features showing a slow, sustained upward trend that has not yet breached the alert threshold? Gradual drift is best caught by trend analysis, not point-in-time thresholds alone.
  5. Check prediction distribution trends. Have class proportions or output score distributions shifted in any consistent direction over the past 30 days? A sustained drift below the alert threshold is still drift.
  6. Run the golden test suite (LLM applications). Run the full golden test suite against the production endpoint and record the similarity scores. Compare to the baseline and to last month's scores. Flag any suite items where similarity has dropped by more than 5 points absolute.
  7. Check retrieval metrics (RAG applications). Run the held-out retrieval eval suite. Record recall@5 and NDCG@5. Compare to baseline and last month. Investigate if either drops by more than 3 points absolute.
  8. Verify ground truth collection is functioning. Check that your labeling pipeline, feedback signal collection, or outcome logging is producing records. Empty label queues for a month mean you are flying blind on concept drift.
  9. Check for embedding model version changes. Query your embedding model provider's changelog or version endpoint. Has the embedding model version changed since last month? If yes, run the retrieval eval suite comparison (pre- and post-change) and consider re-indexing.
  10. Review and close open drift investigation tickets. Any drift incidents flagged this month should have a resolution status. If any are still "investigating" after 14 days without a documented root cause hypothesis, escalate.

Part C: Drift Incident Triage (10 Checks)

Run at the start of every drift alert. This is the structured version of the five-step RCA from Lesson 6.

📋
Drift Incident Triage
  1. Timestamp onset. What is the earliest timestamp in the PSI/JS trend data when the current drift episode began? (Often earlier than the alert timestamp.)
  2. Check data pipelines immediately. Verify no upstream schema changes, ETL failures, NULL proportion shifts, or encoding changes occurred around the onset timestamp.
  3. Check provider or deployment changelog. Did any model update, infrastructure change, or safety filter update happen around the onset timestamp?
  4. Localize to a feature or segment. Which specific input features have elevated PSI? Which user segments, geographies, or product areas are affected?
  5. Collect 100-500 labeled examples from the drifted period. Even a small labeled sample is enough to distinguish data drift without quality impact from concept drift with quality impact.
  6. Run the original holdout evaluation set against the current model. Compare to baseline performance at deployment. Degraded on original holdout = model regression. Stable on original holdout = environmental drift.
  7. Classify the root cause. Using the diagnosis matrix from Lesson 6, assign one of: data drift, concept drift, model regression, pipeline failure, embedding drift, label drift.
  8. Select the response path. Using the decision matrix from Lesson 7: rollback, recalibrate, or retrain. Document which and why.
  9. Implement and verify the fix. After applying the response, recompute drift metrics and rerun the evaluation suite. Confirm metrics have returned to baseline levels.
  10. Write the post-incident record. Document: onset timestamp, detection lag, root cause, fix applied, verification method, and one preventive action to reduce recurrence risk.

Part D: Quarterly Prevention Hygiene (10 Checks)

Run quarterly to maintain long-term drift resilience. These are the strategic investments that reduce the cost and frequency of future incidents.

📋
Quarterly Prevention Review
  1. Review the incident log. How many drift incidents occurred this quarter? What were the root causes? Are 2-3 causes responsible for most incidents? Target the most frequent cause with a structural fix.
  2. Update the golden test suite. Add new test cases that reflect current real-world queries and use cases. Remove outdated test cases. The golden suite should reflect current user behavior, not launch-day assumptions.
  3. Evaluate the detection lag. For each incident this quarter, what was the gap between drift onset and detection? Track this metric over time. If detection lag is not improving, invest in either faster metric computation or better threshold calibration.
  4. Consider scheduled retraining cadence. For models in fast-moving domains (news, pricing, user behavior), evaluate whether a scheduled quarterly or monthly retrain (regardless of detected drift) would be more cost-effective than reactive retraining after incidents accumulate.
  5. Review feature importance stability. Run SHAP or permutation importance on a recent sample of production data. Compare to baseline feature importances. If the feature importance ranking has shifted significantly, the model may be relying on different signals than intended - a form of silent concept drift.
  6. Audit model version pinning. For every LLM application, confirm that all model version references are explicitly pinned (not floating). Any application using a floating model alias is at risk of a silent provider-update incident.
  7. Test rollback readiness. Conduct a rollback drill: simulate a critical drift incident and measure the time from alert to rollback completion. Target under 2 hours. If the drill reveals procedural gaps, fix them before the next real incident.
  8. Review monitoring coverage for new model deployments. Were all models deployed this quarter covered by a pre-deploy baseline setup (Part A)? If any were deployed without baselines, add them now.
  9. Update the drift owner matrix. Have any team or ownership changes occurred that affect drift response responsibilities? Update the owner matrix for all production models. Every model must have a named, current-employee drift owner.
  10. Assess maturity level progress. Using the maturity model below, where did the team start this quarter and where are they now? Set a specific target maturity level for next quarter and identify the single highest-priority gap to close.

The Five-Level Drift Maturity Model

Use this model to assess your team's current drift readiness and identify the most valuable next investment. Most teams fall between Level 1 and Level 2 on the day they start this course; the goal is Level 3 within one quarter and Level 4 within one year.

LevelNameWhat It Looks LikePrimary Gap
1UnawareNo drift monitoring. Drift is discovered when users complain or business metrics drop. Average detection lag: 30-60+ days.No baseline established; no detection running
2ReactiveBasic output monitoring or user feedback signals. Drift is detected but late. RCA is manual and slow. Average detection lag: 7-30 days.Inconsistent monitoring; no structured RCA; no rollback process
3ProactiveAll four monitoring layers running. Fixed baselines in place. PSI and JS alerts configured and routed. RCA uses the five-step framework. Rollback can be completed in hours. Average detection lag: 1-7 days.Golden test suite incomplete; recalibration not automated; incident records sparse
4SystematicFull monitoring coverage. Golden test suite updated quarterly. Automated recalibration for threshold drift. Rollback drills conducted. Detection lag consistently under 3 days. Incident trends drive preventive retraining decisions.Detection lag not yet under 24 hours; some LLM-specific drift modes not covered
5AdaptiveContinuous monitoring with sub-24-hour detection lag. Automated rollback for model regression incidents. Scheduled retraining cadences for fast-moving domains. Embedding and format monitoring fully implemented. Drift maturity review drives model roadmap decisions.Sustainable at this level; focus shifts to continuous improvement of specific metrics

Ten Rules to Remember

  1. No baseline, no monitoring. A deployed model without a statistical baseline cannot be monitored for drift. Establish baselines at deployment, not after.
  2. Compare to the fixed deployment baseline, not last week. Gradual drift is invisible in week-over-week comparisons.
  3. Input stability does not mean concept stability. Inputs can look identical while the correct output changes completely.
  4. PSI > 0.2 is a decision, not a suggestion. Escalate; do not watch and wait.
  5. Rollback before retrain. If the model regressed, rollback is faster, cheaper, and does not require labeled data.
  6. Never retrain on model outputs as labels. Using drifted model predictions as training targets bakes the drift permanently into the new model.
  7. Version every model alias explicitly. Floating aliases are how silent provider updates become midnight incidents.
  8. Name a drift owner before go-live. Unowned alerts are ignored alerts.
  9. Detection lag is a metric. Track it. Target under 3 days. Improve it quarterly.
  10. The best drift response is the one you practiced. Rollback drills and RCA simulations are what make the real incident fast and calm.
📚
You have completed the Model Drift course. The next step is wiring these practices into your broader production reliability story. See Production Readiness Runbook for LLM Systems for the pre-deploy, monitoring, and incident-response framework that wraps everything in this course, and Prompt Patterns That Survive Production for the prompt-level disciplines that reduce LLM drift exposure at the source.

Ready to Go Deeper?

Live instructor-led courses from our partners. Affiliate disclosure.

🧠

AI & ML Courses - 30% Off

Live instructor-led AI, machine learning, and data science courses. Use code Limited30.

Edureka
📊

DataCamp - AI & Data Science

Hands-on Python, ML, and AI courses with interactive exercises and real projects.

DataCamp
🎓

edX - Top AI Courses

University-level AI courses from MIT, Harvard, Stanford. Earn certificates employers recognize.

edX