Advanced

Retrain, Recalibrate, or Rollback

The three-path drift response framework and the decision matrix that tells you which path to take - before the incident turns into a crisis.

The Three Paths

Once root cause analysis has identified the type of drift (Lesson 6), you face a decision: what do you actually do about it? The answer is almost always one of three things - retrain, recalibrate, or rollback - but the wrong choice wastes weeks and sometimes makes the situation worse. This lesson gives you the decision matrix to pick correctly on the first attempt.

Path 1: Rollback

Rollback is the fastest response and the correct choice when the root cause is a model regression - something about the model itself changed and quality degraded. Rollback means reverting to the most recent known-good version of the model (or model configuration) that predates the quality drop.

When rollback is the right choice:

• RCA Step 5 showed degraded performance on the original test set (indicating the model changed, not just the data)
• A provider model version update correlates with the onset of the quality drop
• A recent internal model deployment introduced a regression
• Quality dropped suddenly (not gradually), pointing to a discrete event rather than gradual drift

What rollback looks like in practice:

• LLM API applications: Pin the model alias to a specific date-versioned endpoint. Contact the provider to confirm version history if needed.
• Internally served models: Use your model registry to identify and redeploy the previous checkpoint. A proper MLOps setup should make this a one-command operation; if it takes more than a few hours, add automated rollback to your roadmap.
• RAG/embedding systems: Re-index your document corpus with the prior embedding model version. Maintain a copy of the previous vector index to enable fast rollback without re-computation.

Time to resolution: hours (target) to 1 day.

Path 2: Recalibrate

Recalibration is the middle path: it adjusts a model's decision thresholds or output scaling without touching the underlying weights. It is the correct response when the model's learned mapping is still valid, but the base rates or output calibration have shifted.

When recalibration is the right choice:

• Label drift (prior probability shift): the prevalence of outcomes changed, but the model's discrimination ability is intact
• Confidence miscalibration: model outputs probabilities that no longer match actual frequencies (a model that says 80% confidence is correct 60% of the time)
• Threshold drift: the decision threshold optimized at deployment no longer balances precision and recall appropriately for current business needs

Recalibration techniques:

• Platt scaling: Train a logistic regression on top of the model's raw probability outputs using a small set of recent labeled examples. Fast, requires only 200-500 labeled points.
• Isotonic regression: A non-parametric alternative to Platt scaling; more flexible but requires more labeled data (typically 1,000+ points).
• Threshold optimization: Re-run your operating point selection (precision-recall tradeoff) on a fresh validation set and move the classification threshold accordingly.
• For LLMs - system prompt recalibration: Adjust system prompt instructions to compensate for output style drift. Add explicit format anchoring to counteract format decay. This is the fastest recalibration path for LLM applications.

Time to resolution: hours to 3 days.

Path 3: Retrain

Full retrain is the most expensive response and the correct choice when the underlying learned mapping is no longer valid - when concept drift has made the model's understanding of the input-output relationship obsolete, or when data drift has moved inputs so far outside the training distribution that the model cannot generalize.

When full retrain is the right choice:

• RCA confirmed concept drift: the world changed, and what the model learned is no longer correct
• Data drift is severe enough that recalibration cannot compensate: new user segments or input formats the model has never seen
• A new capability or domain has been added to the application that requires the model to understand things it was never trained on

Two retrain strategies:

Strategy	Data Used	When to Use	Cost
Full retrain	All historical + recent data	Concept drift; model catastrophically wrong	High (GPU hours, weeks of data collection)
Incremental / fine-tune	Recent data only (sliding window)	Gradual drift; model partly right but stale	Medium (10-20% of full retrain cost)

For LLMs - the fine-tuning alternative:

For LLM-backed applications, full retraining of the base model is not an option (you do not have access to the weights). The LLM equivalent of retraining is fine-tuning on recent examples through the provider's fine-tuning API, or switching to a newer base model that has a more recent training cutoff. The faster, cheaper alternative in most cases is RAG refresh: update the document corpus the LLM retrieves from, rather than changing the model itself.

Time to resolution: days to weeks.

The Decision Matrix

Root Cause	First Attempt	If First Attempt Insufficient
Provider/model update causing regression	Rollback to prior version	Recalibrate system prompt; escalate to provider
Internal deployment regression	Rollback in model registry	Debug training pipeline; retrain with fixed data
Data pipeline failure/corruption	Fix pipeline; monitor	Recalibrate if residual degradation
New input segment (data drift)	Recalibrate thresholds	Incremental retrain on new segment data
Concept drift (world changed)	Recalibrate if minor; retrain if severe	Full retrain with recent labeled data
Label drift (base rate shift)	Recalibrate threshold	Update prior probabilities; partial retrain
Embedding drift (RAG)	Re-index with current embedding model	Pin embedding model version; evaluate model switch
LLM output format decay	Update system prompt with stricter format anchoring	Add output parsing with retry; consider fine-tuning
Knowledge staleness drift	Refresh RAG document corpus	Switch to newer base model with more recent cutoff

Verifying the Fix

Whatever path you take, the fix is not complete until you have verified it worked. Verification requires: