What is model drift in MLOps
MLOps, model drift refers to the phenomenon where a deployed machine learning model's performance degrades over time because the statistical properties of input data change. This causes the model to become less accurate or reliable, requiring monitoring and retraining to maintain effectiveness.Model drift is the gradual change in a machine learning model's predictive performance caused by shifts in data distribution or environment after deployment.How it works
Model drift occurs when the data your model sees in production changes from the data it was trained on. Imagine a weather prediction model trained on historical climate data; if the climate patterns shift due to global warming, the model's predictions become less accurate. This is similar to how a GPS app might become less reliable if the road network changes but the app isn't updated.
There are two main types of model drift: data drift (input features change) and concept drift (the relationship between inputs and outputs changes). Both cause the model to perform worse over time unless addressed.
Concrete example
Suppose you have a binary classification model predicting customer churn based on features like monthly usage and customer age. Over time, customer behavior changes due to a new competitor entering the market.
import numpy as np
from sklearn.metrics import accuracy_score
# Original training data distribution
X_train = np.random.normal(loc=0, scale=1, size=(1000, 2))
y_train = (X_train[:, 0] + X_train[:, 1] > 0).astype(int)
# Simulated model predictions on original data
y_pred_train = y_train # perfect prediction for example
# New production data with drift (mean shifted)
X_prod = np.random.normal(loc=1, scale=1, size=(200, 2))
y_prod = (X_prod[:, 0] + X_prod[:, 1] > 1).astype(int)
# Model still predicts using old threshold
y_pred_prod = (X_prod[:, 0] + X_prod[:, 1] > 0).astype(int)
# Accuracy drops due to drift
accuracy_before = accuracy_score(y_train, y_pred_train)
accuracy_after = accuracy_score(y_prod, y_pred_prod)
print(f"Accuracy before drift: {accuracy_before:.2f}")
print(f"Accuracy after drift: {accuracy_after:.2f}")Accuracy before drift: 1.00 Accuracy after drift: 0.75
When to use it
Use model drift detection and management in any production ML system where data or environment can change over time, such as fraud detection, recommendation engines, or demand forecasting. It is critical to monitor models continuously and retrain or update them when drift is detected.
Do not rely on static models without drift monitoring in dynamic environments, as this leads to degraded performance and poor user experience.
Key terms
| Term | Definition |
|---|---|
| Model drift | Performance degradation of a model due to changes in data or environment after deployment. |
| Data drift | Change in the input data distribution over time. |
| Concept drift | Change in the relationship between input features and target variable. |
| MLOps | Practices for deploying, monitoring, and maintaining ML models in production. |
Key Takeaways
- Model drift causes ML models to lose accuracy as real-world data changes after deployment.
- Detect drift by monitoring model performance and input data distribution continuously.
- Retrain or update models promptly when drift is detected to maintain reliability.