Fundamentals of AI and ML
Model Drift and Monitoring
Production models can degrade after deployment when real-world data or relationships change. For AIF-C01, know how data drift, concept drift, and model drift differ, how SageMaker Model Monitor helps detect drift, and why retraining on current data is usually the remediation.
Concepts
- Data drift means input feature distributions shift away from the training data, such as customer demographics changing over time; teams detect it by monitoring production input distributions against a training baseline.
- Concept drift means the relationship between inputs and outputs changes, such as what counts as good credit risk shifting due to economic conditions, even when the input distributions stay the same.
- Model drift is an umbrella term for any degradation in model performance over time.
- How they differ: data drift means the inputs changed; concept drift means the mapping from inputs to outputs changed.
- AWS service: SageMaker Model Monitor detects data quality drift, model quality drift, bias drift, and feature attribution drift in deployed models; it compares live traffic against a training or validation baseline and can trigger Amazon CloudWatch alerts.
- Remediation means retraining the model on recent data that reflects the current distributions and relationships, not just scaling infrastructure.
Exam tips
- Data drift = inputs changed; concept drift = the input-to-output relationship changed.
- SageMaker Model Monitor watches deployed endpoints for data quality, model quality, bias, and feature attribution drift.
- When drift hurts quality, retrain with recent representative data instead of only increasing endpoint capacity.