Implement machine learning model lifecycle and operations
Production Model Monitoring and Retraining
CoreMonitor deployed models for data drift, prediction drift, data quality, feature attribution drift, and performance; watch operational metrics; and route threshold breaches to alerts or retraining.
Aligned to the live AI-300 guide, which publishes no skills-measured date; guide and product behavior verified October 10, 2026.
Why this matters
Models degrade silently as the world changes. Monitoring that compares production data with the right baseline, plus an automated path from an alert to retraining, keeps a model useful long after its first deployment.
Must Know
- Model monitoring signals include data drift, prediction drift, data quality, feature attribution drift, and model performance. Feature attribution drift and model performance are in preview.
- Data drift compares production inputs with a baseline, and prediction drift compares production outputs. Microsoft recommends training data as the baseline for data drift and data quality, and validation data for prediction drift.
- Data quality tracks null value rate, data type error rate, and out-of-bounds rate.
- Model performance monitoring requires ground truth data so predictions can be scored against actual outcomes.
- Online endpoints can collect production inference data automatically; for batch endpoints or models deployed outside Azure Machine Learning you must collect it yourself.
- Monitors run on a schedule. By default, when a threshold is exceeded, an alert email goes to the user who created the monitor.
- Model monitoring does not support the Allow only approved outbound managed network setting.
- Event Grid can react to a monitoring run status change tagged with a threshold breach and start a retraining pipeline through a handler such as Logic Apps or Azure Functions. Pipeline schedules themselves are time-based, not event-based.
Compare and Distinguish
- Data drift versus prediction drift: input distribution change versus output distribution change.
- Data quality versus data drift: broken or invalid values versus valid values whose distribution shifted.
- Model monitoring signals versus endpoint metrics: statistical model health versus operational latency, request rate, and resource use in Azure Monitor.
- Scheduled retraining versus event-driven retraining: a fixed recurrence versus training only when a breach or event occurs.
Scenario examples
- Scenario: Input feature distributions shifted after a product launch, but labels arrive weeks later. Think: data drift and prediction drift now, model performance once ground truth arrives.
- Scenario: A source system started sending nulls in a required column. Think: a data quality signal on null value rate.
- Scenario: Retraining should start only when drift exceeds its threshold. Think: Event Grid on the monitoring run status change filtered to threshold breaches, triggering a pipeline.
Exam traps
- Data drift detection does not need labels; model performance monitoring does.
- A pipeline schedule cannot be triggered by drift events by itself.
- Endpoint latency alerts do not reveal statistical drift in the inputs.
- Enabling Allow only approved outbound on the workspace prevents model monitoring from running.
Key takeaways
- Pick the signal that matches the failure: inputs, outputs, data validity, attributions, or accuracy.
- Use the recommended baselines and thresholds that reflect business tolerance.
- Close the loop with alerts and event-driven or scheduled retraining.
How it works
- A monitoring job computes metrics such as Jensen-Shannon distance or population stability index for each feature and compares them with thresholds.
- Event Grid delivers workspace events to subscribers, which can call a pipeline endpoint or submit a job.
Objects and administrative surfaces
- Monitoring schedules created in studio or with az ml schedule create and a monitor YAML file.
- Azure Monitor metrics and alerts for online endpoints and deployments.
- Event Grid subscriptions on the workspace with handlers such as Logic Apps, Functions, or webhooks.
When to use it
- Use scheduled retraining when data refreshes on a predictable cadence and drift-triggered retraining when change is irregular.
- Use Azure Monitor alerts for latency, errors, and saturation on endpoints.
Security and governance implications
- Only workspace Contributor or Owner can create Event Grid subscriptions, so assign that duty deliberately.
- Protect collected production data, which may contain personal information, with storage access controls.
Troubleshooting signals
- A monitor with no results may lack production data collection on the deployment.
- Spurious drift alerts often come from a baseline that does not represent normal production traffic.
More detail
- Detect and analyze data drift with suitable baselines and metrics.
- Monitor model performance and endpoint operational metrics.
- Configure alerts and retraining triggers for threshold breaches.
Ready for the quiz?
- Which signal needs ground truth?
- Which baseline is recommended for prediction drift?
- How can a drift breach start a retraining pipeline?
- Which managed network mode is incompatible with model monitoring?
Related objectives
- D2.4.S1 — Detect and analyze data drift
- D2.4.S2 — Monitor performance metrics of models deployed to production
- D2.4.S3 — Configure retraining or alert triggers when thresholds are exceeded