Data & model drift detection
Detect feature drift, prediction drift and concept drift using statistical tests (PSI, KS, chi-squared) and ML-based detectors calibrated to each model's sensitivity.
PSI
0.31
KS
0.08
χ²
p=0.41
Drift · performance · bias · retraining
Catch model drift before your users do.
ModelWatch monitors production ML models for feature drift, performance degradation, bias and anomalies, then alerts the right people with the feature, segment and threshold that tripped.
fraud-scorerv3.2 · prod · classification
Sample data
Feature drift (PSI vs. training baseline)
merchant_categorytxn_amountdevice_age_days
| feature | score | status |
|---|---|---|
| merchant_category | 0.31 | drift |
| txn_amount | 0.16 | watch |
| hour_of_day | 0.09 | ok |
| device_age_days | 0.06 | ok |
| country_code | 0.04 | ok |
AUC
0.912
−0.018
precision
0.874
−0.021
parity Δ
0.03
within 0.05
Alert feed
#ml-oncall · PagerDutyFeature drift: merchant_category PSI 0.31 > 0.20
09:42 · critical
Precision on segment new_users down 4.1 pts (7d)
09:15 · warning
Retraining trigger queued: fraud-scorer → pipeline #418
08:03 · action
v3.2 promoted; baseline window reset to 30d
Yesterday · info
Product preview with illustrative sample data.
- production models monitored across pilot workspaces
- 38
- production models monitored across pilot workspaces
- median time from drift onset to alert, pilot sample
- ~9 min
- median time from drift onset to alert, pilot sample
- predictions logged per day in our largest sample workspace
- 1.2M
- predictions logged per day in our largest sample workspace
- alert channels wired up on day one: email, Slack, PagerDuty
- 3
- alert channels wired up on day one: email, Slack, PagerDuty
Illustrative sample metrics from pilot workspaces — not audited results.
the problem
Models degrade quietly. Dashboards built for services don't notice.
A model can return 200 OK on every request while its inputs drift away from training data. ModelWatch watches the data and the predictions, not just the endpoint.
without monitoring
silent failure- Accuracy is checked in a notebook once a quarter
- Upstream schema change ships on Friday, nobody notices
- Bias reviews happen only before launch
- Retraining starts when a stakeholder complains
with ModelWatch
observable- Performance metrics tracked per version, segment and window
- Feature drift flagged within minutes, with the feature that moved
- Fairness thresholds checked continuously on live predictions
- Retraining pipelines triggered by thresholds you define, with an audit trail
capabilities
Platform details One place for every signal a model gives off.
Performance monitoring
Accuracy, precision, recall, F1, AUC and custom metrics across versions, segments and time windows, with degradation alerts.
Bias & fairness tracking
Watch for disparate impact, equalized odds and demographic parity across protected attributes, with thresholds you configure.
Retraining triggers
When drift or degradation crosses a threshold, trigger your retraining pipeline automatically, with a full audit trail of why it ran.
LLM monitoring
Token-level drift, response quality scoring, hallucination checks and cost tracking for LLM-backed features.
Fits the stack you run
SDKs for Python, Java and Go, plus connectors for your ML platform and on-call tools.
- MLflow
- SageMaker
- Vertex AI
- Databricks
- Kubeflow
- PagerDuty
- Slack
- Grafana
how it works
Instrument once. Monitor every version after.
- 1
Log predictions
Send predictions and features with the Python, Java or Go SDK, or batch-upload them. Ground-truth labels can arrive later.
- 2
Set a baseline
Use your training set or a production window as the reference distribution for every feature and output.
- 3
Choose monitors
Pick drift tests (PSI, KS, chi-squared), performance metrics and fairness checks, each with its own threshold.
- 4
Alert and act
Route alerts to email, Slack or PagerDuty, and optionally kick off a retraining pipeline when a threshold is crossed.
score.pypython
import modelwatch as mw
mw.init(project="fraud-scorer", version="v3.2")
mw.log_prediction(
features={"txn_amount": 182.40, "merchant_category": "5411"},
prediction=0.87,
prediction_id=txn.id,
)
# later, when the label is known
mw.log_actual(prediction_id=txn.id, actual=1)pricing
Compare plans Free for three models. Priced by models after that.
Starter
$0
For individual ML engineers monitoring side projects
- 3 models
- Basic drift detection
- Performance dashboards
- Email alerts
Team
most teams$149/mo
For ML teams shipping models to production
- Everything in Starter
- 50 models
- Advanced drift detection
- Bias & fairness monitoring
Enterprise
Custom
For organizations with large-scale ML deployments
- Everything in Team
- Unlimited models
- Custom drift detectors
- Full API access
Put your first model under watch today.
The Starter plan covers three models with drift detection, dashboards and email alerts.