mixle.inference.production.drift module

Model / data drift detection for production: is current data still the data the model was trained on?

Two complementary views:

  • Feature drift – per-field distribution shift between a reference (training) sample and a current (production) sample: Population Stability Index (PSI), Kolmogorov-Smirnov, and Jensen-Shannon. These are model-agnostic and operate on the schema’s fields.

  • Score drift – the model-native signal: the distribution of the model’s own log-density on current data versus on reference data. A fitted mixle model is the reference distribution, so if current data scores systematically lower (or its log-likelihood distribution shifts) the world has moved away from the model – exactly when to retrain.

detect_drift() combines both into a DriftReport with a single drift flag against thresholds, suitable for a monitoring loop (see mixle.inference.production.monitor.Monitor).

population_stability_index(reference, current, *, bins=10)[source]

PSI between two 1-D numeric samples (bin edges from the reference quantiles).

Rule of thumb: < 0.1 no shift, 0.1-0.25 moderate, > 0.25 significant.

Parameters:
Return type:

float

ks_statistic(reference, current)[source]

Two-sample Kolmogorov-Smirnov statistic in [0, 1] (larger = more shift).

Parameters:

  • reference (Any)

  • current (Any)

Return type:

float

js_divergence(reference, current, *, bins=20)[source]

Jensen-Shannon divergence (bits) between two 1-D numeric samples (shared histogram support).

Parameters:
Return type:

float

score_drift(model, reference, current)[source]

The model-native drift signal: how the model’s log-density distribution shifts from reference to current data. Returns the KS statistic between the two log-likelihood samples and their mean shift (mean current log-density minus mean reference; negative => current data is less likely under the model).

Both populations are materialized here exactly once, so a one-shot iterable is safe and the reported counts describe the records actually scored. Alongside the shift statistics the report carries the evidence behind them – how many records each population held and what fraction of each was unscorable – because a shift measured over a handful of surviving finite scores is not the same claim as one measured over the whole sample.

Parameters:
Return type:

dict

class DriftReport(drift, score, per_feature=<factory>, thresholds=<factory>, processed_count=0, reasons=<factory>)[source]

Bases: object

Drift decision, aggregate score, feature details, and thresholds.

reasons names every condition that raised the drift flag, so a verdict driven by lost scoring coverage is distinguishable from one driven by a genuine distribution shift.

The fields cannot be rebound (frozen=True) and the containers are copies of what was passed in, so the caller’s own objects cannot reach into a report after it is built. The containers themselves stay writable by whoever holds the report – see __post_init__().

Parameters:
validate_drift_thresholds(*, psi_threshold=0.25, ks_threshold=0.2, loglik_shift_threshold=-0.5, min_scorable_fraction=0.5, unscorable_shift_threshold=0.1)[source]

Refuse a threshold that cannot be met, or cannot fail.

Every comparison in detect_drift() is a bare > or < against one of these, so an unvalidated threshold decides the verdict silently: NaN makes every comparison False and suppresses drift entirely, and a negative psi_threshold flags drift on identical data, since PSI is non-negative by construction (P05-F16).

Shared with Monitor, which kept a weaker copy of this: it accepted a positive loglik_shift_threshold and a negative psi_threshold at construction and raised only from check()/update(), one call later than the mistake and on every call after (R06-F05).

Parameters:

  • psi_threshold (Any)

  • ks_threshold (Any)

  • loglik_shift_threshold (Any)

  • min_scorable_fraction (Any)

  • unscorable_shift_threshold (Any)

Return type:

None

detect_drift(model, reference, current, *, psi_threshold=0.25, ks_threshold=0.2, loglik_shift_threshold=-0.5, min_scorable_fraction=0.5, unscorable_shift_threshold=0.1, per_feature=True)[source]

Combine score drift and per-feature drift into a single DriftReport.

drift is flagged if the score-distribution KS exceeds ks_threshold, OR the mean log-likelihood drops by more than -loglik_shift_threshold (i.e. mean_loglik_shift < loglik_shift_threshold), OR any feature’s PSI exceeds psi_threshold, OR the current sample lost too much scoring coverage to support a no-drift verdict.

Coverage is part of the verdict, not just of the report. score_drift drops unscorable records before comparing distributions, so a current sample of one matching score plus nine records the model cannot score at all used to produce KS 0, mean shift 0, and drift=False – a confident “no drift” from 10% of the evidence. Two coverage conditions now fail closed:

  • fewer than min_scorable_fraction of current records scored finitely, and

  • the current unscorable rate exceeds the reference’s by more than unscorable_shift_threshold (records moving outside the model’s support IS the world moving away from the model).

Both populations are materialized once and reused for scoring and for the per-field columns, so a one-shot iterable is safe: consuming the stream twice previously left the feature pass with empty columns and manufactured a PSI-infinity DRIFT verdict out of two identical inputs.

Parameters:

  • model (Any)

  • reference (Any)

  • current (Any)

  • psi_threshold (float)

  • ks_threshold (float)

  • loglik_shift_threshold (float)

  • min_scorable_fraction (float)

  • unscorable_shift_threshold (float)

  • per_feature (bool)

Return type:

DriftReport