mixle.evolve.verify module

The champion/challenger gate: “did the challenger SIGNIFICANTLY beat the champion, no regression?”

This is the single decision rule the whole loop turns on, and the anti-regression guarantee lives here. It is pure glue over existing statistics – it owns the policy (alpha, the effect-size floor, the calibration no-regression rule, multiplicity) and never edits the underlying tests:

  1. Paired significance – mixle.inference.model_comparison.paired_score_difference() on the objective’s per-observation vectors, scored from the same held-out batch in the same order.

  2. Pairing integrity – both vectors are required to have equal length (they are produced from one data argument, so the order cannot diverge); a mismatch is a hard error, not a silent skip.

  3. Effect-size floor – |mean_diff| >= min_effect; a statistically significant but practically negligible win does not promote.

  4. Non-nested cross-check – for family swaps (nonnested=True) the challenger must additionally win vuong_test() and clarke_test() (BIC-corrected) on the pointwise log-likelihoods.

  5. ELPD band – when LOO/WAIC pointwise arrays are supplied, compare_elpd()’s 2-SE band is the conservative tie rule.

  6. Calibration no-regression – a more-accurate-but-less-calibrated challenger is refused: its calibration scalar must not exceed the champion’s by calib_tol. Verdict. calibration_status is 'unavailable' only for an explicit, checked applicability decision – e.g. the calibration objective is PIT-based and needs a continuous predictive distribution, so it does not apply to a categorical/discrete model – since that is not itself evidence of a regression. An unexpected failure inside the computation itself (applicability was confirmed, yet it still raised) is 'error', not 'unavailable': an implementation defect must not silently read as calibrated, so 'error' blocks promotion the same as 'failed' – see _calibration_no_regression() and Verdict.calibrated.

  7. Multiplicity – comparing one champion/challenger pair produces exactly one p-value, so this function cannot correct it for multiplicity by itself: every method in mixle.inference.multiple_testing is the identity transform at family size 1 (bonferroni multiplies alpha by 1; BH ranks the one p-value against itself). A caller running many simultaneous challengers at once (e.g. one population generation) must pool the RAW p-values from every comparison and adjust them together, ONCE, via mixle.inference.multiple_testing.adjust_pvalues(), then compare each candidate’s own adjusted p-value to alpha itself – see mixle.evolve.population.Population.step(), which does exactly this. That pool must first drop any Verdict.p_value that is nan – see the scalar-only note below – adjust_pvalues() rejects non-finite input outright rather than silently mishandling it, so an un-filtered pool crashes instead of under-correcting.

  8. Scalar-only objectives – an objective whose pointwise returns None (e.g. calibration_objective(), decision_regret_objective()) has no per-observation vector to pair, so no paired test can run: Verdict.favored is still decided from a bare scalar-delta-vs-min_effect comparison (reported for a human to review), but Verdict. p_value/ci are set to nan as an explicit “not applicable” sentinel (there being no null hypothesis test to report a p-value or CI for) – not a failure to compute one. Downstream code that pools p-values across many verdicts (point 7 above) must treat this nan as “exclude from the pool”, not “coerce to 0 (very significant)” or “coerce to 1 (never significant)”: both would misrepresent a comparison that was never run as one that was. A bare scalar delta alone carries no sampling-uncertainty estimate – no replication, bootstrap, or resampling evidence backs it – so it cannot support the same evidence-bearing promotion guarantee a paired p-value gives: Verdict.promote is False for every scalar-only verdict regardless of favored (see Verdict.has_statistical_evidence). A scalar objective that gains its own documented resampling/replication contract (e.g. multiple independent fit replicates, or a bootstrap over the held-out data) would be a legitimate way to earn auto-promotion eligibility in the future; none does today, so a scalar-only win is flagged in evidence for a human to act on, not auto-promoted.

class Verdict(favored, delta, p_value, ci, calibration_status, evidence=<factory>)[source]

Bases: object

The outcome of a single champion/challenger comparison.

Parameters:
property calibrated: bool

True unless calibration regressed, or its computation broke unexpectedly.

Kept for callers that only need the promotion-relevant boolean; calibration_status carries the more precise state a bare bool cannot: ‘unavailable’ (calibration wasn’t requested, or is genuinely inapplicable to this model/data – an explicit, checked applicability decision, not itself evidence of a regression, so it does not block promotion, the same as ‘passed’) is distinct from both ‘failed’ (calibration was computed for both models and the challenger’s is worse by more than calib_tol) and ‘error’ (the computation raised unexpectedly despite looking applicable – an implementation defect, not a legitimate skip). ‘failed’ and ‘error’ are the only statuses that block promotion; an ‘error’ must not silently read the same as a pass.

property has_statistical_evidence: bool

True unless this verdict came from a scalar-only objective with no paired test to run.

A scalar-only objective (module docstring point 8) has no per-observation vector to pair, so p_value/ci are the explicit nan “not applicable” sentinel: there is no sampling-uncertainty estimate – no p-value, CI, replication, or bootstrap – behind the raw scalar delta at all. Automatic promotion requires this to be True; a scalar-only win is still reported (favored/delta/evidence['scalar_only']) for a human to review, but it can never auto-promote on its own – see promote.

property promote: bool

True iff the challenger is favored, calibration didn’t fail or error, and the verdict is backed by an actual statistical test (see has_statistical_evidence) – the single promotion predicate.

as_dict()[source]

Serialize the verdict into JSON-compatible primitive fields.

Return type:

dict[str, Any]

challenger_beats_champion(champion, challenger, data, *, objective, alpha=0.05, min_effect=0.0, require_calibration=True, nonnested=False, multiplicity=None, calib_tol=1.0e-3, seed=0, elpd_pointwise=None)[source]

Decide whether challenger significantly and non-regressively beats champion on data.

Parameters:

  • champion (Any) – two fitted models scored on the same held-out data.

  • challenger (Any) – two fitted models scored on the same held-out data.

  • data (Any) – the held-out responses (one batch -> both models scored in the same order).

  • objective (Objective) – the Objective to compare on.

  • alpha (float) – significance level for the paired test (and the CI is at 1 - alpha).

  • min_effect (float) – practical effect-size floor on |mean score difference|.

  • require_calibration (bool) – if True, run the calibration no-regression check.

  • nonnested (bool) – if True (a family swap), additionally require Vuong + Clarke to favor the challenger.

  • multiplicity (str | None) – must stay None. A single champion/challenger comparison produces exactly one p-value, so it can never be corrected for multiplicity in isolation – passing a method name here raises rather than silently no-op’ing (every mixle.inference.multiple_testing method is the identity transform at family size 1). Kept as a parameter only to fail loudly on the mistake instead of TypeError’ing on removal; pool the raw p-values from every simultaneous comparison and call mixle.inference.multiple_testing.adjust_pvalues() once yourself, then compare each adjusted p-value to alpha.

  • calib_tol (float) – tolerance on the calibration-error increase the challenger may carry.

  • seed (int) – RNG seed for the (sampled) calibration scalars.

  • elpd_pointwise (tuple[ndarray, ndarray] | None) – optional (pointwise_champion, pointwise_challenger) LOO/WAIC arrays; when given, the compare_elpd() 2-SE band is required to also favor the challenger.

Returns:

A Verdict. verdict.promote is the single promotion predicate; it is always False for a scalar-only objective, regardless of favored – see module docstring point 8 and Verdict.has_statistical_evidence.

Raises:

ValueError – if a policy value is outside its domain (see _check_policy()), or if a pointwise vector does not carry exactly one score per held-out row (see _check_pointwise()).

Return type:

Verdict