mixle.evolve.verify module

The champion/challenger gate: “did the challenger SIGNIFICANTLY beat the champion, no regression?”

This is the single decision rule the whole loop turns on, and the anti-regression guarantee lives here. It is pure glue over existing statistics – it owns the policy (alpha, the effect-size floor, the calibration no-regression rule, multiplicity) and never edits the underlying tests:

  1. Paired significance – mixle.inference.model_comparison.paired_score_difference() on the objective’s per-observation vectors, scored from the same held-out batch in the same order.

  2. Pairing integrity – both vectors are required to have equal length (they are produced from one data argument, so the order cannot diverge); a mismatch is a hard error, not a silent skip.

  3. Effect-size floor – |mean_diff| >= min_effect; a statistically significant but practically negligible win does not promote.

  4. Non-nested cross-check – for family swaps (nonnested=True) the challenger must additionally win vuong_test() and clarke_test() (BIC-corrected) on the pointwise log-likelihoods.

  5. ELPD band – when LOO/WAIC pointwise arrays are supplied, compare_elpd()’s 2-SE band is the conservative tie rule.

  6. Calibration no-regression – a more-accurate-but-less-calibrated challenger is refused: its calibration scalar must not exceed the champion’s by calib_tol.

  7. Multiplicity – when many challengers are tested at once, multiplicity adjusts alpha via mixle.inference.multiple_testing.adjust_pvalues() before the gate.

class Verdict(favored, delta, p_value, ci, calibrated, evidence=<factory>)[source]

Bases: object

The outcome of a single champion/challenger comparison.

Parameters:
favored: str
delta: float
p_value: float
ci: tuple[float, float]
calibrated: bool
evidence: dict
property promote: bool

True iff the challenger is favored and passed calibration – the promotion predicate.

as_dict()[source]
Return type:

dict[str, Any]

challenger_beats_champion(champion, challenger, data, *, objective, alpha=0.05, min_effect=0.0, require_calibration=True, nonnested=False, multiplicity=None, calib_tol=1.0e-3, seed=0, elpd_pointwise=None)[source]

Decide whether challenger significantly and non-regressively beats champion on data.

Parameters:

  • champion (Any) – two fitted models scored on the same held-out data.

  • challenger (Any) – two fitted models scored on the same held-out data.

  • data (Any) – the held-out responses (one batch -> both models scored in the same order).

  • objective (Objective) – the Objective to compare on.

  • alpha (float) – significance level for the paired test (and the CI is at 1 - alpha).

  • min_effect (float) – practical effect-size floor on |mean score difference|.

  • require_calibration (bool) – if True, run the calibration no-regression check.

  • nonnested (bool) – if True (a family swap), additionally require Vuong + Clarke to favor the challenger.

  • multiplicity (str | None) – None or a mixle.inference.multiple_testing.adjust_pvalues() method ('bonferroni' / 'bh' / …) when this is one of many simultaneous challengers – adjusts the p-value before the gate.

  • calib_tol (float) – tolerance on the calibration-error increase the challenger may carry.

  • seed (int) – RNG seed for the (sampled) calibration scalars.

  • elpd_pointwise (tuple[ndarray, ndarray] | None) – optional (pointwise_champion, pointwise_challenger) LOO/WAIC arrays; when given, the compare_elpd() 2-SE band is required to also favor the challenger.

Returns:

A Verdict. verdict.promote is the single promotion predicate.

Return type:

Verdict