mixle.inference.calibration_gate module

Calibration gate – a deployable IC-6 verifier that challenges whether a posterior’s reported uncertainty is actually earned, instead of checking only its shape.

Motivation (found empirically, across several end-to-end pipelines): the UQ in this codebase is locally honest – every module computes a real posterior – but was globally unaudited. A systematically overconfident or miscalibrated posterior sailed straight through to a confident-looking final answer, because nothing in the flow ever challenged the confidence. The calibration primitives already existed (mixle.inference.calibration: pit_ensemble / interval_coverage / coverage_curve / pit_calibration_error); what was missing was a gate that composes them into a pass/fail verifier a pipeline (e.g. mixle.task.knowledge_routing.route_task()) can actually stop on. This module is that gate – a thin composition, not new statistics.

Honest boundary, stated up front rather than buried. This gate catches:

  • overconfidence / underconfidence of a posterior relative to HELD-OUT DATA it can be checked against (posterior_predictive_calibration()), and

  • inference-algorithm bugs – an inference that is not self-consistent under its own generative model – via simulation-based calibration (simulation_based_calibration()).

It does not catch model misspecification under genuine non-uniqueness. If the data you can hold out simply cannot distinguish the true state (the textbook example: surface gravity cannot resolve a source’s depth – shallow and deep bodies produce nearly identical surface fields), then a biased posterior fits the held-out data perfectly and this gate will correctly pass it. Catching that requires either data that can see the biased dimension (e.g. one borehole) or a physics-based prior – not more calibration checking. This gate reports what the available data can support; it never manufactures confidence the data cannot justify, and it never claims to.

class CalibrationVerdict(calibration_status, pit_error, null_threshold, coverage_error, reference_level, coverage_at_reference, mean_interval_width, n_points, power_sufficient=True, expected_count_per_bin=nan, randomness_controlled=True, reasons=<factory>, kind='calibration', p_value=nan, n_null=0, power_estimate=nan, power_alternative='', power_estimate_independent=nan, power_estimate_shared=nan, power_lower_bound=nan, power_replicates=0, null_rejection_ucb=nan, declared_dependence='', coverage_at_reference_null_expectation=nan)[source]

Bases: object

The outcome of a calibration check with an explicit three-way decision state.

The DECISION is a Monte-Carlo p-value compared to 1 - null_quantile (STAT-RR17-08): the predictive gate uses a column-swap randomization null that is exact in exactly TWO regimes – the shared-draw ensemble construction (any cross-row dependence of the held-out data), and fully independent rows AND entries – and in no other: dependent rows with fresh entries break the vector exchangeability and the test false-alarms (measured 83.9% at nominal 5%, STAT-RR19-06/STAT-RR21-10; an earlier version of this docstring claimed exactness “whatever the cross-row dependence”). SBC uses an i.i.d.-uniform null, which its protocol genuinely satisfies. Both use (1 + count)/(1 + B), level-valid at any replicate count. null_threshold remains as a descriptive scale and the score anchor, never the decision constant (deciding against that estimated quantile realized levels of 0.0033-0.0181 at nominal 0.01 depending only on its seed). calibration_status is 'passed' only when the p-value clears alpha AND measured power against the named canonical alternative is adequate AND randomness is controlled; 'indeterminate' means the data cannot support either promotion or rejection; 'failed' means the decision rule rejected, which is real evidence of a problem even when power was otherwise low. A gate may promote only the literal 'passed' state.

Parameters:

  • calibration_status (Literal['passed', 'failed', 'indeterminate'])

  • pit_error (float)

  • null_threshold (float)

  • coverage_error (float)

  • reference_level (float)

  • coverage_at_reference (float)

  • mean_interval_width (float)

  • n_points (int)

  • power_sufficient (bool)

  • expected_count_per_bin (float)

  • randomness_controlled (bool)

  • reasons (list[str])

  • kind (str)

  • p_value (float)

  • n_null (int)

  • power_estimate (float)

  • power_alternative (str)

  • power_estimate_independent (float)

  • power_estimate_shared (float)

  • power_lower_bound (float)

  • power_replicates (int)

  • null_rejection_ucb (float)

  • declared_dependence (str)

  • coverage_at_reference_null_expectation (float)

property passed: bool

Whether this result is affirmative evidence suitable for promotion.

property indeterminate: bool

Whether the check lacks enough evidence to pass or fail.

property low_power: bool

Compatibility diagnostic for indeterminate results caused by inadequate power.

property score: float

1 at zero PIT error, 0.5 exactly AT the calibrated-null threshold, 0 at twice the threshold, linear in between. Normalized by the null threshold so it is comparable across sample sizes. It is NOT 1 for every passing posterior – a genuinely calibrated posterior typically lands around 0.6-0.7, so rank with it, never gate on a fixed cutoff like 0.9 (use calibration_status to gate).

Type:

A 0..1 ranking score

posterior_predictive_calibration(ensemble, held_out_y, *, reference_level=0.90, null_quantile=0.99, pit_tol=None, bins=10, low_power_threshold=1.0, min_expected_count_per_bin=5.0, pit_seed=0, ensemble_dependence='shared-draws')[source]

Check a posterior-predictive ensemble against held-out observations it never saw.

ensemble_dependence declares how the ensemble AND the held-out rows were BUILT. The column-swap p-value is exact in exactly TWO regimes, and the declaration selects which executed power gates promotion:

  • "shared-draws" (the default, fail-safe): ONE set of m posterior draws pushed through every row’s forward model. Exact under ANY cross-row dependence of the held-out data – the columns then carry the same dependence the observations do – which is why this is the construction to use when rows share a fitted posterior or latent state (the typical single-fit evaluation). Power is often low here and promotion honestly refuses.

  • "independent": the caller’s recorded assertion that the ROWS THEMSELVES are mutually independent (no latent shared across held-out points) AND every ensemble entry is a fresh draw. BOTH halves are load-bearing: with rows sharing a latent but entries drawn fresh – a construction a caller might reasonably call “independent draws” – the swap group is NOT an invariance and the test false-alarms on perfectly calibrated data (measured 83.4-84.3% rejection at alpha = 0.05, k = 100, m = 20; STAT-RR19-06). If the held-out rows share anything, do not declare this: rebuild the ensemble with the shared-draw construction, which is exact regardless.

The declaration is recorded in the verdict and the receipt; misdeclaring it is how a low-power non-rejection masquerades as affirmative evidence (pass 18 measured a shared-column mis-calibrated posterior promoted on an independent-rows power fiction of 0.95 against an actual 0.018) or how a dependent-row construction inherits a false-alarm rate of 84% (pass 20).

ensemble is (k, m): for each of k held-out points, m posterior-predictive draws (push m draws from the posterior through the forward model to the observation of each held-out point). held_out_y is the (k,) array of real observed values at those points.

The decision is an exact column-swap randomization test on the randomized PIT of the held-out data (STAT-RR17-08): the observation vector is swapped with each whole ensemble column in turn and the identical statistic recomputed, giving p = (1 + count)/(1 + m). Under calibration, (y, col_1..col_m) are exchangeable as vectors for the shared-draw construction (the same m posterior draws pushed through the forward model for every point – the i.i.d.-uniform reference this replaces wrongly failed a correctly calibrated shared-parameter posterior 298/300 times there) and for FULLY independent data (rows and entries both), so the p-value is level-exact in those two regimes – and in NO OTHER: dependent rows with fresh entries break the exchangeability and the test false-alarms (STAT-RR19-06; see ensemble_dependence above). It is quantized at 1/(m+1): with few draws per point the test cannot reject at small alpha, and the measured power reports that honestly. The decision rejects when p <= 1 - null_quantile. null_threshold is still computed as a descriptive scale and the score anchor, never the decision constant. Pass pit_tol to override with a fixed tolerance if you have a reason to (that branch reports no p-value).

Coverage numbers (coverage_curve() / interval_coverage()) are computed and reported as human-readable diagnostics and to label the direction of any miscalibration (over- vs under-confident), but the pass/fail itself is the PIT test, which subsumes them and is scale-correct.

A non-rejection is promotable only when MEASURED power is adequate: the Monte-Carlo power of this decision at this k against the named canonical alternative (predictive dispersion 0.8x the truth) must reach 0.50, alongside the min_expected_count_per_bin data floor. The previous bin-count/threshold-magnitude heuristics computed no power at all and reported power_sufficient=True where the actual rejection rate against that alternative was ~4.5% (STAT-RR17-08). Power against ONE named alternative is a floor, not a universal power claim. Otherwise the result is indeterminate, never a pass.

Parameters:
Return type:

CalibrationVerdict

simulation_based_calibration(prior_sampler, simulate, fit, *, n_sims=200, param_index=0, null_quantile=0.99, error_tol=None, bins=10, low_power_threshold=1.0, min_expected_count_per_bin=5.0, seed=0)[source]

Simulation-based calibration (Talts et al. 2018): does the inference recover its own generative model? Draw theta ~ prior, simulate y ~ p(y|theta), refit a posterior, and record the rank of the true theta among the posterior draws. Under a correct inference those ranks are Uniform; a systematically over/under-dispersed posterior makes them pile up at the middle or the edges.

PRECONDITION (STAT-RR17-08 / audit GATE-6): rank uniformity requires the posterior draws each fit returns to be (near-)independent – exchangeability of theta_true with the draws. An UNTHINNED autocorrelated MCMC chain from a CORRECT inference produces systematically non-uniform ranks and fails this gate without any defect in the inference; thin the chain to approximate independence first (Talts et al., section 4). The decision itself is the level-exact Monte-Carlo p-value against the i.i.d.-uniform null, which the SBC protocol genuinely satisfies across simulations (each draws its own parameters and data).

prior_sampler(rng) -> theta (a (d,) parameter vector, or scalar-as-(1,)); simulate(theta, rng) -> y (any shape the fitter accepts); fit(y, rng) -> posterior_draws ((n_draws, d) or (n_draws,)). The fitter must derive all of its stochasticity from the supplied RNG; accepting an opaque fit(y) callable would make seed unable to control the inference being calibrated. param_index selects which parameter’s rank to test.

Unlike posterior_predictive_calibration(), this needs no held-out real data – it tests the inference machinery against synthetic ground truth it generates itself. It therefore catches a different class: an inference bug / wrong likelihood / mis-scaled posterior, even on data that would look perfectly fit. (It does not test whether the model matches reality – that is posterior_predictive_calibration()’s job, on real held-out data.)

Parameters:
Return type:

CalibrationVerdict

class CalibrationVerifier(*, reference_level=0.90, null_quantile=0.99, pit_tol=None, ensemble_dependence='shared-draws')[source]

Bases: object

An IC-6-shaped verifier (.verify(claim, context) -> dict) wrapping posterior_predictive_calibration(), so a calibration gate drops straight into mixle.task.knowledge_routing.route_task() (or any IC-6 consumer) as a real verifier that can fail a miscalibrated result – the thing the routing verifiers used in the experiments only ever checked structurally.

The tool result being verified must carry, on claim["payload"] (or context), an "ensemble" (k, m) posterior-predictive draw matrix and a "held_out_y" (k,) array. A payload without them fails closed with an explicit reason – a calibration verifier that silently passes anything it can’t actually check would defeat its own purpose.

Parameters:

  • reference_level (float)

  • null_quantile (float)

  • pit_tol (float | None)

  • ensemble_dependence (Literal['shared-draws', 'independent'])