mixle.inference.calibration module

Calibration diagnostics: “is my probability / interval actually calibrated?”

A forecast is calibrated when its stated probabilities match observed frequencies: events given 70% should happen ~70% of the time, and 90% intervals should contain the truth ~90% of the time. This is distinct from sharpness (how concentrated the forecast is) and from accuracy – a forecaster can be perfectly calibrated and useless (always predict the base rate), so calibration is a necessary, not sufficient, condition that you check separately. These diagnostics are model-free: they look only at predicted probabilities/intervals and what happened.

Three families, by forecast type:

  • Probability classifiers – reliability_curve() (the reliability diagram), and the expected_calibration_error() / maximum_calibration_error() summaries. For multiclass use top_label_confidence() to reduce to the (confidence, correct) calibration problem.

  • Full predictive distributions – the Probability Integral Transform: pit_values() / pit_ensemble(), the pit_histogram(), and pit_calibration_error(). Under a calibrated forecast the PIT values are Uniform(0, 1); a U-shaped histogram means under-dispersion, a hump means over-dispersion, a slope means bias.

  • Intervals / quantiles – interval_coverage() (coverage and mean width at one level) and coverage_curve() (empirical-vs-nominal coverage across a grid of levels).

Several functions take ci=True to attach a nonparametric bootstrap confidence band, so a reliability diagram or ECE comes with honest error bars rather than a bare point estimate.

reliability_curve(prob, outcome, *, bins=10, strategy='uniform', ci=False, n_boot=1000, ci_level=0.95, seed=0)[source]

Reliability diagram: observed frequency vs mean forecast probability, per bin.

Bins the forecasts, then within each bin compares the mean predicted probability to the observed event frequency. A perfectly calibrated forecaster lies on the diagonal observed == predicted.

Parameters:

  • prob (ndarray) – (n,) predicted probabilities of the positive class (or top-label confidences).

  • outcome (ndarray) – (n,) 0/1 outcomes (or correctness indicators).

  • bins (int) – number of bins.

  • strategy (str) – "uniform" (equal-width) or "quantile" (equal-count) bins.

  • ci (bool) – if True attach a percentile bootstrap band on the observed frequency in each bin.

  • n_boot (int) – bootstrap resamples when ci is True.

  • ci_level (float) – central probability of the bootstrap band (e.g. 0.95).

  • seed (int | RandomState | None) – RNG seed for the bootstrap.

Returns:

{'mean_pred', 'obs_freq', 'count', 'bin_edges'} (one entry per non-empty bin), plus 'obs_lo' / 'obs_hi' when ci is True.

Return type:

dict[str, ndarray]

expected_calibration_error(prob, outcome, *, bins=10, strategy='uniform', norm='l1', ci=False, n_boot=1000, ci_level=0.95, seed=0)[source]

Expected Calibration Error: count-weighted average gap between confidence and accuracy.

ECE = sum_b (n_b / n) |obs_b - pred_b| over bins (norm='l2' uses the squared gap, square- rooted). Zero is perfect calibration. For multiclass classifiers reduce with top_label_confidence() first.

Parameters:

  • prob (ndarray) – (n,) predicted probabilities / confidences.

  • outcome (ndarray) – (n,) 0/1 outcomes / correctness indicators.

  • bins (int) – number of bins.

  • strategy (str) – "uniform" or "quantile" binning.

  • norm (str) – "l1" (mean absolute gap) or "l2" (root mean squared gap).

  • ci (bool) – if True also return a percentile bootstrap interval.

  • n_boot (int) – bootstrap controls.

  • ci_level (float) – bootstrap controls.

  • seed (int | RandomState | None) – bootstrap controls.

Returns:

The ECE (float), or (ece, lo, hi) when ci is True.

Return type:

float | tuple[float, float, float]

maximum_calibration_error(prob, outcome, *, bins=10, strategy='uniform')[source]

Maximum Calibration Error: the worst per-bin gap max_b |obs_b - pred_b|.

Unlike expected_calibration_error() this is not count-weighted, so it surfaces a small but badly-miscalibrated region that the average would hide.

Parameters:
Return type:

float

top_label_confidence(prob, labels)[source]

Reduce a multiclass classifier to the (confidence, correct) top-label calibration problem.

Parameters:

  • prob (ndarray) – (n, K) class-probability matrix.

  • labels (ndarray) – (n,) integer true labels.

Returns:

the max predicted probability per row and a 0/1 indicator of whether that argmax class was the true label. Feed these to reliability_curve() / expected_calibration_error().

Return type:

(confidence, correct)

pit_values(y, cdf)[source]

Probability Integral Transform values u_i = F_i(y_i).

Under a calibrated continuous predictive distribution the PIT values are Uniform(0, 1). Pass either the precomputed CDF values F_i(y_i) or a callable cdf(y) -> F(y).

Parameters:
Returns:

(n,) PIT values, clipped to [0, 1].

Return type:

ndarray

pit_ensemble(y, forecasts, *, randomize=True, seed=0)[source]

Rank-based PIT from a finite predictive ensemble.

u_i is the fraction of ensemble members <= y_i. With randomize=True ties are broken by a uniform jitter within the rank gap, which makes the PIT exactly Uniform(0, 1) under calibration even for discrete ensembles (the randomized PIT of Czado et al. 2009).

Parameters:

  • y (ndarray) – (n,) realised values.

  • forecasts (ndarray) – (n, m) ensemble (m draws per observation).

  • randomize (bool) – jitter ties for an exactly-uniform PIT.

  • seed (int | RandomState | None) – RNG seed when randomize is True.

Returns:

(n,) PIT values in [0, 1].

Return type:

ndarray

pit_histogram(pit, *, bins=10)[source]

Histogram of PIT values with the uniform reference level.

Parameters:

  • pit (ndarray) – (n,) PIT values.

  • bins (int) – number of equal-width bins on [0, 1].

Returns:

{'counts', 'density', 'edges', 'uniform'} where density integrates to 1 and uniform is the flat reference density (1.0) a calibrated forecast would match.

Return type:

dict[str, ndarray]

pit_calibration_error(pit, *, bins=10)[source]

Calibration error of a PIT histogram: mean absolute deviation from uniform mass.

sum_b |count_b/n - 1/bins| – 0 when the PIT histogram is perfectly flat (calibrated), larger when it is U-shaped (under-dispersed) or humped (over-dispersed).

Parameters:
Return type:

float

interval_coverage(lower, upper, y)[source]

Empirical coverage and mean width of a set of prediction intervals.

Parameters:

  • lower (ndarray) – (n,) lower endpoints.

  • upper (ndarray) – (n,) upper endpoints.

  • y (ndarray) – (n,) realised values.

Returns:

{'coverage', 'mean_width'} – the fraction of y inside [lower, upper] and the mean interval width. Compare coverage to the nominal level the interval was built for.

Return type:

dict[str, float]

coverage_curve(forecasts, y, *, levels=None)[source]

Empirical-vs-nominal coverage of central intervals across a grid of nominal levels.

For each nominal central level c the per-observation central interval [quantile((1-c)/2), quantile((1+c)/2)] is read off the predictive ensemble and its empirical coverage is measured. A calibrated forecast traces the diagonal empirical == nominal; bowing below the diagonal means the intervals are too narrow (over-confident).

Parameters:

  • forecasts (ndarray) – (n, m) predictive ensemble.

  • y (ndarray) – (n,) realised values.

  • levels (ndarray | None) – nominal central coverage levels in (0, 1); defaults to 0.05 .. 0.95 by 0.05.

Returns:

{'nominal', 'empirical'} arrays of equal length.

Return type:

dict[str, ndarray]

class ProbabilityCalibrator(method='isotonic')[source]

Bases: object

Map raw scores to calibrated probabilities – fit against binary outcomes.

A raw score (a model’s confidence, a self-consistency fraction, a token likelihood) need not be a probability of anything: it can be monotone-but-miscalibrated, or have no relationship to the outcome at all. This learns the transform score -> P(outcome = 1 | score) from labeled data, so the output is a probability of the event you calibrated against.

  • method="isotonic" – monotone, non-parametric (pool-adjacent-violators). Assumes higher score => not-lower probability; flexible, needs enough calibration points.

  • method="platt" – logistic sigmoid(a * score + b). Two parameters, robust on little data, but assumes a sigmoidal relationship.

A near-flat fitted curve is itself the finding: it means the raw score carried little information about the outcome (its “likelihood” was unrelated to the event).

Parameters:

method (str)

fit(scores, outcomes)[source]

Fit the score->probability map on scores with binary outcomes (0/1).

Parameters:
Return type:

ProbabilityCalibrator

predict(scores)[source]

Calibrated probabilities for scores (clamped to [0, 1]).

Parameters:

scores (Any)

Return type:

ndarray

calibrate_probabilities(scores, outcomes, *, method='isotonic')[source]

Fit a ProbabilityCalibrator mapping scores to P(outcome=1 | score).

Parameters:
Return type:

ProbabilityCalibrator