mixle.inference.conformal module

Conformal prediction: distribution-free intervals with finite-sample coverage.

Conformal prediction wraps any point predictor in an interval (or set) guaranteed to contain the truth with probability 1 - alpha in finite samples, assuming only exchangeability – no distributional assumptions about the model or the noise. This module is the array-level toolkit (operating on a fit_predict callable or precomputed residuals), complementing the PPL-fit wrappers in mixle.ppl.conformal:

  • split_conformal() – the fast split/inductive interval from a held-out calibration set, with optional one-sided (boundary) intervals.

  • jackknife_plus() / cv_plus() – leave-one-out (CV+) intervals that use all the data for both fitting and calibration, with the J+/CV+ coverage guarantee (Barber et al. 2021).

  • mondrian_conformal() – group-conditional intervals: a separate quantile per group, so coverage holds within each group, not just marginally.

  • weighted_conformal() – covariate-shift-robust intervals, reweighting the calibration scores by the test/train density ratio (Tibshirani et al. 2019).

fit_predict has the signature fit_predict(X_train, y_train, X_eval) -> y_hat so any estimator plugs in.

split_conformal(cal_pred, cal_y, test_pred, *, alpha=0.1, side='two-sided')[source]

Split (inductive) conformal interval from a calibration set.

Parameters:

  • cal_pred (ndarray) – (n,) model predictions on the calibration set.

  • cal_y (ndarray) – (n,) calibration responses.

  • test_pred (ndarray) – (m,) predictions at the test points.

  • alpha (float) – miscoverage level (1 - alpha coverage).

  • side (str) – "two-sided" (|y - yhat| score), "upper" (one-sided upper bound), or "lower" (one-sided lower bound).

Returns:

(lower, upper) arrays of length m (an unbounded side is -inf / +inf).

Return type:

tuple[ndarray, ndarray]

jackknife_plus(x, y, fit_predict, x_test, *, alpha=0.1)[source]

Jackknife+ intervals (leave-one-out), using all data for both fitting and calibration.

For each training point i the model is refit without i; R_i = |y_i - mu_{-i}(x_i)| is the LOO residual and mu_{-i}(x) the LOO prediction at a test point. The interval aggregates mu_{-i}(x) -/+ R_i across i (Barber et al. 2021), giving ~``1 - 2 alpha`` worst-case and ~``1 - alpha`` typical coverage without a data split. Costs n refits.

Returns:

(lower, upper) arrays of length len(x_test).

Parameters:
Return type:

tuple[ndarray, ndarray]

cv_plus(x, y, fit_predict, x_test, *, alpha=0.1, n_folds=10, seed=0)[source]

CV+ intervals: the K-fold analogue of jackknife_plus() (only n_folds refits).

Each point’s residual uses the model trained on the other folds, and the test prediction uses the same out-of-fold model. Much cheaper than Jackknife+ with nearly the same guarantee.

Returns:

(lower, upper) arrays of length len(x_test).

Parameters:
Return type:

tuple[ndarray, ndarray]

mondrian_conformal(cal_pred, cal_y, cal_groups, test_pred, test_groups, *, alpha=0.1)[source]

Mondrian (group-conditional) split conformal: a separate quantile per group.

Calibrates the conformal quantile within each group (taxonomy), so coverage holds conditional on the group rather than only marginally – the fix when error scale varies across known subpopulations.

Parameters:

  • cal_pred (ndarray) – calibration predictions, responses, and group labels.

  • cal_y (ndarray) – calibration predictions, responses, and group labels.

  • cal_groups (ndarray) – calibration predictions, responses, and group labels.

  • test_pred (ndarray) – test predictions and their group labels.

  • test_groups (ndarray) – test predictions and their group labels.

  • alpha (float) – miscoverage level.

Returns:

(lower, upper) arrays of length len(test_pred).

Return type:

tuple[ndarray, ndarray]

weighted_conformal(cal_pred, cal_y, test_pred, weights, *, alpha=0.1, test_weight=1.0)[source]

Covariate-shift-weighted split conformal (Tibshirani et al. 2019).

Under covariate shift the calibration and test inputs follow different distributions; reweighting the calibration scores by the likelihood ratio w(x) = p_test(x)/p_train(x) restores coverage. Uses the weighted empirical quantile of the calibration scores (each test point shares the same test_weight for its own potential score).

Parameters:

  • cal_pred (ndarray) – calibration predictions and responses.

  • cal_y (ndarray) – calibration predictions and responses.

  • test_pred (ndarray) – (m,) test predictions.

  • weights (ndarray) – (n,) likelihood-ratio weights for the calibration points (need not be normalised).

  • alpha (float) – miscoverage level.

  • test_weight (float) – the weight assigned to a test point (usually the mean test/train ratio; 1.0 when weights are self-normalised around the test density).

Returns:

(lower, upper) arrays of length m (a symmetric interval per test point).

Return type:

tuple[ndarray, ndarray]

conformal_label_threshold(cal_prob_true, *, alpha=0.1)[source]

Calibrate the LAC (least-ambiguous set-valued classifier) score threshold for 1 - alpha coverage.

The nonconformity score of a calibration point is 1 - p_model[true_class] – which needs the model’s class scores to rank well, not to be a true probability (the whole point: a softmax over a ReLU net is not a describable random process, but conformal still gives a finite-sample coverage guarantee from how those scores behave on held-out, exchangeable data). Returns the conformal quantile qhat of the calibration scores; a class is admitted at test time iff 1 - p[c] <= qhat (see conformal_label_sets()).

Parameters:

  • cal_prob_true (ndarray) – (n,) model score assigned to the true class of each calibration point.

  • alpha (float) – miscoverage level (1 - alpha marginal coverage of the returned sets).

Returns:

qhat – the score threshold (+inf when n is too small for the requested alpha).

Return type:

float

conformal_label_sets(cal_prob_true, test_prob, *, alpha=0.1, qhat=None)[source]

Split-conformal prediction sets for a classifier: distribution-free 1 - alpha label coverage.

Calibrates a LAC threshold (conformal_label_threshold()) on the held-out true-class scores, then admits every class whose score clears it. The returned boolean mask has guaranteed marginal coverage: the true label is in the set with probability >= 1 - alpha. A singleton set is a confident prediction; an empty or multi-label set is an honest “I’m not sure” – the signal a cost-aware cascade escalates on.

Parameters:

  • cal_prob_true (ndarray) – (n,) score assigned to the true class of each calibration point.

  • test_prob (ndarray) – (m, K) model class scores at the test points (rows need not sum to 1).

  • alpha (float) – miscoverage level.

  • qhat (float | None) – a precomputed threshold (e.g. from an earlier calibration); recomputed if None.

Returns:

(sets, qhat) – sets is an (m, K) boolean mask, qhat the threshold used.

Return type:

tuple[ndarray, float]