mixle.ppl.conformal module

Split (inductive) conformal prediction — distribution-free, finite-sample valid prediction intervals and label sets around any already-fitted model.

Conformal prediction turns point predictions into calibrated sets using a held-out calibration split, with a coverage guarantee that holds for any model and any data distribution as long as the calibration and test points are exchangeable: a set built at level alpha covers the truth with probability at least 1 - alpha. A wrong model only makes the sets wider, never breaks the guarantee.

The machinery is a nonconformity score plus one order statistic. For regression the score is the absolute residual |y - yhat| and the calibrated interval is predict(x) +/- qhat; for classification the score is 1 - p(true class | x) and the label set is {y : 1 - p(y | x) <= tau}. Both reduce to the conformal quantile qhat / tau — the ceil((n + 1)(1 - alpha)) smallest calibration score, the +1 being the finite-sample correction.

ConformalRegressor wraps a fitted RegressionResult (anything exposing predict(given)); ConformalClassifier wraps a matrix of per-class probabilities (e.g. the posterior of a mixle generative classifier). The conformal() helper is the one-liner entry point for the regression case.

conformal_quantile(scores, alpha)[source]

The level-alpha conformal quantile of calibration scores.

Returns the ceil((n + 1)(1 - alpha)) smallest score (the finite-sample-corrected empirical 1 - alpha quantile). When alpha is too small for the calibration size — (n + 1)(1 - alpha) > n — no finite threshold gives the requested coverage and inf is returned, the honest “the set is everything” answer.

Parameters:
Return type:

float

class ConformalRegressor(result, y_cal, *, given, alpha=0.1)[source]

Bases: object

Split-conformal prediction intervals around a fitted regression result.

Calibrates the absolute-residual nonconformity score on held-out (given, y_cal) and produces symmetric intervals predict(x) +/- qhat with marginal coverage at least 1 - alpha. result is any object with a predict(given) method returning the fitted mean (a RegressionResult, a location-scale result, or a GP regressor).

Parameters:

  • result (Any)

  • y_cal (Any)

  • given (dict)

  • alpha (float)

interval(given)[source]

Return (lower, upper) arrays of the conformal interval at covariates given.

Parameters:

given (dict)

Return type:

tuple[ndarray, ndarray]

covers(y, *, given)[source]

Boolean array: does the interval at given contain each observed y.

Parameters:
Return type:

ndarray

class ConformalClassifier(proba_cal, y_cal, *, alpha=0.1)[source]

Bases: object

Split-conformal label sets from per-class probabilities.

proba_cal is an (n, K) matrix of calibration probabilities p(y | x) (any proper classifier — a mixle generative classifier’s class posterior, a softmax, …) and y_cal the integer labels. The nonconformity score is 1 - p(true) and the calibrated set keeps every label whose score is within the conformal quantile, so it covers the true label with probability at least 1 - alpha and grows from one label (confident) to several (hedging) as the model is unsure.

Parameters:

  • proba_cal (Any)

  • y_cal (Any)

  • alpha (float)

predict_set(proba)[source]

Boolean (n, K) label-inclusion matrix at probabilities proba.

Parameters:

proba (Any)

Return type:

ndarray

covers(proba, y)[source]

Boolean array: is each true label y in the predicted set.

Parameters:
Return type:

ndarray

set_sizes(proba)[source]

Number of labels in the predicted set for each row of proba.

Parameters:

proba (Any)

Return type:

ndarray

class ConformalQuantileRegressor(lo, hi, y_cal, *, given, alpha=0.1)[source]

Bases: object

Conformalized quantile regression (Romano, Patterson, Candes 2019).

Combines two fitted quantile regressions (a lower and an upper conditional quantile) with a split-conformal calibration so the band has exact marginal coverage and the adaptive, heteroscedastic width of quantile regression — wide where the data is noisy, narrow where it is tight, unlike the constant-width absolute-residual band of ConformalRegressor.

The nonconformity score is the signed distance outside the predicted band, E_i = max(qlo(x_i) - y_i, y_i - qhi(x_i)) (negative when y_i is comfortably inside), and the calibrated band is [qlo(x) - qhat, qhi(x) + qhat] with qhat the conformal quantile of the calibration scores. lo and hi are fitted quantile-regression results (from ...fit(..., quantile=tau)), typically at tau = alpha/2 and 1 - alpha/2.

Parameters:

  • lo (Any)

  • hi (Any)

  • y_cal (Any)

  • given (dict)

  • alpha (float)

interval(given)[source]

Return (lower, upper) arrays of the calibrated adaptive band at covariates given.

Parameters:

given (dict)

Return type:

tuple[ndarray, ndarray]

covers(y, *, given)[source]

Boolean array: does the adaptive band at given contain each observed y.

Parameters:
Return type:

ndarray

class ConformalStructure(dist, calibration, *, alpha=0.1)[source]

Bases: object

Split-conformal credible sets over combinatorial structures (rankings, matchings, spanning trees, permutations, …) from a fitted mixle distribution’s exact log-density.

The nonconformity of a structure s is -log p(s): the lower its model probability, the more surprising it is. Calibrating on held-out true structures yields a log-probability threshold, and the conformal set is {s : log p(s) >= threshold} — it contains the true structure with probability at least 1 - alpha whenever the calibration and test structures are exchangeable (for example iid draws), with no assumption on the model being correct.

dist is any structure distribution exposing log_density (PlackettLuceDistribution, MallowsDistribution, MatchingDistribution, SpanningTreeDistribution, …); calibration is a sequence of observed structures. Membership is always available; listing or counting the set additionally needs the distribution’s exact enumerator().

Parameters:

  • dist (Any)

  • calibration (Any)

  • alpha (float)

property log_prob_threshold: float

Structures with log p(s) at or above this value are in the conformal set.

contains(structure)[source]

Is structure in the conformal set (its log-probability above the threshold).

Parameters:

structure (Any)

Return type:

bool

covers(structures)[source]

Boolean array: membership of each structure (use on held-out truths to check coverage).

Parameters:

structures (Any)

Return type:

ndarray

members()[source]

List the structures in the conformal set, highest-probability first.

Requires the distribution’s exact enumerator() (raises EnumerationError otherwise). The enumerator yields structures in descending log-probability, so the scan stops at the threshold.

Return type:

list

size()[source]

Number of structures in the conformal set (needs the exact enumerator()).

Return type:

int

class ConformalLinkPredictor(edge_prob, cal_edges, *, alpha=0.1)[source]

Bases: object

Split-conformal candidate-neighbor sets for a random-graph model from its edge-probability matrix P (P[i, j] = p(edge i--j), e.g. X @ X.T from a fitted RDPG, or an Erdos-Renyi / stochastic-block-model edge probability).

The nonconformity of a present edge (i, j) is 1 - P[i, j]. Calibrating on held-out true edges gives a threshold; the predicted neighbor set of a node keeps every candidate j with 1 - P[i, j] <= tau, so it contains a true neighbor with probability at least 1 - alpha over exchangeable held-out edges (a random split of the observed edges).

Parameters:

  • edge_prob (Any)

  • cal_edges (Any)

  • alpha (float)

neighbor_set(i, candidates=None)[source]

Candidate nodes j in node i’s conformal neighbor set.

Parameters:
Return type:

ndarray

covers(edges)[source]

Boolean array: is each held-out true edge’s endpoint in the predicted neighbor set.

Parameters:

edges (Any)

Return type:

ndarray

set_sizes(nodes=None)[source]

Neighbor-set size per node (defaults to all nodes).

Parameters:

nodes (Any)

Return type:

ndarray

class ConformalKnowledgeGraph(kg, calibration, *, slot='tail', alpha=0.1)[source]

Bases: object

Split-conformal completion sets for a knowledge-graph model (any-slot UQ).

Calibrating on held-out true triples turns the model’s completion posterior into a set of candidate fillers that contains the true one with probability at least 1 - alpha over exchangeable held-out triples. slot selects which slot is predicted – 'tail' for (h, r, ?), 'head' for (?, r, t), 'relation' for (h, ?, t) – using the model’s tail_log_posterior / head_log_posterior / relation_log_posterior. The nonconformity of a triple is 1 - p(true filler), so a confident model gives small completion sets and a recommended completion carries a coverage guarantee.

Parameters:

  • kg (Any)

  • calibration (Any)

  • slot (str)

  • alpha (float)

completion_set(h=None, r=None, t=None)[source]

Candidate fillers in the conformal set for the missing slot of a query.

Parameters:

  • h (int | None)

  • r (int | None)

  • t (int | None)

Return type:

ndarray

covers(triples)[source]

Boolean array: is each held-out true triple’s filler in the completion set.

Parameters:

triples (Any)

Return type:

ndarray

set_sizes(triples)[source]

Completion-set size for each query (the slot of each triple is treated as missing).

Parameters:

triples (Any)

Return type:

ndarray

conformal(result, y_cal, *, given, alpha=0.1)[source]

Split-conformal calibration of a fitted regression result into prediction intervals.

Mirrors fit’s convention (labels positional, given= keyword), a one-liner over ConformalRegressor:

m = Normal(free * Field("x") + free, free).fit(y_tr, given={"x": x_tr})
cp = conformal(m.result, y_cal, given={"x": x_cal}, alpha=0.1)
lo, hi = cp.interval({"x": x_te})
cp.covers(y_te, given={"x": x_te}).mean()   # ~ 0.9
Parameters:
Return type:

ConformalRegressor