mixle.inference.uq module

Unified uncertainty dispatch for models, predictors, ensembles, and LLM-style callables.

uq(thing, data) inspects the object it receives and routes to a compatible uncertainty method, returning a UQResult with the method name and the quantities needed for downstream checks.

• a fitted mixle model (has seq_log_density) -> a Laplace likelihood-curvature approximation; sample fitted models and get an explicitly approximate parameter interval.

• a torch module / any point predictor callable over arrays -> split-conformal calibration from a held-out (X, y); interval(x) returns a prediction interval with finite-sample coverage. Scope of that statement: the 1 - alpha coverage is MARGINAL over the calibration draw and the query jointly, under exchangeability of the calibration rows and incoming queries AND a predictor whose behavior is unchanged since calibration (an in-place mutation moved intervals from [0,0] to [100,100] with coverage 1.0 -> 0.0 while qhat sat unchanged – STAT-RR22-08/ RR23-09); it is not a per-query statement, and shift or a predictor update voids it silently – re-calibrate on drifted traffic or after any model change. Give a LIST of predictors instead and it becomes a deep ensemble (epistemic spread + conformal).

• an LLM-style callable over prompts (returns a string, or samples of strings) -> semantic entropy over meaning classes; confident(prompt) abstains when the model disagrees with itself.

The method is chosen from observed capability rather than a caller-supplied mode string.

class UQResult(kind, method, payload)

Bases: object

The uncertainty of a predictor, with the method that produced it and receipts to check it.

Parameters:

• kind (str)

• method (str)

• payload (dict[str, Any])

sample_models(n=200, *, seed=None)

n fitted models drawn from the parameter approximation.

Parameters:

• n (int)

• seed (int | None)

Return type:

list[Any]

credible_interval(readout, alpha=0.1, *, n=400, seed=0)

A 1-alpha credible interval on readout(model) over the parameter posterior.

Parameters:

• readout (Callable[[Any], float])

• alpha (float)

• n (int)

• seed (int)

Return type:

tuple[float, float]

parameter_interval(readout, alpha=0.1, *, n=400, seed=0)

A central interval over draws from the disclosed parameter approximation.

Parameters:

• readout (Callable[[Any], float])

• alpha (float)

• n (int)

• seed (int)

Return type:

tuple[float, float]

interval(x, alpha=None)

Calibrated prediction interval(s) at x. alpha overrides the calibrated level.

Parameters:

• x (Any)

• alpha (float | None)

Return type:

tuple[ndarray, ndarray]

epistemic_std(x)

Ensemble disagreement (std across members) at x – 0.0 for a single predictor.

Parameters:

x (Any)

Return type:

ndarray

semantic_entropy(prompt, *, n=8)

Entropy (nats) over the meaning classes of n sampled generations for prompt.

Parameters:

• prompt (Any)

• n (int)

Return type:

float

confident(prompt, *, n=8, max_entropy=None)

True when BIAS-CORRECTED semantic entropy is below the threshold.

The plug-in entropy at small n is biased LOW by ~(K-1)/(2n) nats – measured -0.656 at the 8-draw default on a uniform ten-class generator – which makes a hallucinating model look confident, the exact failure this gate exists to catch (STAT-RR17-14). The decision therefore uses the Miller-Madow-corrected estimate: still an estimate, but its residual error is far smaller and the correction errs toward ABSTAINING. The calibrated threshold is a quantile of the SAME corrected statistic at the SAME draw count (STAT-RR19-12: a plug-in-scale threshold against a corrected-scale gate moved acceptance from 98.2% to 41.0% on a fixed generator), so serving a calibrated threshold at a different n is refused – the correction is (K-1)/(2n), and changing n changes the statistic the threshold was calibrated for. An explicit max_entropy override carries no calibration claim and may use any n. Use semantic_entropy_receipt() to see the plug-in value, the correction, n, and K.

Parameters:

• prompt (Any)

• n (int)

• max_entropy (float | None)

Return type:

bool

semantic_entropy_receipt(prompt, *, n=8)

The entropy estimate WITH its approximation receipt (n, K, bias, corrected value).

Parameters:

• prompt (Any)

• n (int)

Return type:

dict[str, Any]

report()

Return uncertainty-quantification metadata and scalar payload fields.

Return type:

dict[str, Any]

uq(thing, data=None, *, alpha=0.1, equivalent=None)

Quantify the uncertainty of thing, choosing the method from what thing is.

Parameters:

• thing (Any) – a fitted mixle model, a torch module / point-predictor callable (or a list of them for a deep ensemble), or an LLM-style callable that maps a prompt to a generation.

• data (Any) – for a mixle model, the fitting data (builds a Laplace likelihood-curvature approximation); for a point predictor, (X_cal, y_cal) calibration data; for an LLM, optional example prompts used to calibrate an abstention threshold.

• alpha (float) – target miscoverage / abstention level (1 - alpha coverage).

• equivalent (Callable[[Any, Any], bool] | None) – for the LLM path, an optional meaning-equivalence predicate over generations (default: exact string match after stripping).

Returns:

A UQResult exposing the method-appropriate accessors and its own calibration numbers.

Return type:

UQResult