mixle.inference.uq module

Unified uncertainty dispatch for models, predictors, ensembles, and LLM-style callables.

uq(thing, data) inspects the object it receives and routes to a compatible uncertainty method, returning a UQResult with the method name and the quantities needed for downstream checks.

• a fitted mixle model (has seq_log_density) -> a Laplace parameter posterior; sample fitted models, read any summary, get a credible interval (epistemic uncertainty over parameters).

• a torch module / any point predictor callable over arrays -> split-conformal calibration from a held-out (X, y); interval(x) returns a prediction interval with finite-sample coverage. Give a LIST of predictors instead and it becomes a deep ensemble (epistemic spread + conformal).

• an LLM-style callable over prompts (returns a string, or samples of strings) -> semantic entropy over meaning classes; confident(prompt) abstains when the model disagrees with itself.

The method is chosen from observed capability rather than a caller-supplied mode string.

class UQResult(kind, method, payload)

Bases: object

The uncertainty of a predictor, with the method that produced it and receipts to check it.

Parameters:

• kind (str)

• method (str)

• payload (dict[str, Any])

kind: str

method: str

payload: dict[str, Any]

sample_models(n=200, *, seed=None)

n fitted models drawn from the parameter posterior (epistemic ensemble).

Parameters:

• n (int)

• seed (int | None)

Return type:

list[Any]

credible_interval(readout, alpha=0.1, *, n=400, seed=0)

A 1-alpha credible interval on readout(model) over the parameter posterior.

Parameters:

• readout (Callable[[Any], float])

• alpha (float)

• n (int)

• seed (int)

Return type:

tuple[float, float]

interval(x, alpha=None)

Calibrated prediction interval(s) at x. alpha overrides the calibrated level.

Parameters:

• x (Any)

• alpha (float | None)

Return type:

tuple[ndarray, ndarray]

epistemic_std(x)

Ensemble disagreement (std across members) at x – 0.0 for a single predictor.

Parameters:

x (Any)

Return type:

ndarray

semantic_entropy(prompt, *, n=8)

Entropy (nats) over the meaning classes of n sampled generations for prompt.

Parameters:

• prompt (Any)

• n (int)

Return type:

float

confident(prompt, *, n=8, max_entropy=None)

True when semantic entropy is below the threshold – else the model disagrees with itself.

Parameters:

• prompt (Any)

• n (int)

• max_entropy (float | None)

Return type:

bool

report()

Return type:

dict[str, Any]

uq(thing, data=None, *, alpha=0.1, equivalent=None)

Quantify the uncertainty of thing, choosing the method from what thing is.

Parameters:

• thing (Any) – a fitted mixle model, a torch module / point-predictor callable (or a list of them for a deep ensemble), or an LLM-style callable that maps a prompt to a generation.

• data (Any) – for a mixle model, the fitting data (builds the Laplace posterior); for a point predictor, (X_cal, y_cal) calibration data; for an LLM, optional example prompts used to calibrate an abstention threshold.

• alpha (float) – target miscoverage / abstention level (1 - alpha coverage).

• equivalent (Callable[[Any, Any], bool] | None) – for the LLM path, an optional meaning-equivalence predicate over generations (default: exact string match after stripping).

Returns:

A UQResult exposing the method-appropriate accessors and its own calibration numbers.

Return type:

UQResult