mixle.inference.calibrate_fit module

Calibration reports as a post-condition of fitting.

A fit provides parameters; it does not by itself show whether predictive probabilities are calibrated on held-out data. calibration_report() returns the held-out mean log-density and, when the model exposes a predictive CDF, a probability-integral-transform (PIT) calibration check.

Calibration is opt-in because it reserves held-out data. When requested through the higher-level fitting surfaces, the resulting report is attached to the model or artifact.

class CalibrationReport(n, mean_log_density, pit_error=None, pit_histogram=None, bins=10, method='', note='')[source]

Bases: object

Whether a fitted model’s uncertainty is calibrated on held-out data.

pit_error is the total-variation distance of the PIT histogram from uniform (0 = perfectly calibrated). It has a finite-sample floor ~``sqrt(bins/n)`` even for a perfect model, so is_calibrated() judges against that floor rather than a fixed constant.

Parameters:
n: int
mean_log_density: float
pit_error: float | None = None
pit_histogram: dict[str, Any] | None = None
bins: int = 10
method: str = ''
note: str = ''
noise_floor()[source]

The PIT-error a perfectly calibrated model would show at this sample size (sampling noise).

Return type:

float

is_calibrated(tol=None)[source]

True when the PIT error is within tolerance. Default tol = 2.5x the finite-sample noise floor (so genuine miscalibration, not sampling noise, is what fails). Unknown -> False, conservatively.

Parameters:

tol (float | None)

Return type:

bool

as_dict()[source]
Return type:

dict[str, Any]

calibration_report(model, data)[source]

The calibration of model on held-out data (see module docstring).

data should be data the model was NOT fitted on – calibration measured on the training set is optimistic. Runs the PIT test when the model has a scalar predictive CDF; always reports the held-out mean log-density.

Parameters:
Return type:

CalibrationReport