mixle.inference.calibrate_fit module

Calibration reports as a post-condition of fitting.

A fit provides parameters; it does not by itself show whether predictive probabilities are calibrated on held-out data. calibration_report() returns the held-out mean log-density and, when the model exposes a predictive CDF, a probability-integral-transform (PIT) calibration check.

Calibration is opt-in because it reserves held-out data. When requested through the higher-level fitting surfaces, the resulting report is attached to the model or artifact.

class CalibrationReport(n, mean_log_density, pit_error=None, pit_histogram=None, bins=10, method='', note='', n_invalid_scores=0)[source]

Bases: object

Whether a fitted model’s uncertainty is calibrated on held-out data.

pit_error is the total-variation distance of the PIT histogram from uniform (0 = perfectly calibrated). It has a finite-sample floor ~``sqrt(bins/n)`` even for a perfect model, so is_calibrated() judges against that floor rather than a fixed constant.

Parameters:
noise_floor()[source]

The PIT-error a perfectly calibrated model would show at this sample size (sampling noise).

Return type:

float

scoring_is_valid()[source]

Whether every held-out row got a usable (finite or explicitly impossible) log-density.

Return type:

bool

max_pit_error()[source]

The largest PIT error this histogram statistic can attain: all mass in one bin.

sum_b |freq_b - 1/bins| peaks at (1 - 1/bins) + (bins-1)/bins == 2(1 - 1/bins) – 1.8 for the default ten bins. A tolerance at or above this bound accepts every possible histogram, so the gate is not a test at all.

Return type:

float

has_enough_data()[source]

Whether the holdout can support a per-bin uniformity judgement at all.

The histogram statistic needs a usable expected count per bin; below MIN_EXPECTED_PER_BIN the noise floor sqrt(bins/n) grows until 2.5x it exceeds max_pit_error() and the acceptance region becomes vacuous – at ten bins and a single observation the threshold was 7.91 against a statistic that cannot exceed 1.8, so even maximally clumped PIT values were reported calibrated.

Return type:

bool

is_calibrated(tol=None)[source]

True when the PIT error is within tolerance AND the model actually scored the holdout.

Default tol = 2.5x the finite-sample noise floor (so genuine miscalibration, not sampling noise, is what fails). Unknown -> False, conservatively. An invalid predictive score on any row -> False: the PIT histogram alone cannot certify a model whose density is unavailable. Too little held-out data for has_enough_data() -> False: absence of evidence is not evidence of calibration.

Raises:

ValueError – if tol is not finite and nonnegative, or is at/above max_pit_error() (a tolerance that no histogram can exceed would approve any model).

Parameters:

tol (float | None)

Return type:

bool

as_dict()[source]

Return rounded calibration metrics as JSON-compatible data.

Return type:

dict[str, Any]

to_dataframe()[source]

Return this report as a pandas.DataFrame.

When a PIT histogram was computed (method == "PIT"), returns one row per bin – bin_left, bin_right, count, density, uniform – the natural columnar shape of a calibration report (a table you’d plot). Otherwise (no scalar predictive CDF, so no PIT histogram) returns a single summary row with the same unrounded fields as as_dict(). Requires the pandas extra (pip install mixle[pandas]).

Return type:

Any

to_parquet(path, **kwargs)[source]

Write this report to a Parquet file; see to_dataframe().

kwargs forward to DataFrame.to_parquet (e.g. engine=, compression=). Needs a Parquet engine in addition to pandas – pip install mixle[arrow] (pyarrow) or fastparquet.

Parameters:
Return type:

None

calibration_report(model, data, *, seed=0)[source]

The calibration of model on held-out data (see module docstring).

data should be data the model was not fitted on – calibration measured on the training set is optimistic. Runs the PIT test when the model has a scalar predictive CDF; always reports the held-out mean log-density.

A predictive law with ATOMS is handled with the randomized PIT rather than the continuous one: F(Y) is uniform only for a continuous law, so routing a discrete model through the ordinary PIT reports a large error for an exactly correct model. seed fixes that randomization (it is recorded in note) and is unused for a continuous law, whose method is unchanged.

Scoring validity is checked first and carried in the report. The verdict used to consult only the PIT histogram, so a model whose CDF was exact but whose per-row log-densities were all NaN – or which returned one score for a thousand rows – passed the gate as "calibrated". Exactly one usable score per row is now required for a calibrated verdict; a cardinality mismatch raises and NaN scores are counted into n_invalid_scores, which forces is_calibrated() to False.

Raises:

ValueError – if the model’s batch scoring does not return exactly one log-density per row.

Parameters:
Return type:

CalibrationReport