mixle.evolve.objective module

The measure contract for self-improvement: a model-agnostic fitness.

An Objective turns (model, data) into a single comparable scalar and, when it can, an honest per-observation paired vector. The paired vector is what lets the verify gate (mixle.evolve.verify) run a paired significance test instead of comparing two bare score totals. Objectives that cannot produce a paired vector (a pure summary like a calibration-error scalar) set pointwise to return None; the gate then falls back to a CI-exclusion check on the bootstrapped scalar.

Every builder here is a thin adapter over an existing, verified scorer:

  • nll_objective -> per-obs -log p(y_i) from model.seq_log_density.

  • log_score_objective -> mixle.inference.scoring.log_score() on the predictive densities.

  • crps_objective -> mixle.inference.scoring.crps_ensemble() on a sampled ensemble.

  • interval_objective -> mixle.inference.scoring.interval_score() (Winkler) on ensemble quantiles.

  • calibration_objective-> PIT-based calibration error (mixle.inference.calibration()), scalar-only.

  • decision_regret_objective -> realized regret of mixle.inference.decision.bayes_action().

The model-to-array bridges (encoding, per-obs log density, ensemble sampling) live in this module so the scorers stay pure array functions and the objectives stay five-line adapters.

class Objective(*args, **kwargs)[source]

Bases: Protocol

A lower-is-better-or-higher-with-a-flag scalar fitness with an optional paired vector.

name: str
lower_is_better: bool
pointwise(model, data)[source]

The (n,) per-observation score, or None if the objective is scalar-only.

Parameters:
Return type:

ndarray | None

scalar(model, data)[source]

The single comparable fitness (mean of pointwise by default).

Parameters:
Return type:

float

nll_objective()[source]

Negative log-likelihood: per-obs -log p(y_i) (strictly proper, lower is better).

Return type:

Objective

log_score_objective()[source]

Logarithmic score (log loss) of the predictive density at the realised outcomes.

Return type:

Objective

crps_objective(*, ensemble=256, seed=0)[source]

Continuous Ranked Probability Score from a sampled predictive ensemble (lower is better).

Parameters:

  • ensemble (int) – number of predictive draws per observation.

  • seed (int) – RNG seed for the ensemble (reproducible).

Return type:

Objective

interval_objective(level=0.9, *, ensemble=256, seed=0)[source]

Winkler interval score for the central level predictive interval (lower is better).

Parameters:

  • level (float) – central coverage of the interval (e.g. 0.9 for a 90% interval).

  • ensemble (int) – number of predictive draws used to read off the interval endpoints.

  • seed (int) – RNG seed for the ensemble.

Return type:

Objective

calibration_objective(*, ensemble=256, seed=0, bins=10)[source]

PIT calibration error of the predictive distribution (scalar-only, lower is better).

Uses the rank-based Probability Integral Transform of a sampled ensemble: under a calibrated continuous forecast the PIT values are Uniform(0, 1), and pit_calibration_error measures the histogram’s mean absolute deviation from uniform. There is no honest per-observation paired vector for a histogram statistic, so pointwise returns None and the verify gate scores this on the bootstrapped scalar.

(For classification models the natural calibration scalar is mixle.inference.calibration.expected_calibration_error(); this builder targets the continuous-predictive case, which is the common one for the streaming/auto-select loop.)

Parameters:
Return type:

Objective

decision_regret_objective(loss, actions, *, n=2000, seed=0)[source]

Realized decision regret under a fitted predictive posterior (scalar-only, lower is better).

For the chosen Bayes action a* = bayes_action(posterior(model), loss, actions) this reports the expected loss E_draw[ loss(a*, draw) ] – the realized cost of acting optimally under the model’s own belief. A better-calibrated model yields a lower expected loss for the same loss and action set, so it is a first-class promotion metric. Scalar-only (the regret is an expectation over posterior draws, not a per-observation quantity), so pointwise returns None.

Parameters:

  • loss (Callable[[Any, Any], float]) – loss(action, draw) -> float (or a numpy-vectorized loss(action, draws) -> array).

  • actions (Sequence[Any]) – the finite candidate-action set.

  • n (int) – posterior draws for the Monte-Carlo expectation.

  • seed (int) – RNG seed.

Return type:

Objective

pointwise_log_density(model, data)[source]

Per-observation log p(y_i) under model via the vectorized seq_log_density path.

Models exposing a seq_log_density_raw(rows) (e.g. an affine recalibration whose change-of-variables needs the raw values) are scored through that split-safe path instead of the encode-then-score path, which cannot recover the raw responses from an encoded handle.

Parameters:
Return type:

ndarray

sample_ensemble(model, n, m, *, seed)[source]

Draw an (n, m) predictive ensemble: m iid draws repeated for n observations.

The plug-in predictive is exchangeable across observations, so one (m,) draw row is broadcast to all n rows – the per-observation CRPS/interval scores then differ only through y_i.

Parameters:
Return type:

ndarray