mixle.evolve.objective module

The measure contract for self-improvement: a model-agnostic fitness.

An Objective turns (model, data) into a single comparable scalar and, when it can, a per-observation paired vector. The paired vector is what lets the verify gate (mixle.evolve.verify) run a paired significance test instead of comparing two bare score totals. Objectives that cannot produce a paired vector (a pure summary like a calibration-error scalar) set pointwise to return None; the gate then falls back to a bare scalar-delta-vs- min_effect comparison, with p_value/ci set to the explicit nan “not applicable” sentinel – no bootstrap, replication, or other resampling evidence backs a raw scalar delta, so a scalar-only verdict is reported for a human to review but can never auto-promote on its own (see mixle.evolve.verify module docstring point 8 and has_statistical_evidence).

Every builder here is a thin adapter over an existing, verified scorer:

  • nll_objective -> per-obs -log p(y_i) from model.seq_log_density.

  • log_score_objective -> per-obs -log p(y_i) computed directly from the log-density (the same quantity as nll_objective, kept as a separate name for the decision-theoretic framing). Computed straight from the log-density, never via exp then re-log through mixle.inference.scoring.log_score() – that round trip underflows to 0.0 for any log-density past about -745, after which the clip-and-relog step can no longer tell two bad fits apart.

  • crps_objective -> mixle.inference.scoring.crps_ensemble() on a sampled ensemble.

  • interval_objective -> mixle.inference.scoring.interval_score() (Winkler) on ensemble quantiles.

  • calibration_objective-> PIT-based calibration error (mixle.inference.calibration()), scalar-only.

  • decision_regret_objective -> mixle.inference.decision.bayes_action()’s chosen action, scored by realized loss against the actual data – not the model’s own posterior draws.

The model-to-array bridges (encoding, per-obs log density, ensemble sampling) live in this module so the scorers stay pure array functions and the objectives stay five-line adapters.

class Objective(*args, **kwargs)[source]

Bases: Protocol

A lower-is-better-or-higher-with-a-flag scalar fitness with an optional paired vector.

pointwise(model, data)[source]

The (n,) per-observation score, or None if the objective is scalar-only.

Parameters:
Return type:

ndarray | None

scalar(model, data)[source]

The single comparable fitness (mean of pointwise by default).

Parameters:
Return type:

float

nll_objective()[source]

Negative log-likelihood: per-obs -log p(y_i) (strictly proper, lower is better).

Return type:

Objective

log_score_objective()[source]

Logarithmic score (log loss) of the predictive density at the realised outcomes.

Computed directly from the log-density (-log p(y_i)), never by exponentiating the log-density to a plain probability and then re-deriving a log-scale score from it. That round trip is destructive: exp(x) underflows to exactly 0.0 for any x below about -745 (the float64 underflow boundary), so every log-density past that point becomes the identical clipped probability and hence the identical score, even though the underlying fits are genuinely – and very differently – bad. Working from the log-density keeps that information: float64 has ample range in log-space, just not in probability-space.

Return type:

Objective

crps_objective(*, ensemble=256, seed=0)[source]

Continuous Ranked Probability Score from a sampled predictive ensemble (lower is better).

Parameters:

  • ensemble (int) – number of predictive draws per observation (an exact positive integer).

  • seed (int) – RNG seed for the ensemble (reproducible).

Return type:

Objective

interval_objective(level=0.9, *, ensemble=256, seed=0)[source]

Winkler interval score for the central level predictive interval (lower is better).

Parameters:

  • level (float) – central coverage of the interval (e.g. 0.9 for a 90% interval).

  • ensemble (int) – number of predictive draws used to read off the interval endpoints (an exact positive integer).

  • seed (int) – RNG seed for the ensemble.

Return type:

Objective

calibration_objective(*, ensemble=256, seed=0, bins=10)[source]

PIT calibration error of the predictive distribution (scalar-only, lower is better).

Uses the rank-based Probability Integral Transform of a sampled ensemble: under a calibrated continuous forecast the PIT values are Uniform(0, 1), and pit_calibration_error measures the histogram’s mean absolute deviation from uniform. There is no per-observation paired vector for a histogram statistic, so pointwise returns None and the verify gate falls back to a bare scalar comparison with no p-value or CI, reported for human review only – see this module’s docstring and mixle.evolve.verify module docstring point 8.

(For classification models the natural calibration scalar is mixle.inference.calibration.expected_calibration_error(); this builder targets the continuous-predictive case, which is the common one for the streaming/auto-select loop.)

ensemble and bins must both be exact positive integers – they are the budget the reported calibration error is computed from.

Parameters:
Return type:

Objective

decision_regret_objective(loss, actions, *, n=2000, seed=0, vectorized=None)[source]

Realized decision regret of a model’s chosen action against actual data (scalar-only, lower is better).

For the chosen Bayes action a* = bayes_action(posterior(model), loss, actions) – optimal under the model’s OWN predictive belief – this reports the realized loss mean(loss(a*, y_i) for y_i in data) against the actual data, not draws from that same model’s own posterior. Scoring against self-generated draws would let a confidently wrong model look flawless: its chosen action and its “outcomes” both come from the same wrong belief, so they always agree with each other regardless of how far that belief is from reality. Checking the chosen action against real, shared data is what makes this a genuine promotion metric rather than a self-consistency check. Scalar-only (this is a single realized-loss number per model, not a per-observation quantity), so pointwise returns None.

Parameters:

  • loss (Callable[[Any, Any], float]) – loss(action, draw) -> float (or a numpy-vectorized loss(action, draws) -> array).

  • actions (Sequence[Any]) – the finite candidate-action set.

  • n (int) – posterior draws used only to pick the Bayes-optimal action under the model’s own predictive belief; the reported score is the realized loss of that action on data.

  • seed (int) – RNG seed for the action-selection draws.

  • vectorized (bool | None) – the loss’s calling convention, forwarded to bayes_action() and used for the realized-loss evaluation. True -> called once with the whole outcome array; False -> called once per outcome; None (default) -> read loss.vectorized if the loss declares it, else auto-detect by probing the array call once. A probe is a real invocation of the loss, and a search scores many models against the same loss, so the resolved convention is reused for the rest of the search rather than re-probed per model. A loss that keeps state or has side effects should declare its convention.

Return type:

Objective

pointwise_log_density(model, data)[source]

Per-observation log p(y_i) under model via the vectorized seq_log_density path.

Models exposing a seq_log_density_raw(rows) (e.g. an affine recalibration whose change-of-variables needs the raw values) are scored through that split-safe path instead of the encode-then-score path, which cannot recover the raw responses from an encoded handle.

Parameters:
Return type:

ndarray

sample_ensemble(model, n, m, *, seed)[source]

Draw an (n, m) predictive ensemble: m iid draws repeated for n observations.

The plug-in predictive is exchangeable across observations, so one (m,) draw row is broadcast to all n rows – the per-observation CRPS/interval scores then differ only through y_i.

The realized draw count is checked against m rather than accepted and broadcast. Nothing verified it before, so a sampler that always returned two draws produced shape (n, 2) for m=10 – and m=0 still produced two. The resulting CRPS/interval score is then a genuinely different (far noisier) estimator than the one the objective’s ensemble argument advertises, with nothing in the score or the verdict to say so.

Parameters:
Return type:

ndarray