Evolution And Analysis

This tutorial connects analysis diagnostics to an auditable improvement loop. It is for cases where you already have a working model and want controlled change, not blind hyperparameter search.

Start With A Champion

Assume a champion model already exists. The improvement loop needs data, an objective, and a verification standard.

from mixle.evolve import EvolutionLedger, improve, nll_objective

ledger = EvolutionLedger()

result = improve(
    champion,
    data,
    objective=nll_objective(),
    holdout=0.25,
    alpha=0.05,
    min_effect=0.01,
    ledger=ledger,
)

champion = result.model

The returned model is only a verified challenger if result.verified is true. Otherwise the original champion remains in place.

Record both outcomes. A rejected challenger is useful evidence: it tells future searches which direction did not clear the gate.

Add A Tail Diagnostic

Likelihood can improve while tail behavior gets worse. Use analysis utilities to inspect the residuals or losses that matter operationally.

import numpy as np
from mixle.analysis import peaks_over_threshold, return_level

residuals = np.asarray([abs(y - champion.predict(x)) for x, y in validation])
tail = peaks_over_threshold(residuals, threshold=np.quantile(residuals, 0.95))
level = return_level(tail, period=100)

You can track this diagnostic in the ledger metadata or use it to define a custom objective.

The reason to keep this separate from the primary likelihood objective is governance. A model can improve average log score while becoming worse exactly where the application is most sensitive.

Define A Promotion Gate

A promotion decision should combine the objective result and the diagnostics that matter for the application.

passed = (
    result.verified
    and level < champion_tail_limit
    and result.delta >= 0.01
)

if passed:
    champion = result.model
    ledger.record(
        operator="promotion_gate",
        delta=result.delta,
        verdict={"promote": True},
        cost=0.0,
        parent_hash=result.parent_hash,
        meta={"tail_level": level},
    )
else:
    ledger.record(
        operator="promotion_gate",
        delta=result.delta,
        verdict={"promote": False},
        cost=0.0,
        parent_hash=result.parent_hash,
        meta={"tail_level": level},
    )

The exact fields depend on the ledger object and your application, but the principle is stable: the gate should be explicit enough to audit later.

Search A Typed Space

For a larger model-design question, define a typed search space and a builder.

from mixle.evolve import Categorical, Integer, Real, Space, search

space = Space({
    "components": Integer(1, 5),
    "alpha": Real(0.1, 4.0, log=True),
    "family": Categorical(["gaussian", "student_t"]),
})

def build_fn(config):
    return fit_candidate(data, config)

found = search(
    space,
    data,
    objective=nll_objective(),
    build_fn=build_fn,
    method="bo",
    n_iter=25,
)

challenger = found.best_model

Search proposes candidates. Verification still decides promotion.

Typed spaces are preferable to ad hoc dictionaries because the search algorithm knows which dimensions are categorical, integer, continuous, or log-scaled.

Promote Deliberately

Before replacing a model, ask:

• Did the challenger improve the primary objective?

• Did it preserve calibration?

• Did it avoid worse tail behavior or decision regret?

• Is the evaluation split representative of production traffic?

• Are the rejected candidates and reasons recorded?

That discipline is what makes automatic improvement compatible with professional model governance.

Read Analysis Utilities for diagnostics and Evolution And Search for search spaces, objectives, verification, and ledgers.