mixle.models.eval_harness module

General-capability eval harness: per-checkpoint receipts + regression tracking across a checkpoint sequence.

Roadmap F10. Mirrors the .report()-style receipt convention of mixle.utils.parallel.training_health.TrainingHealthMonitor (F4) and mixle.evolve.population.OperatorBandit / GenerationReport: no I/O, pure in-process accounting, JSON-serializable output.

The eval suite is a small, honest SYNTHETIC proxy, not a real published benchmark. It does not claim to measure MMLU-style world knowledge or HellaSwag-style commonsense; it measures four narrow, well-defined capability axes a decoder-only mixle.models.transformer.CausalLM can plausibly discriminate on at toy scale, each generated from a fixed, seedable procedure so scores are exactly reproducible and comparable across checkpoints:

• held-out perplexity – cross-entropy against sequences drawn from a fixed order-1 Markov chain over the model’s vocabulary (a stand-in for “does the model fit a held-out next-token distribution”, not any particular pretraining corpus).

• modular-arithmetic reasoning – a + b = ? under a small modulus, a single next-token prediction; a minimal, unambiguous proxy for symbolic/algorithmic reasoning.

• parity (counting) reasoning – the XOR parity of a random bitstring, a second algorithmic axis deliberately orthogonal to arithmetic (it requires tracking a running count, not a lookup table).

• in-context induction – a synthetic induction-head probe (... A B ... A -> ?, correct answer B): plant a bigram once in the context, then ask the model to complete it after seeing the first token again. This is the standard synthetic proxy for in-context learning (Olsson et al.’s induction heads).

One command per checkpoint (F10’s acceptance bar): evaluate_checkpoint() takes a model and returns a complete EvalReport – no multi-step manual orchestration.

Regression tracking across rungs and across the J2 compression ladder: track_regression() takes a sequence of EvalReport (successive training rungs, or successive steps of a not-yet-built J2 checkpoint-to-family compression ladder – e.g. one report per rung of mixle.models.qat / mixle.task.quantize.quantize_mlp() applied progressively) and flags any metric that moved measurably worse than its best-so-far value, beyond a stated relative threshold.

Integration points for later roadmap items (neither is required to exist for F10 to be useful today):

• E7 (long-context referee) – a not-yet-built long-context judge would slot in as one more task in _TASKS (or a second harness whose EvalReport merges into this one via EvalReport.tasks); nothing here assumes a fixed task count. EvalReport.metadata is free-form so a referee verdict can ride alongside these four scores without a schema change.

• J2 (compression ladder) – track_regression() takes an arbitrary ordered sequence of reports, so a ladder of checkpoints (fp32 -> QAT int8 -> QAT int4, or successive distillation students) is scored by calling evaluate_checkpoint() once per rung and handing the list straight to track_regression(); the checkpoint_id on each report is exactly the label J2 would assign to a ladder rung.

class TaskResult(name, score, higher_is_better, n_examples, details=<factory>)

Bases: object

One capability-axis score: a name, a scalar, and which direction is “better” for regression math.

Parameters:

• name (str)

• score (float)

• higher_is_better (bool)

• n_examples (int)

• details (dict[str, Any])

class EvalReport(checkpoint_id, tasks, seed, metadata=<factory>)

Bases: object

The per-rung receipt: every task’s score for one checkpoint, JSON-serializable via report().

Parameters:

• checkpoint_id (str)

• tasks (list[TaskResult])

• seed (int)

• metadata (dict[str, Any])

scores()

{task_name: score} – the compact view track_regression() consumes.

Return type:

dict[str, float]

class RegressionFlag(task, checkpoint_id, checkpoint_index, current_score, reference_score, reference_checkpoint_id, reference_index, relative_delta, threshold)

Bases: object

One metric that regressed beyond threshold relative to its best-so-far value in the sequence.

Parameters:

• task (str)

• checkpoint_id (str)

• checkpoint_index (int)

• current_score (float)

• reference_score (float)

• reference_checkpoint_id (str)

• reference_index (int)

• relative_delta (float)

• threshold (float)

class RegressionReport(flags, n_checkpoints, threshold, reference)

Bases: object

The regression-tracking receipt over an ordered sequence of EvalReport.

Parameters:

• flags (list[RegressionFlag])

• n_checkpoints (int)

• threshold (float)

• reference (str)

evaluate_checkpoint(model, *, checkpoint_id='checkpoint', seed=0, n_examples=256, metadata=None)

Run the full synthetic capability suite on model and return one structured EvalReport.

model is a real (or toy) mixle.models.transformer.CausalLM – anything exposing .vocab, .block, and __call__(x) -> (batch, vocab) next-token logits works. This is the single entry point: no separate setup per task, no manual orchestration – the “one command per checkpoint” F10 asks for.

Parameters:

• model (Any)

• checkpoint_id (str)

• seed (int)

• n_examples (int)

• metadata (dict[str, Any] | None)

Return type:

EvalReport

track_regression(reports, *, threshold=0.05, reference='best')

Flag metrics that regressed by more than threshold (relative) across a sequence of checkpoints.

reports is an ordered sequence – successive training rungs, or successive steps of a J2 compression ladder. For each task, each report (from the second onward) is compared against either the best score seen so far in the sequence (reference="best", the default – catches slow drift, not just one-step drops) or the immediately-prior report (reference="prior" – catches only step-to-step drops). A task is flagged when it moved worse than the reference by more than threshold as a fraction of the reference’s magnitude, direction-aware (a drop for accuracy-like metrics, a rise for perplexity-like metrics).

Parameters:

• reports (Sequence[EvalReport])

• threshold (float)

• reference (str)

Return type:

RegressionReport

markov_transition_matrix(vocab)

The fixed order-1 Markov chain the perplexity task scores against.

Keyed only on vocab (via a module-level constant seed), not on the caller’s sample seed – the benchmark distribution is a fixed property of the eval suite (like a real held-out benchmark), while the per-call seed only controls which samples from it are drawn for a given run. This is what lets a training loop legitimately learn this chain (it is a fixed, nameable distribution) while individual eval runs still draw fresh, unseen sequences from it.

Parameters:

vocab (int)

Return type:

ndarray