mixle.scorecard module

System-level scorecard for held-out question sets.

Named SystemScorecard (not Scorecard) to avoid colliding with mixle.task.scorecard.Scorecard – a different, narrower comparison (one student solution vs its teacher on a task); this one evaluates a whole System (teacher, captured cache, degraded modes, budget, all of it) end to end.

class SystemScorecard(quality, calibration, realized_cost, grounded_fraction, n)

Bases: object

One evaluation of a System against a fixed held-out question set.

Parameters:

• quality (float)

• calibration (float)

• realized_cost (float)

• grounded_fraction (float)

• n (int)

to_dict()

Serialize the scorecard into primitive fields.

Return type:

dict[str, Any]

evaluate(system, question_set, *, scorer=None)

Evaluate system over a fixed [(query, expected_answer), ...] set.

• quality – fraction of questions scorer judges correct (default: case-insensitive substring match of expected in the reply).

• grounded_fraction – fraction answered WITHOUT a degraded mode (teacher or captured, not a store-only fallback / refusal / failure): the fraction of answers you can actually trust came from a real answer path, not a fault-boundary guess.

• realized_cost – total spend units (total_units()) across the set.

• calibration – a coarse proxy (currently equal to quality) until answer carries a real per-answer confidence to calibrate against; extend THIS function, not call sites, once that lands.

Parameters:

• system (System)

• question_set (Sequence[tuple[Query, str]])

• scorer (Callable[[str | None, str], bool] | None)

Return type:

SystemScorecard

class RegressionReport(regressed, reasons=<factory>)

Bases: object

Whether current is worse than baseline on any tracked axis, and exactly why.

Parameters:

• regressed (bool)

• reasons (list[str])

detect_regression(baseline, current, *, tolerance=1e-9)

Compare two scorecards from the SAME held-out set across improve-rounds.

Never silently accepts a round that answers worse, less groundedly, or for more cost than the round before it – each tracked axis that got worse (beyond tolerance) is named in reasons.

Parameters:

• baseline (SystemScorecard)

• current (SystemScorecard)

• tolerance (float)

Return type:

RegressionReport