mixle.system.scorecard module

System-level scorecard for held-out question sets.

Named SystemScorecard (not Scorecard) to avoid colliding with mixle.task.scorecard.Scorecard – a different, narrower comparison (one student solution vs its teacher on a task); this one evaluates a whole System (teacher, captured cache, degraded modes, budget, all of it) end to end.

DEFAULT_SCORER_VERSION = 'substring/v2'

Bumped whenever _default_scorer()’s judgement changes, so cards judged by different versions of the default judge get different question_set_identity() values and cannot be compared.

question_set_identity(question_set, *, scorer=None)[source]

The identity two scorecards must share before detect_regression() may compare them.

A regression comparison is only meaningful between cards measured on the SAME held-out set with the SAME judge: a “perfect” card over a different (or empty) question set, or the same set scored by a laxer scorer, is not evidence that a round did not regress. Digesting the (query, expected) pairs in order plus the scorer’s identity turns that documented precondition into something detect_regression() can actually check, instead of silently accepting mismatched evidence.

The pairs are digested by CONTENT (mixle.data.hashing.dataset_hash(), order-sensitive), never by repr (MXR-080-1902). Query.fingerprint is typed Any, and repr of an ordinary object embeds its memory address, so two equal question sets built from equal fingerprint objects digested differently – and detect_regression() then reported an improved round against the same held-out set as comparable=False, regressed=True, a fabricated regression on the exact gate this identity exists to make trustworthy. dataset_hash encodes a Query structurally by its declared dataclass fields and refuses (rather than repr-ing) any value it cannot encode canonically, which is what makes an identity computed in one process match one computed in another.

Note that this changes the digest of every question set relative to the repr-based scheme, so an identity persisted by an older build will not match one computed here. That is unavoidable and correct: the old digest’s whole problem is that it did not identify what it claimed to.

Parameters:
Return type:

str

class SystemScorecard(quality, realized_cost, grounded_fraction, n, question_set_id='')[source]

Bases: object

One evaluation of a System against a fixed held-out question set.

Every metric is validated on construction: a card carrying NaN/out-of-range numbers or a negative case count is not a weaker measurement, it is not a measurement at all, and detect_regression() silently treated such a card as “no regression” (fail-open on the exact gate meant to catch a worsening round). question_set_id binds the card to the held-out set and judge it was measured with (see question_set_identity()).

Parameters:
to_dict()[source]

Serialize the scorecard into primitive fields.

Return type:

dict[str, Any]

evaluate(system, question_set, *, scorer=None)[source]

Evaluate system over a fixed [(query, expected_answer), ...] set.

Every expected is validated up front: a blank reference cannot discriminate, and a question set containing one is rejected rather than scored (it used to make every non-None reply correct and report quality=1.0).

  • quality – fraction of questions scorer judges correct. The default judge is _default_scorer(), a deliberately weak lexical baseline (case-insensitive substring match) that cannot see negation or paraphrase; read its docstring before treating quality as authoritative, and pass a task-specific scorer where correctness actually matters.

  • grounded_fraction – fraction answered WITHOUT a degraded mode (teacher or captured, not a store-only fallback / refusal / failure): the fraction of answers you can actually trust came from a real answer path, not a fault-boundary guess.

  • realized_cost – total spend units (total_units()) across the set.

  • question_set_id – the identity of the held-out set and judge this card was measured with, so detect_regression() can refuse to compare cards from different evidence.

There is deliberately NO calibration field. One used to exist and was assigned quality verbatim: a deterministic system that emits no confidence at all scored a perfect “calibration”, and detect_regression() never compared the field anyway, so a drop from 1.0 to 0.0 was not a regression. answer() carries no per-answer confidence to calibrate against, so there is nothing here to compute; when it does, add a real calibration statistic to THIS function and a matching axis to detect_regression() in the same change, rather than re-adding a copied number that reads as evidence of something never measured.

This is a genuinely read-only pass: every call goes through answer() with read_only=True, so evaluating a held-out question_set can never itself teach system the held-out answers (no promotion into the harvest a later improve() could capture, no change to system.total_spend). Without that, a held-out set would stop being held-out the moment it was first evaluated: a later improve() could promote its answers into the captured cache, and re-evaluating the “held-out” set afterward would be trivially free and perfect. Never call system.answer without read_only=True from this function.

Parameters:
Return type:

SystemScorecard

class RegressionReport(regressed, reasons=<factory>, comparable=True)[source]

Bases: object

Whether current is worse than baseline on any tracked axis, and exactly why.

comparable is False when the two cards are not evidence about the same thing (different held-out set/judge, different case counts, an unidentified card). An incomparable pair is reported as regressed=True: a comparison that could not be made is not a passed comparison.

Parameters:
detect_regression(baseline, current, *, tolerance=1e-9)[source]

Compare two scorecards from the SAME held-out set across improve-rounds.

Never silently accepts a round that answers worse, less groundedly, or for more cost than the round before it – each tracked axis that got worse (beyond tolerance) is named in reasons.

The “same held-out set” precondition is enforced, not merely documented: both cards must carry the same non-empty question_set_identity() and the same case count. Comparing a card against evidence from a different (or empty) question set cannot show that a round did not regress, so such a pair comes back comparable=False, regressed=True rather than a clean bill of health. tolerance must itself be finite and nonnegative – a NaN tolerance makes every ordered comparison below false, which silently turned this gate into an unconditional “no regression”.

Parameters:

  • baseline (SystemScorecard)

  • current (SystemScorecard)

  • tolerance (float)

Return type:

RegressionReport