mixle.task.scorecard module

scorecard – the receipts: tiny model vs frontier on YOUR task, measured, in one table.

The artifact that wins arguments. Point it at a deployed Solution (or a Router), the teacher it replaces, and a held-out test set; it MEASURES — never projects — end-to-end accuracy against the teacher, local-answer agreement, escalation rate, wall-clock latency for both sides, artifact size, and (given per-request costs) realized $/1k:

card = scorecard(sol, teacher, test_inputs, student_cost=0.0001, teacher_cost=0.03)
print(card.table())

metric                     student      teacher
end-to-end accuracy         1.000          —      (escalations answered BY the teacher)
local agreement             0.964          —
escalation rate             0.11           —
p50 latency                 0.08 ms      2.1 ms
artifact size               210 KB         —
cost / 1k requests          $3.41        $30.00

“End-to-end accuracy” counts escalated requests as correct-by-construction (the teacher answered them); “local agreement” is the student alone on the requests it chose to answer — both are reported so the honest story is visible: the system is never worse than the teacher, and here is exactly how much of the work the tiny model absorbed.

Every solve shape gets receipts, with agreement meaning that shape’s own promise: classification = exact label match; RegressionSolution = within the caller’s tol; MultiLabelSolution = exact set match; StructuredSolution = every categorical field exact and every numeric field within its own tol.

class Scorecard(task: 'str', n_test: 'int', end_to_end_accuracy: 'float', local_agreement: 'float', escalation_rate: 'float', student_p50_ms: 'float', student_p95_ms: 'float', teacher_p50_ms: 'float', artifact_bytes: 'int | None', student_cost_per_1k: 'float | None', teacher_cost_per_1k: 'float | None')[source]

Bases: object

Parameters:

  • task (str)

  • n_test (int)

  • end_to_end_accuracy (float)

  • local_agreement (float)

  • escalation_rate (float)

  • student_p50_ms (float)

  • student_p95_ms (float)

  • teacher_p50_ms (float)

  • artifact_bytes (int | None)

  • student_cost_per_1k (float | None)

  • teacher_cost_per_1k (float | None)

task: str
n_test: int
end_to_end_accuracy: float
local_agreement: float
escalation_rate: float
student_p50_ms: float
student_p95_ms: float
teacher_p50_ms: float
artifact_bytes: int | None
student_cost_per_1k: float | None
teacher_cost_per_1k: float | None
as_dict()[source]
Return type:

dict[str, Any]

table()[source]
Return type:

str

scorecard(student, teacher, test_inputs, *, student_cost=None, teacher_cost=None, task='task')[source]

Measure a deployed student against the teacher it replaces on held-out inputs (see module docstring).

Parameters:

  • student (Any) – any solve shape — Solution, RegressionSolution, MultiLabelSolution, StructuredSolution (or anything exposing cascade.model.decide) — the escalate-aware system under test.

  • teacher (Any) – the callable being replaced; also the accuracy reference.

  • test_inputs (Any) – held-out inputs (the teacher is called once per input for the reference labels).

  • teacher_cost (float | None) – optional per-request costs for the $/1k rows. The blended student cost prices escalated requests at teacher_cost.

  • task (str) – a label for the table header.

  • student_cost (float | None)

  • teacher_cost

Return type:

Scorecard