mixle.task.scorecard module

scorecard measures a deployed task route against its teacher.

Point it at a deployed Solution (or a Router), the teacher it replaces, and a held-out test set. The result measures end-to-end accuracy against the teacher, local-answer agreement, escalation rate, wall-clock latency for both sides, artifact size, and, when costs are supplied, realized dollars per thousand requests:

card = scorecard(sol, teacher, test_inputs, student_cost=0.0001, teacher_cost=0.03)
print(card.table())

metric                     student      teacher
end-to-end accuracy         1.000          —      (escalations answered by the teacher)
local agreement             0.964          —
escalation rate             0.11           —
p50 latency                 0.08 ms      2.1 ms
artifact size               210 KB         —
cost / 1k requests          $3.41        $30.00

“End-to-end accuracy” counts escalated requests as correct because the teacher answered them. “Local agreement” is the student alone on requests it chose to answer. Reporting both avoids hiding local-model errors behind the fallback.

Every solve shape gets receipts, with agreement meaning that shape’s own promise: classification = exact label match; RegressionSolution = within the caller’s tol; MultiLabelSolution = exact set match; StructuredSolution = every categorical field exact and every numeric field within its own tol.

class Scorecard(task, n_test, end_to_end_accuracy, local_agreement, escalation_rate, student_p50_ms, student_p95_ms, teacher_p50_ms, artifact_bytes, student_cost_per_1k, teacher_cost_per_1k)[source]

Bases: object

Evaluation summary for a distilled task service.

Parameters:

  • task (str)

  • n_test (int)

  • end_to_end_accuracy (float)

  • local_agreement (float)

  • escalation_rate (float)

  • student_p50_ms (float)

  • student_p95_ms (float)

  • teacher_p50_ms (float)

  • artifact_bytes (int | None)

  • student_cost_per_1k (float | None)

  • teacher_cost_per_1k (float | None)

as_dict()[source]

Return the scorecard fields as a plain dictionary.

Return type:

dict[str, Any]

table()[source]

Render a compact comparison table for local and teacher service metrics.

Return type:

str

scorecard(student, teacher, test_inputs, *, student_cost=None, teacher_cost=None, task='task')[source]

Measure a deployed student against the teacher it replaces on held-out inputs (see module docstring).

Parameters:

  • student (Any) – any solve shape — Solution, RegressionSolution, MultiLabelSolution, StructuredSolution (or anything exposing cascade.model.decide) — the escalate-aware system under test.

  • teacher (Any) – the callable being replaced; also the accuracy reference.

  • test_inputs (Any) – held-out inputs (the teacher is called once per input for the reference labels).

  • teacher_cost (float | None) – optional per-request costs for the $/1k rows. The blended student cost prices escalated requests at teacher_cost.

  • task (str) – a label for the table header.

  • student_cost (float | None)

  • teacher_cost

Return type:

Scorecard