mixle.task.scorecard module

scorecard measures a deployed task route against teacher and task truth.

Point it at a deployed Solution (or a Router), the teacher it replaces, and a held-out test set. Optionally provide an independent task_truth callable or aligned label panel to measure real end-to-end accuracy. Teacher agreement remains a separate distillation metric. Teacher outputs are cached from the same calls used for latency, and realized cost charges the student attempt on every request plus the teacher on escalations:

card = scorecard(
    sol, teacher, test_inputs, task_truth=gold_labels,
    student_cost=0.0001, teacher_cost=0.03,
)
print(card.table())

metric                     student      teacher
end-to-end accuracy         0.982          —      (against task truth)
served teacher agreement    0.991          —
local teacher agreement     0.964          —
escalation rate             0.11           —
p50 latency                 0.08 ms      2.1 ms
artifact size               210 KB         —
cost / 1k requests          $3.41        $30.00

Without task_truth, end-to-end accuracy is explicitly unmeasured rather than declaring every teacher escalation correct by construction. teacher_agreement reports agreement of the served route (local answer or teacher fallback) with the cached teacher output; local_agreement covers locally answered rows only.

Every solve shape gets receipts, with agreement meaning that shape’s own promise: classification = exact label match; RegressionSolution = within the caller’s tol; MultiLabelSolution = exact set match; StructuredSolution = every categorical field exact and every numeric field within its own tol.

class Scorecard(task, n_test, end_to_end_accuracy, teacher_agreement, local_agreement, local_accuracy, truth_source, escalation_rate, student_p50_ms, student_p95_ms, teacher_p50_ms, artifact_bytes, student_cost_per_1k, teacher_cost_per_1k)[source]

Bases: object

Evaluation summary for a distilled task service.

Parameters:

  • task (str)

  • n_test (int)

  • end_to_end_accuracy (float | None)

  • teacher_agreement (float)

  • local_agreement (float)

  • local_accuracy (float | None)

  • truth_source (str)

  • escalation_rate (float)

  • student_p50_ms (float)

  • student_p95_ms (float)

  • teacher_p50_ms (float)

  • artifact_bytes (int | None)

  • student_cost_per_1k (float | None)

  • teacher_cost_per_1k (float | None)

as_dict()[source]

Return the scorecard fields as a plain dictionary.

Return type:

dict[str, Any]

table()[source]

Render a compact comparison table for local and teacher service metrics.

Return type:

str

scorecard(student, teacher, test_inputs, *, task_truth=None, student_cost=None, teacher_cost=None, task='task')[source]

Measure a deployed route against its teacher and optional independent truth.

Parameters:

  • student (Any) – any solve shape — Solution, RegressionSolution, MultiLabelSolution, StructuredSolution (or anything exposing cascade.model.decide) — the escalate-aware system under test.

  • teacher (Any) – the callable being replaced; used only as the distillation-agreement reference.

  • test_inputs (Any) – held-out inputs. The teacher is called exactly once per input; that call supplies both the cached reference output and its latency.

  • task_truth (Callable[[Any], Any] | Sequence[Any] | None) – independent truth callable or an aligned sequence of gold outputs. When omitted, end_to_end_accuracy is None rather than a tautological teacher-fallback score.

  • teacher_cost (float | None) – optional non-negative per-request costs. A served route costs student_cost on every attempted request plus teacher_cost on every escalation.

  • task (str) – a label for the table header.

  • student_cost (float | None)

  • teacher_cost

Return type:

Scorecard