mixle.task.capability module

Behavioral capability profiles for distilled students.

Two students can share the same clean-holdout accuracy and still differ wildly on what they capture: one degrades gracefully under typos, the other collapses; one honors the teacher’s case-insensitivity, the other doesn’t. A capability suite names the input distribution’s corruptions (severity levels), invariances (meaning-preserving rewrites the teacher is expected to honor), and edge-case probes; capture_profile() runs both the student and the teacher through all three and reports a JSON-serializable profile. The profile intentionally leaves pass/fail policy to the caller.

class CapabilitySuite(corruptions=<factory>, invariances=<factory>, probes=<factory>)[source]

Bases: object

The behavioral spec an example distillation is checked against.

corruptions maps a named severity level (e.g. "typo_10") to a text -> text corruption; insertion order is the intended severity order (mild first) so callers can read the profile’s ordering directly. invariances maps a name to a meaning-preserving rewrite (case jitter, whitespace, a synonym swap) – a well-behaved model’s prediction should not change under it. probes are fixed edge-case inputs whose raw predictions are recorded without assuming ground truth.

Parameters:
keyboard_typo_corruption(rate, *, seed=0)[source]

A corruption: replace each letter with a random lowercase letter independently with probability rate.

Deterministic given seed – the same corruption function always maps the same text to the same output.

Parameters:
Return type:

Callable[[str], str]

case_jitter_invariance(text)[source]

A meaning-preserving rewrite: swap the case of every letter.

Parameters:

text (str)

Return type:

str

whitespace_invariance(text)[source]

A meaning-preserving rewrite: collapse all whitespace runs to single spaces.

Parameters:

text (str)

Return type:

str

capture_profile(student, teacher, texts, suite)[source]

Run student and teacher through suite and return a profile.

Returns a plain, json.dumps-safe dict:

  • "clean_agreement" – student/teacher label agreement on the uncorrupted texts;

  • "corruptions" – per corruption name, student/teacher agreement on the corrupted texts (in the suite’s insertion order, mild-to-severe by convention);

  • "invariances" – per invariance name, {"student_violation_rate", "teacher_violation_rate"}: how often each side’s prediction changes under a rewrite that should not change it. A student must not be penalized for an invariance the teacher itself violates – both rates are reported, never one diff;

  • "probes" – {"student": [...], "teacher": [...]} raw predictions on the fixed probe inputs, or omitted if the suite has no probes;

  • "abstention" – present only if student or teacher exposes a decision API (decide / batch_decide): each side’s escalation rate on texts (None for a side with no decision API).

There is deliberately no single aggregate score field.

Parameters:
Return type:

dict[str, Any]