mixle.task.structured_out module

solve_structured – replace rigid code that returns a dict with one joint coverage contract.

The structured-output shape of the solve loop: teacher(x) -> {"field": value, ...} with a consistent schema (an enricher, a triager, a quote builder). Rather than inventing new machinery, each output field decomposes onto the shape that already carries guarantees:

• a categorical/string field -> a solve() classifier (conformal singleton);

• a numeric field -> a solve_regression() student (conformal interval + the caller’s tol precision rule – required per numeric field).

The composition rule is strict: a separate held-out slice calibrates the maximum nonconformity across all fields. The input is answered locally only when every categorical prediction set is a singleton and every numeric interval meets its tolerance. Thus the complete record, rather than each field marginally, has the advertised 1 - alpha split-conformal coverage contract.

Scope of that statement. The joint 1 - alpha coverage is finite-sample and MARGINAL over the calibration draw and the query jointly, under exchangeability of the calibration rows and incoming traffic. It is NOT an accuracy guarantee conditional on answering locally: serving conditions on every field simultaneously clearing its singleton/tolerance gate, and coverage conditional on that event is not controlled. Answered-slice quality is a MEASUREMENT, never a guarantee, and report() separates the two kinds of numbers: the per-field holdout_agreement values are RAW route-independent sub-model metrics (the joint serving gate never runs in them), while answered_slice applies the real joint gate to a disjoint record-level evaluation slice and reports how many records it answered, how many answered records matched the teacher’s record (categoricals exactly, numerics within their tolerance), and an exact 95% Clopper-Pearson interval for that conditional agreement (None when it answered none). Distribution shift or any other break of exchangeability voids the statement silently; re-measure on drifted traffic. report() carries this scope machine-readably in coverage_contract_scope.

improve() pushes each harvested (input, dict) down into every field’s own harvest buffer and runs each sub-solution’s anti-regression improve. No structured-level OOD gate yet (the classifier fields’ own gates are off here to avoid redundant vetoes) – noted, not hidden.

class StructuredSolution(fields_cat, fields_num, teacher, joint_qhat=inf, alpha=0.1, numeric_tolerances=<factory>, eval_rows=0, answered_eval_n=0, answered_eval_correct=0, calibration_receipt=<factory>, n_requests=0, n_escalated=0, harvested_inputs=<factory>, harvested_outputs=<factory>)

Bases: AnsweredSliceGuard

Per-field calibrated students in front of the dict-valued routine they replace.

Parameters:

• fields_cat (dict[str, Solution])

• fields_num (dict[str, RegressionSolution])

• teacher (Callable[[...], Any])

• joint_qhat (float)

• alpha (float)

• numeric_tolerances (dict[str, float])

• eval_rows (int)

• answered_eval_n (int)

• answered_eval_correct (int)

• calibration_receipt (dict[str, Any])

• n_requests (int)

• n_escalated (int)

• harvested_inputs (list)

• harvested_outputs (list)

property schema: dict[str, str]

categorical or numeric.

Type:

Return each output field’s inferred kind

try_local(x)

The fully-decided output dict, or None when ANY field is unsure (= must escalate).

Parameters:

x (Any)

Return type:

dict[str, Any] | None

decide(x)

Return the local structured-output decision, or None when the example should escalate.

Parameters:

x (Any)

Return type:

dict[str, Any] | None

report()

Return per-field calibration details and aggregate serving/harvest counts.

Return type:

dict[str, Any]

save(path)

Persist every field’s sub-artifact under one directory; load() restores the whole schema.

Parameters:

path (str)

Return type:

str

classmethod load(path, teacher, *, device='cpu')

Reconstitute a serving StructuredSolution (fields serve locally; escalation runs teacher).

Parameters:

• path (str)

• teacher (Callable[[...], Any])

• device (str)

Return type:

StructuredSolution

improve()

Refuse unsafe adaptive reuse of the joint calibration slice.

Re-solve with the harvested rows as prelabeled= and a fresh base sample. That path creates a new untouched joint calibration slice; silently recycling the old one would void coverage.

Return type:

bool

solve_structured(teacher, inputs, *, schema=None, tol=None, alpha=0.1, prelabeled=None, seed=0, **sub_kw)

Replace a dict-valued routine with per-field calibrated students (see module docstring).

Parameters:

• teacher (Callable[[...], Any]) – teacher(x) -> dict with a consistent schema; called once per example input.

• inputs (Sequence[Any]) – example inputs (text or dict/tuple records).

• tol (dict[str, float] | float | None) – the precision requirement for numeric fields — a scalar for all, or {field: tol}. Required when the schema has numeric fields.

• alpha (float) – shared miscoverage level for every field’s calibration.

• prelabeled (tuple[Sequence[Any], Sequence[dict]] | None) – already-teacher-labeled (inputs, output_dicts) — typically harvested escalations from a serving deployment — fanned down into every field’s TRAINING split only, never calibration (each sub-solution’s guarantee stays a fresh split of inputs). The schema stays authoritative from the inputs pass; a pair missing a field is simply skipped for that field.

• **sub_kw (Any) – knobs forwarded to every sub-solve (epochs, hidden, dim, …).

• schema (dict[str, str] | None)

• seed (int)

• **sub_kw

Return type:

StructuredSolution