mixle.task.solve module

Train, calibrate, and serve a task model from an existing teacher.

solve converts a callable teacher into a deployable Solution. The teacher may be a rule cascade, legacy scoring routine, API client, or any other callable that currently performs the task. The solver labels example inputs with that teacher, trains a student matched to the input shape, calibrates an answer-or-escalate rule on held-out data, verifies agreement against the teacher, and returns a callable object that answers locally when calibrated confidence is sufficient.

Escalated requests remain useful after deployment. They are teacher-labeled examples from the part of the input space where the student abstained. Solution.improve() re-distills with those harvested labels and promotes the new student only when it preserves the verified agreement and escalation gates.

def route(ticket): … # existing production rule or service sol = solve(route, tickets) # dataset <- route(t) for t in tickets; train; calibrate sol(ticket) # answer locally or escalate to route() sol.report() # agreement, escalation rate, realized cost sol.improve() # fold escalations back in; promote only if better

solve is deterministic given seed. Only student training requires the optional neural dependency; the teacher remains an external callable.

load_harvested(path)[source]

Read harvested serving feedback into (inputs, answers).

Two JSONL formats are supported: {"input": ..., "label": ...} for classification feedback and {"input": ..., "answer": ...} for solution feedback. Classification labels are string-coerced; solution answers keep their JSON shape. Input JSON lists are restored as tuples so record-shaped examples can be passed back into solve/distillation workflows.

Parameters:

path (str)

Return type:

tuple[list, list]

class Solution(cascade, teacher, kind, train_inputs, train_labels, cal_inputs, cal_labels, holdout_agreement, escalation_rate, promoted, target_agreement, distill_kw=<factory>, ood=None, seed=0, synthesized=0, gate_inputs=<factory>, edge=None)[source]

Bases: object

A deployed task: a calibrated student in front of the teacher, plus the loop to improve it.

Call it like the original function. promoted says whether the student passed verification – when False the callable simply runs the teacher instead of deploying an unverified student.

Parameters:
report()[source]

What you would want on a dashboard: verification, live escalation, realized cost.

Return type:

dict

improve()[source]

Re-distill with the harvested (escalated) labels; promote only if it verifies at least as well.

Returns True when a better student was promoted. The calibration slice is never trained on, so the conformal guarantee and the agreement comparison remain valid across rounds.

Return type:

bool

health(recent_inputs=None, *, p_threshold=0.01)[source]

Check whether live escalation behavior has drifted from calibration.

The conformal answer-or-escalate rule is calibrated under an exchangeability assumption. When the input distribution shifts, the live escalation rate may move away from the verified baseline. This method compares the live rate with the baseline using an exact binomial test and, when recent_inputs and an OOD gate are available, compares the gate hit rate with its design quantile.

Returns a dictionary with drifted, live and baseline rates, and p-values where enough observations are available. A drift alarm means traffic has changed and retraining or review may be needed; abstained inputs still route to the teacher.

Parameters:

  • recent_inputs (Any)

  • p_threshold (float)

Return type:

dict[str, Any]

save(path)[source]

Persist the calibrated student as a load-anywhere artifact, with its verification record.

Every deployed artifact carries how it was verified — held-out agreement with the teacher, the escalation rate, the conformal alpha, and how much of its training data was synthetic — so “is this model trustworthy” is answerable from the artifact alone.

Parameters:

path (str)

Return type:

str

deploy(name, root='./mixle_data/registry')[source]

Save into the serving layout — {root}/tasks/{name} — the directory the mixle-mlops /v1/tasks routes serve from. Returns the artifact path.

Parameters:
Return type:

str

classmethod load(path, teacher, *, cost=None, device='cpu')[source]

Reconstitute a serving Solution from a saved artifact — the deploy path for a fresh process.

The loaded Solution answers locally / escalates to teacher and harvests labels exactly like the original. It carries no training or calibration data, so improve() raises — collect the harvested pairs and re-solve (real + harvested inputs) to train the next round.

Parameters:
Return type:

Solution

solve(teacher, inputs, *, alpha=0.1, target_agreement=None, holdout=0.25, kind=None, ood=0.02, propose=None, propose_budget=8, synthesize=0, prelabeled=None, device=None, device_space=None, cost=None, seed=0, **distill_kw)[source]

Replace teacher (the code currently doing the job) with a calibrated, self-improving model.

Parameters:

  • teacher (Callable[[...], Any]) – The callable performing the task today (per-item or batched). It labels the dataset and remains the fallback for inputs the student does not handle confidently.

  • inputs (Sequence[Any]) – Example inputs (text, or tuple/dict records) covering the task. The teacher labels them.

  • alpha (float) – Escalation honesty – answer locally only when a single label is conformally covered at >= 1 - alpha; otherwise fall back to the teacher.

  • target_agreement (float | None) – Optional gate. If the student’s held-out agreement with the teacher misses it, the returned Solution routes everything to the teacher (promoted=False).

  • holdout (float) – Fraction reserved for calibration + verification (never trained on).

  • kind (str | None) – Force the student path, 'text' or 'record'; default sniffs the first input.

  • ood (float | None) – Fit a p(x) gate over the training inputs and escalate inputs whose log p(x) falls below this quantile floor — so a wildly novel input escalates even when the softmax looks confident. On by default (0.02); None disables.

  • propose (str | None) – "auto" searches the student recipe (dim/hidden/epochs/lr, Bayesian-optimized on a val slice carved from the training split) instead of using the defaults. Teacher-free — the labels are already computed, so candidates cost only student fits.

  • propose_budget (int) – Total candidate recipes tried when propose="auto".

  • synthesize (int) – When example inputs are scarce, sample this many synthetic inputs from a generative model fit to the real training inputs (record inputs only) and have the teacher label them. Labels are always real (teacher-produced); the calibration slice and the OOD gate stay real-inputs-only, so the conformal guarantee and the p(x) floor reflect the true distribution.

  • prelabeled (tuple[Sequence[Any], Sequence[Any]] | None) – Already-teacher-labeled (inputs, labels) pairs — typically load_harvested("harvested.jsonl") from a serving deployment — folded into the TRAINING split (and the OOD gate: they are real traffic) but never into calibration, which stays a fresh split of inputs. This is the re-solve half of the serving loop.

  • device (Any) – A DeviceSpec makes this “give me this capability on that device”: the student is found by distill_for_edge() — a structure x precision x recipe search under the device’s hard byte/ops/torch-free budget (reusing the already-computed labels; the teacher is not re-called) — and the result’s footprint, Pareto front, and design ledger land on Solution.edge. If nothing fits the budget the Solution is demoted (everything routes to the teacher). Incompatible with propose="auto" (the device search subsumes it). A plain string (e.g. "cpu") keeps its old meaning: the torch training device.

  • device_space (Any) – Optional EdgeSpace constraining the device search (families, size ranges, precisions); default spans the standard space.

  • cost (Any) – Optional CostModel for realized-savings reporting.

  • seed (int) – Split + fit determinism.

  • **distill_kw (Any) – Student knobs forwarded to distillation (dim, hidden, epochs, lr, …). student="generative" swaps the hashed-feature MLP for mixle’s generative student — per-class token models for text (mixle.task.generative_text) or the structure-learned joint for records (distill_structured_from_labels()): exact posteriors, no torch needed at inference, and a built-in log p(x).

Returns:

A Solution – call it like the original function; report() / improve() / save().

Return type:

Solution