mixle.task.solve module

Train, calibrate, and serve a task model from an existing teacher.

solve converts a callable teacher into a deployable Solution. The teacher may be a rule cascade, legacy scoring routine, API client, or any other callable that currently performs the task. The solver labels example inputs with that teacher, trains a student matched to the input shape, calibrates an answer-or-escalate rule on held-out data, verifies agreement against the teacher, and returns a callable object that answers locally when calibrated confidence is sufficient.

Escalated requests remain useful after deployment. They are teacher-labeled examples from the part of the input space where the student abstained. Solution.improve() re-distills with those harvested labels and promotes the new student only when it preserves the verified agreement and escalation gates.

def route(ticket): … # existing production rule or service sol = solve(route, tickets) # dataset <- route(t) for t in tickets; train; calibrate sol(ticket) # answer locally or escalate to route() sol.report() # agreement, escalation rate, realized cost sol.improve() # fold escalations back in; promote only if better

solve is deterministic given seed. Only student training requires the optional neural dependency; the teacher remains an external callable.

load_harvested(path)

Read harvested serving feedback into (inputs, answers).

Two JSONL formats are supported: {"input": ..., "label": ...} for classification feedback and {"input": ..., "answer": ...} for solution feedback. Classification labels are string-coerced; solution answers keep their JSON shape. Input JSON lists are restored as tuples so record-shaped examples can be passed back into solve/distillation workflows.

Parameters:

path (str)

Return type:

tuple[list, list]

class Solution(cascade, teacher, kind, train_inputs, train_labels, cal_inputs, cal_labels, holdout_agreement, escalation_rate, promoted, target_agreement, distill_kw=<factory>, ood=None, seed=0, synthesized=0, gate_inputs=<factory>, edge=None, device=None, device_space=None, propose_budget=8, verification_digest=None, sel_inputs=<factory>, sel_labels=<factory>, selection_uses=0, calibration_evidence='solve-split', selection_receipt=<factory>, sel_rows=0, answered_sel_n=0, answered_sel_correct=0)

Bases: object

A deployed task: a calibrated student in front of the teacher, plus the loop to improve it.

Call it like the original function. promoted says whether the student passed verification – when False the callable simply runs the teacher instead of deploying an unverified student. Verification covers the live rule, not just raw agreement: a student whose calibrated answer-or-escalate rule answered zero selection rows would escalate every request, so it is demoted rather than presented as a working local answerer.

Parameters:

• cascade (Cascade)

• teacher (Callable[[...], Any])

• kind (str)

• train_inputs (list)

• train_labels (list)

• cal_inputs (list)

• cal_labels (list)

• holdout_agreement (float)

• escalation_rate (float)

• promoted (bool)

• target_agreement (float | None)

• distill_kw (dict)

• ood (float | None)

• seed (int)

• synthesized (int)

• gate_inputs (list)

• edge (Any)

• device (Any)

• device_space (Any)

• propose_budget (int)

• verification_digest (str | None)

• sel_inputs (list)

• sel_labels (list)

• selection_uses (int)

• calibration_evidence (str)

• selection_receipt (list[dict])

• sel_rows (int)

• answered_sel_n (int)

• answered_sel_correct (int)

property selection_evidence_is_single_use: bool

True while holdout_agreement is still an untouched-once held-out number.

report()

What you would want on a dashboard: verification, live escalation, realized cost.

Return type:

dict

improve(evidence_inputs=None)

Re-distill with the harvested (escalated) labels; promote only if it verifies at least as well.

Returns True when a better student was promoted. Neither reserved role is ever trained on.

Roles (MXR-080-1891). The promote/reject decision reads only the SELECTION rows and the conformal threshold is set only from the CALIBRATION rows, so the promoted student is no longer chosen with the same data that certifies its coverage. Passing evidence_inputs supplies a fresh, teacher-labeled batch that REPLACES both roles, which is the only way to keep the reported agreement a genuine single-use held-out number across rounds; without it the existing selection rows decide again and selection_uses records that the number is now a selection score. The reuse is recorded rather than refused because the no-argument loop is the documented serving workflow and a caller with no fresh traffic still needs the anti-regression gate.

What reuse does to the threshold’s guarantee (STAT-RR12-1). The role split does NOT restore the finite-sample coverage statement for a no-argument promotion: the harvested escalations that train the candidate were selected per-query by the OLD threshold, itself a function of the calibration rows – so those rows helped construct the candidate and cannot also certify it (the regression loop’s all-or-none version of this leak measured 0.8857 against a claimed 0.90; the per-query selection here is the stronger channel, and unlike regression the escalated slice is threshold-selected, so it cannot serve as fresh calibration either). After such a promotion report()['calibration_evidence'] says "reused-after-adaptive-harvest" and the deployed threshold’s nominal statement should be treated as empirical; a promotion fed by evidence_inputs records "fresh-evidence" and keeps the exact statement.

When the original solve() ran under a device budget (self.edge is set), the harvested labels are re-searched under that SAME DeviceSpec/EdgeSpace – warm-started from the prior search’s design ledger – instead of being refit with a generic, unconstrained student. A candidate that no longer fits the device is rejected exactly like any other anti-regression failure, so a promoted student never silently exceeds the budget device= promised.

Parameters:

evidence_inputs (Sequence[Any] | None)

Return type:

bool

health(recent_inputs=None, *, p_threshold=0.01)

Check whether live escalation behavior has drifted from calibration.

The conformal answer-or-escalate rule is calibrated under an exchangeability assumption. When the input distribution shifts, the live escalation rate may move away from the verified baseline. The baseline is an ESTIMATE from the selection rows, not a known constant, so the live and baseline samples are compared with a TWO-SAMPLE exact test (Fisher) on the two escalation counts. Testing the live count against the plug-in baseline rate as if it were exact false-alarms at rates far above nominal – measured 57-71% at p_threshold=0.01 under NO drift across (n_selection, n_live) of (10, 500), (50, 1000), (100, 5000), versus 0.4-0.8% for the two-sample test on the same draws – and worsens as live traffic grows, because any baseline estimation error eventually dominates. The OOD comparison is a one-sample exact test against the gate’s DESIGN quantile – a chosen budget, not an estimate – and it is one-sided (greater): the gate exists to catch novelty floods, so only firing significantly ABOVE budget alarms; under-firing is reported in the rates but is not drift.

Returns a dictionary with drifted, live and baseline rates, and p-values where enough observations are available (the escalation p-value needs the selection-slice counts, which every solved or loaded 0.8.0 Solution carries; an object with sel_rows == 0 reports rates without an escalation p-value). A drift alarm means traffic has changed and retraining or review may be needed; abstained inputs still route to the teacher.

Parameters:

• recent_inputs (Any)

• p_threshold (float)

Return type:

dict[str, Any]

save(path)

Persist the calibrated student as a load-anywhere artifact, with its verification record.

The artifact holds the student, its calibration, and the verification record – not the teacher, which is an arbitrary callable: reloading in a fresh process is Solution.load(path, teacher) with the escalation target re-supplied.

Every deployed artifact carries how it was verified — held-out agreement with the teacher, the escalation rate, the conformal alpha, and how much of its training data was synthetic — so “is this model trustworthy” is answerable from the artifact alone.

Parameters:

path (str)

Return type:

str

deploy(name, root='./mixle_data/registry')

Save into the serving layout — {root}/tasks/{name} — the directory the mixle-mlops /v1/tasks routes serve from. Returns the artifact path.

name must be a single path component. It was joined onto root unchecked, and both pathlib and os.path.join will leave a root when asked to: name="../../escaped" traversed out of it and an absolute name discarded it entirely, so a deployment could write anywhere the process could (MXR-080-1910). A serving name arrives from an API request or a config file, so this is the ordinary case, not an exotic one.

Parameters:

• name (str)

• root (str)

Return type:

str

classmethod load(path, teacher, *, cost=None, device='cpu')

Reconstitute a serving Solution from a saved artifact — the deploy path for a fresh process.

The loaded Solution answers locally / escalates to teacher and harvests labels exactly like the original. It carries no training or calibration data, so improve() raises — collect the harvested pairs and re-solve (real + harvested inputs) to train the next round.

Parameters:

• path (str)

• teacher (Callable[[...], Any])

• cost (Any)

• device (str)

Return type:

Solution

solve(teacher, inputs, *, alpha=0.1, target_agreement=None, holdout=0.25, kind=None, ood=0.02, propose=None, propose_budget=8, synthesize=0, prelabeled=None, device=None, device_space=None, cost=None, seed=0, teacher_mode='auto', **distill_kw)

Replace teacher (the code currently doing the job) with a calibrated, self-improving model.

Parameters:

• teacher (Callable[[...], Any]) – The callable performing the task today (per-item or batched). It labels the dataset and remains the fallback for inputs the student does not handle confidently.

• inputs (Sequence[Any]) – Example inputs (text, or tuple/dict records) covering the task. The teacher labels them.

• alpha (float) – Miscoverage budget for the conformal prediction sets. Under exchangeability of the calibration rows and incoming queries, a set covers the teacher’s label with probability >= 1 - alpha MARGINALLY over queries; the student answers locally only when its set is a single label and escalates otherwise. Marginal set coverage is NOT a bound on the error of the locally-answered slice (answering conditions on the set being a singleton), so answered-slice agreement is reported as a measurement – report()['answered_slice'] runs the real answer-or-escalate rule over the selection rows and carries the answered denominator and an exact 95% Clopper-Pearson interval (None when the rule answered none of them, in which case the Solution is demoted) – certify an answered-slice target separately with mixle.task.calibrate.CalibratedTaskModel.calibrate_selective(). Both statements fail silently under distribution shift; the ood gate below mitigates, and drifted traffic calls for re-measurement. The statement covers the INITIAL solve and any improve(evidence_inputs=...) promotion; a no-argument promotion recalibrates on rows that already shaped the candidate through the harvest, voiding the finite-sample statement for that artifact – report()['calibration_evidence'] says which regime applies (see Solution.improve()).

• target_agreement (float | None) – Optional gate. If the student’s held-out agreement with the teacher misses it, the returned Solution routes everything to the teacher (promoted=False).

• holdout (float) – Fraction reserved for calibration + verification (never trained on). Those are two IMMUTABLE roles, split deterministically the moment the holdout is drawn: the calibration rows set the conformal threshold and the selection rows produce holdout_agreement / escalation_rate and decide every Solution.improve() promotion. Nothing reads both.

• kind (str | None) – Force the student path, 'text' or 'record'; default sniffs the first input.

• ood (float | None) – Fit a p(x) gate over the training inputs and escalate inputs whose log p(x) falls below this quantile floor — so a wildly novel input escalates even when the softmax looks confident. On by default (0.02); None disables.

• propose (str | None) – "auto" searches the student recipe (dim/hidden/epochs/lr, Bayesian-optimized on a val slice carved from the training split) instead of using the defaults. Teacher-free — the labels are already computed, so candidates cost only student fits.

• propose_budget (int) – Total candidate recipes tried when propose="auto".

• synthesize (int) – When example inputs are scarce, sample this many synthetic inputs from a generative model fit to the real training inputs (record inputs only) and have the teacher label them. Labels are always real (teacher-produced); the calibration slice and the OOD gate stay real-inputs-only, so the conformal guarantee and the p(x) floor reflect the true distribution.

• prelabeled (tuple[Sequence[Any], Sequence[Any]] | None) – Already-teacher-labeled (inputs, labels) pairs — typically load_harvested("harvested.jsonl") from a serving deployment — folded into the TRAINING split (and the OOD gate: they are real traffic) but never into calibration, which stays a fresh split of inputs. This is the re-solve half of the serving loop.

• device (Any) – A DeviceSpec makes this “give me this capability on that device”: the student is found by distill_for_edge() — a structure x precision x recipe search under the device’s hard byte/ops/torch-free budget (reusing the already-computed labels; the teacher is not re-called) — and the result’s footprint, Pareto front, and design ledger land on Solution.edge. If nothing fits the budget the Solution is demoted (everything routes to the teacher). Incompatible with propose="auto" (the device search subsumes it). A plain string (e.g. "cpu") keeps its old meaning: the torch training device.

• device_space (Any) – Optional EdgeSpace constraining the device search (families, size ranges, precisions); default spans the standard space.

• cost (Any) – Optional CostModel for realized-savings reporting.

• seed (int) – Split + fit determinism.

• **distill_kw (Any) – Student knobs forwarded to distillation (dim, hidden, epochs, lr, …). The default student is a torch MLP, so it requires the optional torch extra (pip install "mixle[torch]"); on a base install the check runs before any teacher calls are spent. student="generative" swaps the hashed-feature MLP for mixle’s generative student — per-class token models for text (mixle.task.generative_text) or the structure-learned joint for records (distill_structured_from_labels()): exact posteriors, no torch needed at inference, and a built-in log p(x).

• teacher_mode (str)

• **distill_kw

Returns:

A Solution – call it like the original function; report() / improve() / save().

Return type:

Solution