mixle.task.solve module

solve – point mixle at a function and get back a deployable model that does its job.

The closed loop the task spine was building toward, in one call. teacher is whatever currently performs the task – a rule cascade, a legacy scoring routine, an expensive API. solve uses it to create the dataset (the teacher labels the example inputs), proposes and trains a student matched to the input shape (text / record / structured), calibrates an honest answer-or-escalate rule on held-out data (conformal, so “confident” has a coverage guarantee), verifies the student against the teacher, and returns a Solution: a drop-in callable that answers locally when it is sure and falls back to the teacher when it is not. Wrong answers are bounded by the calibration; unfamiliar inputs route to the code that already works.

The loop then compounds: every escalated request is a teacher-labeled example exactly where the student is weak. Solution.improve() re-distills with those harvested labels and promotes the new student only if it verifies at least as well (anti-regression) – so the deployed thing gets cheaper the longer it runs.

def route(ticket): … # 400 lines of if/elif that must not break sol = solve(route, tickets) # dataset <- route(t) for t in tickets; train; calibrate sol(ticket) # answers locally or calls route() – safe to deploy today sol.report() # agreement, escalation rate, realized cost sol.improve() # fold escalations back in; promote only if better

solve is deterministic given seed. Only the student fit needs torch; the teacher stays opaque.

load_harvested(path)[source]

Read the harvested.jsonl a serving endpoint accumulates into (inputs, answers).

Two line formats are understood: {"input": ..., "label": ...} (the mixle-mlops /v1/tasks/{name}/feedback classification format — labels are str-coerced) and {"input": ..., "answer": ...} (the /v1/solutions/{name}/feedback format for every solve shape — the answer keeps its shape: a number for regression, a list of labels for multilabel, a dict for structured). Input JSON lists coerce back to tuples — the record shape the solve trained on — so the pairs feed straight into solve*(..., prelabeled=load_harvested(p)).

Parameters:

path (str)

Return type:

tuple[list, list]

class Solution(cascade, teacher, kind, train_inputs, train_labels, cal_inputs, cal_labels, holdout_agreement, escalation_rate, promoted, target_agreement, distill_kw=<factory>, ood=None, seed=0, synthesized=0, gate_inputs=<factory>, edge=None)[source]

Bases: object

A deployed task: a calibrated student in front of the teacher, plus the loop to improve it.

Call it like the original function. promoted says whether the student passed verification – when False the callable simply runs the teacher (an honest failure, never a silently bad model).

Parameters:
cascade: Cascade
teacher: Callable[[...], Any]
kind: str
train_inputs: list
train_labels: list
cal_inputs: list
cal_labels: list
holdout_agreement: float
escalation_rate: float
promoted: bool
target_agreement: float | None
distill_kw: dict
ood: float | None = None
seed: int = 0
synthesized: int = 0
gate_inputs: list
edge: Any = None
report()[source]

What you would want on a dashboard: verification, live escalation, realized cost.

Return type:

dict

improve()[source]

Re-distill with the harvested (escalated) labels; promote only if it verifies at least as well.

Returns True when a better student was promoted. The calibration slice is never trained on, so the conformal guarantee and the agreement comparison stay honest across rounds.

Return type:

bool

health(recent_inputs=None, *, p_threshold=0.01)[source]

Is the calibration still holding on live traffic? The guarantee-watcher.

The conformal escalate-or-answer rate was calibrated under exchangeability; when the input distribution drifts, the live escalation rate moves away from the verified baseline — which is exactly the observable to alarm on. This compares the live rate against the baseline with an exact binomial test, and (when recent_inputs are supplied and an OOD gate exists) also checks the gate’s out-of-distribution hit-rate against its design quantile.

Returns a dict with drifted (bool), the rates, and p-values. Honest caveat: a drift alarm means “the world changed, escalations (and cost) will rise, re-solve soon” — the SYSTEM stays correct throughout because unsure inputs still go to the teacher.

Parameters:

  • recent_inputs (Any)

  • p_threshold (float)

Return type:

dict[str, Any]

save(path)[source]

Persist the calibrated student as a load-anywhere artifact, with its verification record.

Every deployed artifact carries how it was verified — held-out agreement with the teacher, the escalation rate, the conformal alpha, and how much of its training data was synthetic — so “is this model trustworthy” is answerable from the artifact alone.

Parameters:

path (str)

Return type:

str

deploy(name, root='./mixle_data/registry')[source]

Save into the serving layout — {root}/tasks/{name} — the directory the mixle-mlops /v1/tasks routes serve from. Returns the artifact path.

Parameters:
Return type:

str

classmethod load(path, teacher, *, cost=None, device='cpu')[source]

Reconstitute a serving Solution from a saved artifact — the deploy path for a fresh process.

The loaded Solution answers locally / escalates to teacher and harvests labels exactly like the original. It carries no training or calibration data, so improve() raises — collect the harvested pairs and re-solve (real + harvested inputs) to train the next round.

Parameters:
Return type:

Solution

solve(teacher, inputs, *, alpha=0.1, target_agreement=None, holdout=0.25, kind=None, ood=0.02, propose=None, propose_budget=8, synthesize=0, prelabeled=None, device=None, device_space=None, cost=None, seed=0, **distill_kw)[source]

Replace teacher (the code currently doing the job) with a calibrated, self-improving model.

Parameters:

  • teacher (Callable[[...], Any]) – The callable performing the task today (per-item or batched). It labels the dataset and remains the fallback for inputs the student is not sure about.

  • inputs (Sequence[Any]) – Example inputs (text, or tuple/dict records) covering the task. The teacher labels them.

  • alpha (float) – Escalation honesty – answer locally only when a single label is conformally covered at >= 1 - alpha; otherwise fall back to the teacher.

  • target_agreement (float | None) – Optional gate. If the student’s held-out agreement with the teacher misses it, the returned Solution routes everything to the teacher (promoted=False).

  • holdout (float) – Fraction reserved for calibration + verification (never trained on).

  • kind (str | None) – Force the student path, 'text' or 'record'; default sniffs the first input.

  • ood (float | None) – Fit a p(x) gate over the training inputs and escalate inputs whose log p(x) falls below this quantile floor — so a wildly novel input escalates even when the softmax looks confident. On by default (0.02); None disables.

  • propose (str | None) – "auto" searches the student recipe (dim/hidden/epochs/lr, Bayesian-optimized on a val slice carved from the training split) instead of using the defaults. Teacher-free — the labels are already computed, so candidates cost only student fits.

  • propose_budget (int) – Total candidate recipes tried when propose="auto".

  • synthesize (int) – When example inputs are scarce, sample this many synthetic inputs from a generative model fit to the real training inputs (record inputs only) and have the teacher label them. Labels are always real (teacher-produced); the calibration slice and the OOD gate stay real-inputs-only, so the conformal guarantee and the p(x) floor reflect the true distribution.

  • prelabeled (tuple[Sequence[Any], Sequence[Any]] | None) – Already-teacher-labeled (inputs, labels) pairs — typically load_harvested("harvested.jsonl") from a serving deployment — folded into the TRAINING split (and the OOD gate: they are real traffic) but never into calibration, which stays a fresh split of inputs. This is the re-solve half of the serving loop.

  • device (Any) – A DeviceSpec makes this “give me this capability on that device”: the student is found by distill_for_edge() — a structure x precision x recipe search under the device’s hard byte/ops/torch-free budget (reusing the already-computed labels; the teacher is not re-called) — and the result’s footprint, Pareto front, and design ledger land on Solution.edge. If nothing fits the budget the Solution is demoted (everything routes to the teacher — an honest failure). Incompatible with propose="auto" (the device search subsumes it). A plain string (e.g. "cpu") keeps its old meaning: the torch training device.

  • device_space (Any) – Optional EdgeSpace constraining the device search (families, size ranges, precisions); default spans the standard space.

  • cost (Any) – Optional CostModel for realized-savings reporting.

  • seed (int) – Split + fit determinism.

  • **distill_kw (Any) – Student knobs forwarded to distillation (dim, hidden, epochs, lr, …). student="generative" swaps the hashed-feature MLP for mixle’s generative student — per-class token models for text (mixle.task.generative_text) or the structure-learned joint for records (distill_structured_from_labels()): exact posteriors, no torch needed at inference, and a built-in log p(x).

Returns:

A Solution – call it like the original function; report() / improve() / save().

Return type:

Solution