mixle.task.economics module

Cost comparisons for local distillation, cascades, and teacher-only serving.

Every routing choice has a cost model. This module combines a measured escalation rate (mixle.task.calibrate.CalibratedTaskModel.escalation_rate(), the empirical p_escalate) with unit costs:

  • frontier-only – pay c_frontier for every request, forever.

  • local-only – distill once (n_label teacher calls + training), then pay c_local per request.

  • cascade – run the low-cost local model first, escalate only the ambiguous fraction: per request c_local + p_escalate * c_frontier.

p_escalate is a measured selection rate, and alpha buys only MARGINAL prediction-set coverage under exchangeability – it is not a bound on the error of the locally-answered slice, because answering locally conditions on the set being a singleton. Treat answered-slice risk as a measurement, or certify it separately with mixle.task.calibrate.CalibratedTaskModel.calibrate_selective(); both statements fail silently under distribution shift.

break_even_volume() is the request count at which a distilled route recovers its one-time setup cost. recommend_route() picks the lowest-cost route at a given volume, optionally constrained by a maximum tolerated escalation rate, and reports the savings.

class CostModel(c_frontier, c_local=0.0, c_label=0.0, train_cost=0.0)[source]

Bases: object

Unit costs in any consistent currency.

Parameters:
setup_cost(n_label)[source]

Return the one-time label and training cost for a local model.

Parameters:

n_label (int)

Return type:

float

cascade_cost_per_request(cost, p_escalate)[source]

Expected per-request cost of the cascade: always run local, escalate the p_escalate fraction.

Parameters:

  • cost (CostModel)

  • p_escalate (float)

Return type:

float

break_even_volume(cost, n_label, *, p_escalate=0.0)[source]

Requests after which a distilled route undercuts frontier-only (inf if it never does).

Setup is amortized against the per-request saving c_frontier - per_request(route). With p_escalate=0 this is the local-only break-even; pass the model’s escalation rate for the cascade break-even.

Parameters:

  • cost (CostModel)

  • n_label (int)

  • p_escalate (float)

Return type:

float

class RoutePlan(route, volume, per_request, total, savings_vs_frontier, p_escalate, break_even, options)[source]

Bases: object

Costed route comparison for a fixed request volume.

Parameters:
recommend_route(cost, *, volume, n_label, p_escalate, max_escalation=None, local_only_certified=False)[source]

Pick the lowest-cost route over volume requests.

local_only is never inferred from an escalation preference: forcing ambiguous requests through a local model changes serving semantics. It is offered only when local_only_certified=True and the measured escalation probability is exactly zero.

Parameters:

  • cost (CostModel)

  • volume (int)

  • n_label (int)

  • p_escalate (float)

  • max_escalation (float | None)

  • local_only_certified (bool)

Return type:

RoutePlan

select_alpha_for_cost(model, cal_texts, cal_labels, probe_texts, cost, *, certification_texts, certification_labels, volume, n_label, alphas=(0.01, 0.05, 0.1, 0.15, 0.2, 0.3))[source]

Select alpha from a CostModel target.

The sweep connects recommend_route() to the calibration step so threshold selection reflects both model behavior and the caller’s cost assumptions.

model is anything with the CalibratedTaskModel shape: a mutable alpha attribute, calibrate(texts, labels), and escalation_rate(texts). For each candidate in alphas, this recalibrates model on policy-selection calibration data and measures its realized escalation rate on probe_texts, then scores that escalation rate with recommend_route() over volume requests. The winner is the alpha whose recommended route is lowest-cost overall; only after selection is complete is model calibrated at that alpha on the independent certification_texts/certification_labels. Thus the coverage-bearing threshold never participates in policy selection. Returns (best_alpha, best_plan, plan_by_alpha) so the full sweep remains auditable; the plans are policy-selection cost estimates, while the model’s final threshold is the independent certification result.

The in-place recalibration is all-or-nothing (MXR-080-1895): if any step raises, model’s attributes are restored to what they were on entry before the exception propagates, so a failed sweep can never leave a serving model carrying a threshold fitted on policy-selection rows. See _attribute_snapshot() for what “restored” does and does not cover.

Parameters:
Return type:

tuple[float, RoutePlan, dict[float, RoutePlan]]