mixle.system package

Local application facade: answer/ingest/improve a Mixle system, with its cost ledger, degradation policy, held-out scorecard, meta-improvement allocator, and model registry.

Formerly six cross-importing top-level modules (mixle.system, mixle.spend, mixle.fault, mixle.scorecard, mixle.meta, mixle.registry) – one cohesive package now:

  • mixle.system.core – System, SystemConfig, Query: the answer/ingest/improve facade.

  • mixle.system.spend – Spend: the budget ledger System is metered against.

  • mixle.system.fault – with_fallback(), abstain_on_timeout(), route_past(), DegradedResult: named degradation modes instead of silently-successful fallbacks.

  • mixle.system.scorecard – SystemScorecard, evaluate(), detect_regression(): measuring a System against a held-out question set.

  • mixle.system.meta – ImprovementOption, improve_by_regret(): spend an improvement budget on the highest realized gain-per-dollar option, stopping on regression.

  • mixle.system.registry – Registry, RegistryEntry: the local directory-backed catalog of fitted task models these workflows read from and write into.

Every name below is importable directly from mixle.system (from mixle.system import System, Spend, Registry); the individual submodules remain the place to look for the source and docstrings. The old flat top-level names (mixle.spend, mixle.fault, mixle.scorecard, mixle.meta, mixle.registry) still work via a deprecation shim; new code should import from here.

Resolution is lazy (PEP 562, mirroring mixle itself): importing one submodule – e.g. mixle.system.fault, which callers such as mixle.doe.oracle use for its zero-dependency degradation helpers – never pulls in the other five. core.py alone reaches mixle.task; an eager __init__ that imported every submodule up front would force that whole chain onto any caller of the leaf modules, including a cycle back through mixle.task.propose.

class DegradedResult(value, degraded, mode=None, reason=None, attempts=())[source]

Bases: object

The outcome of a fault-boundary call: a value, whether it came from a fallback, and, if so, why.

attempts carries the structured causal evidence for a multi-tier route: one (tier_name, exception_repr) pair per tier that was tried and failed, so a receipt records what actually went wrong rather than only that something did.

degraded, mode and reason are one fact in three fields, so they are checked against each other at construction: a degraded result names its mode and carries a reason, and a non-degraded one carries neither. Without that, DegradedResult("v", False, "teacher_down", "failure") serialized a receipt asserting that a successful answer was produced by a degradation, and DegradedResult("v", True) one asserting a degradation nobody can attribute – both of which downstream audit code reads as ordinary records.

mode is deliberately NOT restricted to DEGRADATION_MODES. This is a reusable fault boundary and callers name their own routes (mixle.inference.production.serving degrades under model_unavailable/batch_unavailable); a closed set would turn an unlisted caller’s recoverable failure into a hard constructor error on the fallback path, which is precisely the outage this class exists to survive. The name must simply be a real one.

Parameters:
to_receipt_fields()[source]

Return degraded_mode and degraded_reason receipt fields.

Return type:

dict[str, Any]

class ImprovementOption(name, cost, run, estimated_regret)[source]

Bases: object

Candidate improvement action with estimated cost and recoverable quality gain.

cost must be finite and nonnegative and estimated_regret finite. Neither was checked, so under budget=0 an option costing -5 ran and reported spent=-5 (running work created budget), and a NaN-cost option ran too, because the over-budget comparison spent + cost > budget is false for NaN – the ceiling failed open – and left spent=NaN behind.

Parameters:
property regret_per_dollar: float

Estimated recoverable gain per unit cost.

class MetaImprovementReport(order=<factory>, skipped=<factory>, scorecard_before=None, scorecard_after=None, realized_gain_per_dollar=<factory>, spent=0.0, stopped_on_regression=None)[source]

Bases: object

Execution report for a budgeted meta-improvement run.

Parameters:

  • order (list[str])

  • skipped (list[str])

  • scorecard_before (SystemScorecard | None)

  • scorecard_after (SystemScorecard | None)

  • realized_gain_per_dollar (dict[str, float])

  • spent (float)

  • stopped_on_regression (RegressionReport | None)

class Query(text, task='', fingerprint=None, expected_output=None, scope='local')[source]

Bases: object

The typed problem contract for System.answer().

task and expected_output align with the mixle-knowledge ContextPacket contract’s task and expected_output_schema fields (see from_knowledge_dict()). scope is a Query-level routing boundary and is not inferred from the packet.

Parameters:
classmethod from_knowledge_dict(packet, *, scope='local')[source]

Build a Query from a mixle-knowledge-shaped ContextPacket dict.

text comes from payload["rendered"]. task and expected_output map from the packet’s task and expected_output_schema. scope is supplied by the caller.

Parameters:
Return type:

Query

class Registry(dir)[source]

Bases: object

A dir-backed catalog of registered models: register writes an artifact + index entry; find_for/tier_stack query it. Re-opening the same dir in a fresh process sees every entry.

Parameters:

dir (str)

register(model, *, capabilities, fingerprint=None, profile=None, cost=0.0, entry_id=None)[source]

Save model’s artifact under dir and add its index entry; return the entry.

model is a fitted TaskModel, a CalibratedTaskModel, or a mixle_pde field posterior (a “field_posterior” kind, IC-2) – the three artifact-saveable model kinds. capabilities names what this model answers (matched by find_for()); fingerprint is typically task_fingerprint()’s vector for the training data; profile is free-form (e.g. a capture_profile() dict); cost is the per-request cost used to order tier_stack(). An explicit entry_id that already exists (in the index or as an artifact directory) raises rather than duplicating the index row and silently overwriting the artifact; auto-generated ids scan past taken ones.

entry_id must be a single safe path component (_safe_entry_id()): an unvalidated value such as "../escaped" would otherwise write the artifact outside dir instead of inside it.

Every field of the index row is validated (by constructing the RegistryEntry) BEFORE the id is claimed or the artifact written, so a rejected argument leaves the registry byte-for-byte unchanged and the id free to retry – see the comment at that construction (MXR-080-1902).

Allocating an id and persisting the index is a read-modify-write over the on-disk index: two Registry instances opened on the same dir (or two threads/processes sharing one) each cache their own self._entries snapshot from construction time and otherwise never refresh it, so one instance’s register can silently overwrite another’s already-persisted index row when it writes back its own stale-plus-one view – even when the two calls pick different, non-colliding entry ids. Fixed by serializing id-allocation-through-index-write with an fcntl.flock on a lock file at the registry root, re-reading the on-disk index fresh under the lock rather than trusting the cached snapshot. An independent claim-file guard (O_CREAT | O_EXCL) additionally protects the artifact write itself, belt-and-suspenders alongside the lock, mirroring mixle.inference.production.registry.Registry.register()’s fix for the same class of bug.

Parameters:
Return type:

RegistryEntry

load(entry_id)[source]

Reload a registered model by entry_id (round-trips through the artifact on disk).

Always resolves through _get() against the registry’s own validated state – never a record a caller was handed by find_for() and may have altered.

Parameters:

entry_id (str)

Return type:

TaskModel | CalibratedTaskModel | Any

find_for(query, *, top_k=None)[source]

Entries matching query: a capability name (str, containment match) or a task fingerprint vector (array-like of floats, nearest-neighbor match). top_k caps how many are returned – every capability match by default, or the single nearest fingerprint match by default.

Returns defensive copies: the results are a view of the catalog, not handles onto it, so altering one cannot silently rewrite where a later load() reads from or which entries a later query matches (registration is the only way to change the catalog).

An entry whose fingerprint does not share the query’s dimension is skipped individually rather than aborting the whole search: one length-3 record used to make every two-dimensional query raise a broadcasting ValueError, hiding every healthy entry behind one incompatible one.

Parameters:
Return type:

list[RegistryEntry]

tier_stack(task, *, frontier, costs=None, names=None)[source]

Ascending-cost (name, model, cost) tiers for capability task, frontier appended last.

Matching entries are loaded (load()) and ordered by their effective cost – the costs override when given, else the registered per-entry cost. The result is exactly the shape Router takes as tiers=: each non-final tier exposes decide(x), the final tier is the callable frontier fallback. costs (one entry per matching solution in registered-cost order, plus one for frontier, mirroring from_solutions()) overrides the registered per-entry costs when given.

Parameters:
Return type:

list[tuple[str, Any, float]]

class RegistryEntry(entry_id, path, kind, capabilities=<factory>, fingerprint=None, profile=<factory>, cost=0.0)[source]

Bases: object

One catalog record: where the artifact lives, what it’s registered under, and how much it costs to run.

Frozen, and validated on construction. A durable catalog record is a persisted fact, not a scratch object: Registry.find_for() used to hand out the very objects stored in Registry._entries, so rebinding a lookup result’s path silently redirected the next Registry.load() and clearing its capabilities removed it from later matching – all without any registry mutation API, validation, persistence, or receipt. Lookups now return defensive copies (see copy()), and a rebinding attempt raises instead of quietly rewriting routing.

Parameters:
copy()[source]

A defensive copy: mutating its containers cannot reach the registry’s own state.

Return type:

RegistryEntry

to_dict()[source]

Serialize the registry entry into JSON-compatible fields.

Return type:

dict[str, Any]

classmethod from_dict(d)[source]

Create a registry entry from a JSON index record.

The index is untrusted persisted input, not a trusted in-process value: entry_id, kind, fingerprint and cost are validated here exactly as they are on Registry.register() (see Registry._read_index(), which additionally re-derives path from the registry root rather than trusting the serialized one).

Parameters:

d (dict[str, Any])

Return type:

RegistryEntry

class RegressionReport(regressed, reasons=<factory>, comparable=True)[source]

Bases: object

Whether current is worse than baseline on any tracked axis, and exactly why.

comparable is False when the two cards are not evidence about the same thing (different held-out set/judge, different case counts, an unidentified card). An incomparable pair is reported as regressed=True: a comparison that could not be made is not a passed comparison.

Parameters:
class Spend(frontier_calls=0, oracle_calls=0, wall_ms=0.0, dollars=0.0)[source]

Bases: object

A summable cost total. total_units() is the scalar figure budget= is checked against.

Every dimension is checked on construction, and therefore again after each __add__() and after deserialization, because a receipt is a ledger: work only ever consumes budget. Without these invariants Spend(frontier_calls=-5).total_units() returned -5, so doing work could hand budget back, and a NaN wall_ms serialized as an ordinary receipt field while making every hard-ceiling comparison against it false – failing open on the check meant to stop overspending.

Parameters:
total_units()[source]

The scalar cost a budget= integer is measured against.

Currently frontier_calls + oracle_calls – the two countable, per-call costs the existing routes measure budget in. wall_ms and dollars are carried and reported on every receipt but are not yet priced into the hard-ceiling check; extend this method when a concrete dollar cost model is introduced.

Return type:

float

to_dict()[source]

Serialize the spend ledger into primitive numeric fields.

Return type:

dict[str, float | int]

class System(config)[source]

Bases: object

Constructed from a SystemConfig; exposes answer/ingest/improve.

Parameters:

config (SystemConfig)

answer(query, *, budget=None, read_only=False)[source]

Thin shell: route straight to the teacher, wrap the reply in a minimal H-style receipt.

Checks the captured cache first (see improve()): an exact repeat of a query (same text, task, AND scope – two queries that merely share text but differ in task/scope are different questions and must not share a cache entry) already promoted by a prior improve() call is served free, no budget spent, captured=True.

budget is a hard ceiling (total_units): if it cannot afford even one frontier call, the request is refused – reply is None and the receipt names the exact shortfall – rather than silently answering over budget. A served answer’s cost is added to total_spend, which every receipt also carries as total_spend.

If the teacher call itself raises, this falls back to teacher_down degraded mode: answer from the store alone (a plain retrieval over config.store, scoped to the QUERY’s own scope – never config.scope) when one is configured and has anything relevant, flagging degraded_mode="teacher_down" on the receipt; if there is no store (or nothing relevant in it), the failure is reported explicitly (status="failed"), never masked as a normal answer.

read_only=True runs the identical routing/answering logic but takes a genuine snapshot: the call is guaranteed not to promote into _harvest (so a later improve() can never capture it) and not to accumulate into total_spend – no trace of the call is left for the system’s future behavior to depend on. This is what evaluate() uses so that scoring a system against a held-out question set can never itself teach the system the held-out answers: an evaluation call must never be able to change what a later call to this same method returns. Every other observable (the reply, the receipt fields a scorer reads, whether the call degraded) is identical to a normal call, since a snapshot must still faithfully measure the real answering path.

budget is validated as an exact, non-Boolean, nonnegative call count – the same _count() contract Spend holds its own dimensions to, since the ceiling is compared directly against total_units() (MXR-080-1902). int(budget) TRUNCATED instead: budget=1.9 silently became a one-call ceiling and budget=True became one too, and the receipt then reported the truncated number as if it were what the caller asked for. A negative budget is a caller error and raises, matching improve(), rather than being served back as an ordinary “refused, shortfall N” receipt that reads like a legitimately under-funded request.

Parameters:

  • query (Query)

  • budget (int | None)

  • read_only (bool)

Return type:

tuple[str | None, dict[str, Any]]

ingest(model_output, *, source)[source]

Turn a model output into stored knowledge.

Uses the belief store when it is importable; otherwise records a plain substrate item rather than requiring optional knowledge-substrate components.

Every route writes under SystemConfig.scope, and the effective scope is named on the report. The belief path used to call assimilate() without its scope argument, so it defaulted to "local" no matter how the system was configured – a system configured for "tenant-A" ingested two claims and got back two apparently successful BeliefItem``s whose scope, and whose stored substrate scope, were both ``"local". The import-fallback path did use config.scope, so which boundary a write landed in depended on whether an optional module happened to be importable.

If the store write raises before anything is committed, this falls back to store_down degraded mode: the model output is acknowledged but not accumulated, and the report is flagged with degraded_mode="store_down".

Claims are assimilated one at a time and there is no transaction, so a failure partway through leaves the earlier claims stored. That case is reported as status="partial_accumulation" with the exact committed items, not as degraded_no_accumulation: denying durable state invites a retry that duplicates the committed evidence, or reasoning from a receipt that says nothing was written when something was.

Parameters:
Return type:

dict[str, Any]

improve(budget)[source]

Promote harvested (query, reply) pairs from answer() into the captured cache, up to budget.

Promotion is costed at a flat _PROMOTE_COST_PER_ITEM (currently 1) unit per (query, reply) pair – the only cost model defined for this step, since promoting a harvested pair is a pure cache write, not an external call like the frontier/oracle spend answer() meters. budget is a hard ceiling on how many pairs THIS call may promote, the same hard-ceiling spirit as answer’s budget: a negative budget is a caller error and raises ValueError rather than being silently treated as zero or unlimited.

Candidates are promoted in harvest order (the order answer() first harvested them – harvested pairs carry no other recency or confidence signal to rank by, so this is the simplest available, documented priority). Whatever does not fit the budget is left in the harvest for a later, better-funded call rather than being discarded.

Reports that there is nothing to improve when nothing has been harvested yet. Otherwise this is the cold-start capture step: after this call, a repeat of a captured query is answered from the local cache (see answer()). The return value reports both the requested ceiling (budget) and the realized spend (realized_spend) so a caller can tell a fully-funded round from a partial one.

budget is an exact, non-Boolean, nonnegative promotion count (_count(), MXR-080-1902). int(budget) truncated first and only then rejected a negative, so improve(1.9) quietly promoted one pair against a budget the caller wrote as nearly two, improve(True) promoted one, and improve(-0.5) truncated to 0 and returned an ordinary “insufficient_budget” report instead of naming the error that the same call written as improve(-1) raises for.

Parameters:

budget (int)

Return type:

dict[str, Any]

class SystemConfig(teacher, registry_dir=None, store=None, default_budget=1, scope='local')[source]

Bases: object

Configuration required to run a System.

Secrets such as endpoints and keys are read from the environment by from_env(); they are not hardcoded in the config object.

Parameters:
classmethod from_env(*, store=None, registry_dir=None)[source]

Build a config whose teacher is an OpenAICompatLLM sourced entirely from env vars.

Reads MIXLE_TEACHER_BASE_URL (required), MIXLE_TEACHER_MODEL (required), and the optional MIXLE_TEACHER_API_KEY. Raises ValueError naming the missing variable rather than silently constructing an unusable teacher.

Parameters:

  • store (Any)

  • registry_dir (str | None)

Return type:

SystemConfig

class SystemScorecard(quality, realized_cost, grounded_fraction, n, question_set_id='')[source]

Bases: object

One evaluation of a System against a fixed held-out question set.

Every metric is validated on construction: a card carrying NaN/out-of-range numbers or a negative case count is not a weaker measurement, it is not a measurement at all, and detect_regression() silently treated such a card as “no regression” (fail-open on the exact gate meant to catch a worsening round). question_set_id binds the card to the held-out set and judge it was measured with (see question_set_identity()).

Parameters:
to_dict()[source]

Serialize the scorecard into primitive fields.

Return type:

dict[str, Any]

abstain_on_timeout(fn, *, timeout_error=TimeoutError)[source]

oracle_timeout mode: run fn(); if it raises timeout_error, abstain (value=None) rather than guess. Other exceptions propagate.

The abstention records its DegradedResult.attempts like every other degraded result this module produces (MXR-080-1902). It used to return attempts=(): an oracle_timeout abstention was the one degradation whose receipt carried no structured causal evidence at all, so audit code walking attempts – the field DegradedResult documents as “the structured causal evidence” – saw an empty tuple and could not distinguish a timed-out oracle from a route that never degraded, even though with_fallback() and route_past() both record theirs. reason alone is a human-readable string, not the same thing.

Parameters:
Return type:

DegradedResult

detect_regression(baseline, current, *, tolerance=1e-9)[source]

Compare two scorecards from the SAME held-out set across improve-rounds.

Never silently accepts a round that answers worse, less groundedly, or for more cost than the round before it – each tracked axis that got worse (beyond tolerance) is named in reasons.

The “same held-out set” precondition is enforced, not merely documented: both cards must carry the same non-empty question_set_identity() and the same case count. Comparing a card against evidence from a different (or empty) question set cannot show that a round did not regress, so such a pair comes back comparable=False, regressed=True rather than a clean bill of health. tolerance must itself be finite and nonnegative – a NaN tolerance makes every ordered comparison below false, which silently turned this gate into an unconditional “no regression”.

Parameters:

  • baseline (SystemScorecard)

  • current (SystemScorecard)

  • tolerance (float)

Return type:

RegressionReport

evaluate(system, question_set, *, scorer=None)[source]

Evaluate system over a fixed [(query, expected_answer), ...] set.

Every expected is validated up front: a blank reference cannot discriminate, and a question set containing one is rejected rather than scored (it used to make every non-None reply correct and report quality=1.0).

  • quality – fraction of questions scorer judges correct. The default judge is _default_scorer(), a deliberately weak lexical baseline (case-insensitive substring match) that cannot see negation or paraphrase; read its docstring before treating quality as authoritative, and pass a task-specific scorer where correctness actually matters.

  • grounded_fraction – fraction answered WITHOUT a degraded mode (teacher or captured, not a store-only fallback / refusal / failure): the fraction of answers you can actually trust came from a real answer path, not a fault-boundary guess.

  • realized_cost – total spend units (total_units()) across the set.

  • question_set_id – the identity of the held-out set and judge this card was measured with, so detect_regression() can refuse to compare cards from different evidence.

There is deliberately NO calibration field. One used to exist and was assigned quality verbatim: a deterministic system that emits no confidence at all scored a perfect “calibration”, and detect_regression() never compared the field anyway, so a drop from 1.0 to 0.0 was not a regression. answer() carries no per-answer confidence to calibrate against, so there is nothing here to compute; when it does, add a real calibration statistic to THIS function and a matching axis to detect_regression() in the same change, rather than re-adding a copied number that reads as evidence of something never measured.

This is a genuinely read-only pass: every call goes through answer() with read_only=True, so evaluating a held-out question_set can never itself teach system the held-out answers (no promotion into the harvest a later improve() could capture, no change to system.total_spend). Without that, a held-out set would stop being held-out the moment it was first evaluated: a later improve() could promote its answers into the captured cache, and re-evaluating the “held-out” set afterward would be trivially free and perfect. Never call system.answer without read_only=True from this function.

Parameters:
Return type:

SystemScorecard

improve_by_regret(system, question_set, options, *, budget)[source]

Run options by estimated gain per dollar and stop on measured regression.

budget must be finite and nonnegative, and option names must be unique – order, skipped and realized_gain_per_dollar are all keyed by name, so duplicates silently overwrite each other’s measured result.

Note what this does NOT do: opt.run() mutates the live system before any measurement, and a regression detected afterwards only stops further options – the regressing change stays applied. Callers who need rejection to restore the previous state must run options against a system they can themselves discard. report.stopped_on_regression names the round that regressed so that decision can be made.

Parameters:
Return type:

MetaImprovementReport

route_past(tiers, *, names=None, recoverable=None)[source]

model_error mode: try each tier in order; a tier that fails recoverably is skipped (not fatal) in favor of the next. The result is degraded unless the first tier answers cleanly. Raises the last tier’s exception if every tier fails.

recoverable is an explicit allowlist of failure classes this route may skip a tier for; omit it for the default policy (anything except NON_RECOVERABLE_FAULTS). A non-recoverable failure propagates immediately instead of being reclassified as model_error: an internal TypeError routed past becomes the next tier’s ordinary answer and the defect is never seen, and skipping a PermissionError turns a denial into a served result from a tier the caller was not authorized to reach.

Every skipped tier is recorded on DegradedResult.attempts as a (tier_name, exception_repr) pair – the causal evidence for why the answer came from where it did, not merely that it was degraded.

Parameters:
Return type:

DegradedResult

with_fallback(fn, fallback, *, mode, recoverable=None)[source]

Run fn(); on a recoverable exception, run fallback(exc) instead and flag it under mode.

recoverable is an explicit allowlist of failure classes this route may degrade past; omit it for the default policy (anything except NON_RECOVERABLE_FAULTS). A non-recoverable failure propagates unchanged: an internal TypeError handed back as a fallback answer hides a defect, and a PermissionError degraded into a successful-looking reply bypasses a fail-closed boundary rather than reporting that access was denied.

If fallback itself raises, that exception propagates. A fallback that cannot produce a value is a real failure, not a second implicit fallback.

mode is checked BEFORE fn runs: an unnamed mode makes the eventual receipt unattributable, and discovering that only on the fallback path would convert a survivable outage into a hard failure at exactly the wrong moment.

Parameters:
Return type:

DegradedResult

Submodules