mixle.system.fault module

Degradation policy with named failure modes for receipts.

Subsystem failures should not silently serve degraded answers as ordinary successes. with_fallback() runs a primary path and, on exception, runs a named fallback. The result records whether the primary path succeeded or which degradation mode produced the fallback value. abstain_on_timeout() and route_past() apply the same receipt discipline to timeout abstention and multi-tier routing.

The named modes are teacher_down (fall back to captured or store-only reasoning; see mixle.system.core.System.answer()), store_down (reason without accumulated knowledge; see mixle.system.core.System.ingest()), oracle_timeout (abstain or escalate rather than guess), and model_error (route past the failing tier to the next one). The timeout and routing helpers are reusable fault-boundary primitives.

A fault boundary is for recoverable failures – a provider outage, a saturated endpoint, an unreachable store. It is not a general exception swallower. NON_RECOVERABLE_FAULTS names the classes that must never be degraded into an ordinary-looking answer: a programming error (TypeError, AttributeError, …) means the fallback answer is built on a defect nobody will see, and an authorization/policy failure (PermissionError) routed past a fail-closed boundary is a security hole, not a degraded mode. Pass an explicit recoverable= allowlist to narrow a particular route further.

NON_RECOVERABLE_FAULTS: tuple[type[BaseException], ...] = (<class 'TypeError'>, <class 'AttributeError'>, <class 'NameError'>, <class 'SyntaxError'>, <class 'AssertionError'>, <class 'RecursionError'>, <class 'SystemError'>, <class 'MemoryError'>, <class 'PermissionError'>)

Failure classes a fault boundary must never convert into a degraded answer. Programmer errors (a defect in this process, not a failing dependency), integrity/resource failures the process cannot honestly continue past, and authorization/policy denials – which a route-past would turn into “the next tier answered it”, silently bypassing a fail-closed boundary.

DEGRADATION_MODES: frozenset[str] = frozenset({'model_error', 'oracle_timeout', 'store_down', 'teacher_down'})

The degradation modes this module’s own helpers emit, for readers of a receipt. Informational, not a closed set: DegradedResult accepts any named mode (see its docstring for why).

class DegradedResult(value, degraded, mode=None, reason=None, attempts=())[source]

Bases: object

The outcome of a fault-boundary call: a value, whether it came from a fallback, and, if so, why.

attempts carries the structured causal evidence for a multi-tier route: one (tier_name, exception_repr) pair per tier that was tried and failed, so a receipt records what actually went wrong rather than only that something did.

degraded, mode and reason are one fact in three fields, so they are checked against each other at construction: a degraded result names its mode and carries a reason, and a non-degraded one carries neither. Without that, DegradedResult("v", False, "teacher_down", "failure") serialized a receipt asserting that a successful answer was produced by a degradation, and DegradedResult("v", True) one asserting a degradation nobody can attribute – both of which downstream audit code reads as ordinary records.

mode is deliberately NOT restricted to DEGRADATION_MODES. This is a reusable fault boundary and callers name their own routes (mixle.inference.production.serving degrades under model_unavailable/batch_unavailable); a closed set would turn an unlisted caller’s recoverable failure into a hard constructor error on the fallback path, which is precisely the outage this class exists to survive. The name must simply be a real one.

Parameters:
to_receipt_fields()[source]

Return degraded_mode and degraded_reason receipt fields.

Return type:

dict[str, Any]

with_fallback(fn, fallback, *, mode, recoverable=None)[source]

Run fn(); on a recoverable exception, run fallback(exc) instead and flag it under mode.

recoverable is an explicit allowlist of failure classes this route may degrade past; omit it for the default policy (anything except NON_RECOVERABLE_FAULTS). A non-recoverable failure propagates unchanged: an internal TypeError handed back as a fallback answer hides a defect, and a PermissionError degraded into a successful-looking reply bypasses a fail-closed boundary rather than reporting that access was denied.

If fallback itself raises, that exception propagates. A fallback that cannot produce a value is a real failure, not a second implicit fallback.

mode is checked BEFORE fn runs: an unnamed mode makes the eventual receipt unattributable, and discovering that only on the fallback path would convert a survivable outage into a hard failure at exactly the wrong moment.

Parameters:
Return type:

DegradedResult

abstain_on_timeout(fn, *, timeout_error=TimeoutError)[source]

oracle_timeout mode: run fn(); if it raises timeout_error, abstain (value=None) rather than guess. Other exceptions propagate.

The abstention records its DegradedResult.attempts like every other degraded result this module produces (MXR-080-1902). It used to return attempts=(): an oracle_timeout abstention was the one degradation whose receipt carried no structured causal evidence at all, so audit code walking attempts – the field DegradedResult documents as “the structured causal evidence” – saw an empty tuple and could not distinguish a timed-out oracle from a route that never degraded, even though with_fallback() and route_past() both record theirs. reason alone is a human-readable string, not the same thing.

Parameters:
Return type:

DegradedResult

route_past(tiers, *, names=None, recoverable=None)[source]

model_error mode: try each tier in order; a tier that fails recoverably is skipped (not fatal) in favor of the next. The result is degraded unless the first tier answers cleanly. Raises the last tier’s exception if every tier fails.

recoverable is an explicit allowlist of failure classes this route may skip a tier for; omit it for the default policy (anything except NON_RECOVERABLE_FAULTS). A non-recoverable failure propagates immediately instead of being reclassified as model_error: an internal TypeError routed past becomes the next tier’s ordinary answer and the defect is never seen, and skipping a PermissionError turns a denial into a served result from a tier the caller was not authorized to reach.

Every skipped tier is recorded on DegradedResult.attempts as a (tier_name, exception_repr) pair – the causal evidence for why the answer came from where it did, not merely that it was degraded.

Parameters:
Return type:

DegradedResult