mixle.experimental.typed_runtime.benchmark module

Time-to-target traces and negative-control failure receipts.

class BenchmarkPoint(step, objective, elapsed_seconds, operation_count=0, model_evaluations=0, bytes_read=0, bytes_written=0, collective_bytes=0, peak_memory_bytes=0, maximum_staleness_steps=0, accepted_updates=0, rejected_updates=0, work_counter_semantics=CounterSemantics.CUMULATIVE, peak_memory_semantics=CounterSemantics.HIGH_WATER_MARK, staleness_semantics=CounterSemantics.HIGH_WATER_MARK)[source]

Bases: object

One cumulative quality/work observation from an actual run.

Parameters:

  • step (int)

  • objective (float)

  • elapsed_seconds (float)

  • operation_count (int)

  • model_evaluations (int)

  • bytes_read (int)

  • bytes_written (int)

  • collective_bytes (int)

  • peak_memory_bytes (int)

  • maximum_staleness_steps (int)

  • accepted_updates (int)

  • rejected_updates (int)

  • work_counter_semantics (CounterSemantics)

  • peak_memory_semantics (CounterSemantics)

  • staleness_semantics (CounterSemantics)

as_dict()[source]

Return a JSON-compatible benchmark observation.

Return type:

dict[str, Any]

class FailureKind(*values)[source]

Bases: StrEnum

Failure families used by Stage-0 negative controls.

class FailureLedger(receipts=())[source]

Bases: object

Append-only in-memory ledger for benchmark failure oracles.

Append-only in fact, not only in the docstring (MXR-080-1905). receipts was a public list on a dataclass, so a ledger holding one failed oracle answered all_oracles_passed: False until ledger.receipts.clear(), after which it answered True – the negative control it was recording had been erased through the very attribute that reports it, and the duplicate-case check in record() was bypassable the same way. receipts is a detached tuple now.

Parameters:

receipts (Sequence[FailureReceipt])

property receipts: tuple[FailureReceipt, ...]

Every recorded failure-oracle receipt, in the order recorded.

record(receipt)[source]

Record one uniquely identified case.

Parameters:

receipt (FailureReceipt)

Return type:

None

property failed_oracles: tuple[FailureReceipt, ...]

Expected failures missed or clean controls falsely flagged.

as_dict()[source]

Return a JSON-compatible ledger.

Return type:

dict[str, Any]

class FailureReceipt(benchmark_id, case_id, kind, oracle, expected_failure, detected, observed, details=<factory>)[source]

Bases: object

Outcome of one expected or naturally occurring failure case.

Parameters:
property oracle_passed: bool

Whether the detector behaved as expected for this case.

as_dict()[source]

Return a JSON-compatible negative-control receipt.

Return type:

dict[str, Any]

class ObjectiveTarget(name, direction, threshold, tolerance=0.0)[source]

Bases: object

Explicit quality threshold for a time-to-target benchmark.

Parameters:

  • name (str)

  • direction (TargetDirection)

  • threshold (float)

  • tolerance (float)

reached(value)[source]

Whether value reaches this target within tolerance.

Parameters:

value (float)

Return type:

bool

as_dict()[source]

Return a JSON-compatible target declaration.

Return type:

dict[str, Any]

class TargetDirection(*values)[source]

Bases: StrEnum

Whether reaching a target requires increasing or decreasing a metric.

class TimeToTargetTrace(benchmark_id, strategy, target, points=())[source]

Bases: object

Monotone cumulative work trace for one strategy and one target.

Not a dataclass any more (MXR-080-1905): points was a public list, so the chronology and cumulative-counter contract that record() enforces could be walked straight past – trace.points.append(BenchmarkPoint(0, 0.95, 0.0)) after a step-1 point produced a trace whose steps ran 1, 0 and whose elapsed times ran 1.0, 0.0, and achieved then reported True from that out-of-order point. points is a read-only view now and record() is the only way in. The constructor signature is unchanged.

Parameters:

  • benchmark_id (str)

  • strategy (str)

  • target (ObjectiveTarget)

  • points (Sequence[BenchmarkPoint])

property points: tuple[BenchmarkPoint, ...]

The recorded observations, oldest first, as a detached tuple.

record(point)[source]

Append a cumulative point after checking chronology and counters.

Parameters:

point (BenchmarkPoint)

Return type:

None

property first_target_point: BenchmarkPoint | None

First observed point that reaches the declared quality target.

property achieved: bool

Whether this trace reaches its declared target.

as_dict()[source]

Return the complete trace without collapsing time and operations.

Return type:

dict[str, Any]