mixle.doe.oracle module

Verifiable oracle boundary for de novo optimization.

Given a design goal for which there is no data yet but there IS a way to check a candidate (a simulator, an executable test, held-out truth, an assay), optimize_under_oracle() proposes candidates, verifies them against the oracle, and keeps a full receipted history of what was tried and why – the design-test-learn loop, made accountable, rather than “synthesize data and train” with no account of what verified it.

The one hard precondition, checked before anything else: there must be a verifiable oracle. VerifiableOracle rejects a “self-graded by a model” tier at CONSTRUCTION – that is the banned reward this boundary exists to forbid – and optimize_under_oracle() refuses to run at all without one (oracle=None -> the explicit “no verifiable objective; cannot optimize” refusal, never a fabricated candidate).

This is a first, deliberately narrow slice: continuous/low-dimensional candidate spaces only, using the GP Bayesian-optimization loop already in mixle.doe (BayesianOptimizer) as the proposal model, validated here against a low-cost closed-form oracle before any domain oracle exists. Not in this slice: structured/discrete candidate spaces (a protein sequence, a program), amortizing the oracle into a calibrated surrogate, the shared expected-information- gain acquisition, and full receipt objects – each is a separate surface and is left explicit rather than half-built here.

A timeout is an abstention, not a data point: it means the oracle did not answer, not that the true score at that candidate is catastrophic. OracleResult carries an explicit abstained flag for exactly this, and optimize_under_oracle() never feeds an abstained result to the proposal model’s fit as if it were ground truth, even though it is still kept in DesignRun.history for the receipted record. VerifiableOracle also validates every result crossing its boundary (return type, finite score unless abstained, finite nonnegative cost, receipt shape) rather than letting a malformed score_fn return silently corrupt the run. Timeout workers are bounded by a circuit breaker, cooperative cancellation is requested, and eventual late outcomes are linked back to their run by immutable call IDs so side-effecting cost is never silently lost.

class OracleResult(score, receipt=<factory>, cost=1.0, abstained=False, passed=True, valid=True, call_id=None, call_started=True)[source]

Bases: object

One candidate’s verification outcome: its score, a receipt of how it was scored, and its cost.

Construction validates its own shape (MXR-080-0189): a real, finite score unless abstained=True, a finite nonnegative cost, and a dict receipt. This is the boundary a score_fn’s raw, untrusted return value crosses (see VerifiableOracle._validate_returned), so a wrong type, a NaN/inf that leaked in from some computation, or a negative “cost” is caught immediately here rather than silently reaching a GP fit or a cost report.

abstained marks a result that is NOT a genuine observation of the oracle’s true objective at the scored candidate – for example, VerifiableOracle’s timeout abstention. An abstained result’s score is a placeholder (-inf by convention, never a claim about the true objective) kept only so the receipted history stays complete; consumers such as optimize_under_oracle() gate on abstained and must never feed one to a fit as if it were ground truth.

Parameters:
class LateOracleResult(call_id, candidate, ok, result, error)[source]

Bases: object

A timed-out oracle call’s outcome, captured after the fact.

VerifiableOracle cannot forcibly kill the worker thread a timeout abandons – Python has no API to do so (see _call_with_timeout). If a score_fn that ignores (or does not support) the cooperative cancel_event eventually finishes anyway, its outcome is appended to VerifiableOracle.late_results instead of being silently discarded, so a side-effecting oracle’s eventual cost/outcome is still visible somewhere, even though the caller had already moved on and received an abstention in its place.

Parameters:

  • call_id (str)

  • candidate (Any)

  • ok (bool)

  • result (OracleResult | None)

  • error (str | None)

class VerifiableOracle(name, tier, score_fn, fidelity=None, timeout=None, max_outstanding_timeouts=4)[source]

Bases: object

A callable candidate -> OracleResult that declares its verifiability tier and fidelity.

score_fn does the actual verification (wrap a simulator, an executable check, a held-out ground-truth lookup, or a real measurement pipeline; mixle.task.toolcall’s ToolCaller is the same “external check as a callable” shape for tool calls). Construction raises if tier is not one of VERIFIABILITY_TIERS – “self-graded by a model” is not a valid tier and is rejected here, not silently accepted and discovered later.

score_fn may optionally accept a cancel_event: threading.Event keyword argument (detected once, at construction, via introspection – see _accepts_keyword). A side-effecting score_fn that wraps something cancellable (a subprocess, an HTTP request) should periodically check cancel_event.is_set() and tear its underlying call down promptly once set. This is a best-effort, cooperative cancellation boundary, not a forced kill – Python cannot forcibly terminate a thread. Calls that ignore cancellation remain quarantined, count against max_outstanding_timeouts, and eventually open a circuit that prevents additional worker creation. Their outcomes are captured in late_results when they finish. See _call_with_timeout.

Parameters:

  • name (str)

  • tier (str)

  • score_fn (Callable[[Any], OracleResult])

  • fidelity (str | None)

  • timeout (float | None)

  • max_outstanding_timeouts (int)

property late_results: tuple[LateOracleResult, ...]

Immutable snapshot of late outcomes accumulated so far.

property outstanding_timeouts: int

Number of timed-out workers that have not cooperatively stopped or completed.

class DesignCandidate(x, result, phase=None)[source]

Bases: object

One proposed-and-verified candidate: the point tried and what the oracle said about it.

Parameters:

  • x (ndarray)

  • result (OracleResult)

  • phase (str | None)

class DesignRun(oracle_name, oracle_tier, oracle_fidelity, _oracle=None)[source]

Bases: object

The full receipted history of a design loop: every candidate tried, and the oracle’s identity.

Parameters:

  • oracle_name (str)

  • oracle_tier (str)

  • oracle_fidelity (str | None)

  • _oracle (VerifiableOracle | None)

property history: tuple[DesignCandidate, ...]

Immutable chronological snapshot of the append-only candidate ledger.

append(candidate)[source]

Append one immutable candidate record; existing ledger entries can never be replaced.

Parameters:

candidate (DesignCandidate)

Return type:

None

property oracle_calls: int

Return the number of candidates attempted against the oracle (abstentions included).

property candidate_attempts: int

All proposed candidates, including circuit-breaker abstentions that started no external call.

property genuine_history: tuple[DesignCandidate, ...]

Return history filtered to genuine (non-abstained) observations.

An abstention (see OracleResult.abstained – e.g. a timeout) is not a real observation of the oracle’s objective, so it is excluded here; use history directly for the full, unfiltered receipted record, abstentions included.

property best: DesignCandidate

Return the highest-scoring GENUINE candidate in the run history.

Abstained candidates (see OracleResult.abstained) never win: an abstention is not a real observation of the oracle’s objective, and surfacing one as “the best candidate found” would report a fabricated result as if it were verified. Raises if every candidate abstained (no genuine observation exists to report), distinct from the empty-history case.

scores()[source]

Return the run’s oracle scores in chronological order, abstentions included (not hidden).

Return type:

ndarray

report()[source]

Named receipt of the run: which oracle, at what tier/fidelity, the best candidate found.

provisional_total_cost is the cost known when calls returned. Late outcomes linked by immutable call IDs are reconciled into settled_late_cost and settled_total_cost. total_cost is only populated once no timed-out call remains outstanding or unresolved.

Return type:

dict[str, Any]

optimize_under_oracle(oracle, bounds, *, n_init=5, n_iter=15, seed=None, **bo_kwargs)[source]

Run a propose-verify-refit design loop under a fixed oracle budget.

The loop proposes candidates, verifies each one with oracle, keeps the receipted history, refits the proposal model on every GENUINE observation, and repeats under an n_init + n_iter budget of oracle calls.

oracle=None raises immediately with the explicit refusal (“no verifiable objective; cannot optimize”) – the hard precondition checked before any candidate is proposed. Continuous/low-dimensional bounds only (see module docstring); the proposal model is BayesianOptimizer, maximizing the oracle’s score.

An abstained result (OracleResult.abstained – e.g. a timeout; see VerifiableOracle) is never told to the proposal model as if it were a real observation (MXR-080-0189): a timeout means the oracle did not answer, not that the true score at that candidate is -inf, and training a fit on a fabricated -inf “observation” can corrupt it. The abstained candidate is still appended to run.history – the receipted record keeps every attempt, genuine or not – it is simply excluded from what the optimizer learns from. BayesianOptimizer.ask is explicitly designed to tolerate this (its initial-design dispensing is gated on points asked, not points told), so skipping tell for an abstention does not desynchronize or corrupt the proposal model’s state.

If the entire initial phase abstains, however, the model has no observation at all to fit an acquisition step on, and ask raises rather than guessing. The adaptive phase is therefore not entered in that case: the run stops and is returned with its receipts intact, so an all-abstention run still yields its audit report (status="no_verified_result") instead of dying inside the proposal model (MXR-080-1488).

Parameters:
Return type:

DesignRun