mixle.doe.amplify module

Capture an amplified teacher with collapse monitoring.

The “amplified teacher” here is optimize_under_oracle()’s search itself: round 1 spends an n_init + n_iter oracle-call budget searching, and must be verified stronger than a budget-matched random baseline of the same size before anything else happens. A search’s best-of-N compared against a single extra random draw is a multiple-comparisons trap, not a fair test: for N+1 values drawn i.i.d. from the same distribution, exchangeability alone gives P(max of the first N > the last one) = N / (N+1), so passage becomes increasingly automatic as the budget N grows even when the search adds no real capability, and the one extra draw was never counted against the stated budget either. Matching the baseline’s budget to the search’s and testing “is round 1’s best score bigger than the baseline’s best score by more than a fair reshuffling of the same 2N scores would typically produce” with a one-sided permutation test (rather than a bare, un-tested “is my number bigger” check on two single points) is what keeps the gate a calibrated statistical claim instead of a counter that trends toward “always passes”. If the search does not clear the gate at the significance level, there is nothing to capture, and this function stops with an explicit reason rather than distilling nothing.

The student captured from round 1 is a low-cost regression surrogate of the oracle’s score landscape, fit only from round 1’s oracle-verified (x, score) pairs. It never grades a candidate itself; it only proposes where round 2 should spend its matched oracle-call budget – against a FRESH candidate pool that excludes round 1’s own points, so round 2 cannot pad its score by re-verifying already-known winners (see amplify_and_capture). student(x) is a plain candidate -> predicted_score callable, the same shape any other teacher/task-model in this codebase is called with. Every accepted score in round 2 still comes from the oracle. No student or LLM self-grade enters DesignRun.history.

mixle.task.collapse.collapse_monitor() checks the two-round trajectory for regression or mode collapse; it is reused here rather than reimplemented.

class StudentTeacher(coef, degree)[source]

Bases: object

Captured regression surrogate used to rank follow-up design candidates.

The object is callable as student(x) -> predicted_score so it can be used like other teacher/task-model callables, but its role here is only proposal ranking. Accepted scores still come from the verifiable oracle.

Parameters:
fit_student(run, *, degree=2)[source]

Fit StudentTeacher from run’s GENUINE (non-abstained) oracle-verified history.

Fits against run.genuine_history, not run.history: an abstained candidate’s placeholder -inf score (OracleResult.abstained – e.g. a timeout) is not a real observation of the oracle’s objective, and least-squares regression has no tolerance for a non-finite target at all – unlike the permutation test in amplify_and_capture, where a lone -inf only nudges a p-value, here a SINGLE abstained candidate anywhere in run.history drives every coefficient in coef to nan (confirmed: a clean fit against 8 well-posed genuine points, then adding one abstained candidate to the same history turns the entire coefficient vector to nan), silently turning the captured student that guides round 2’s candidate ranking into a useless, uniformly-nan predictor with no visible error at the call site.

Parameters:

  • run (DesignRun)

  • degree (int)

Return type:

StudentTeacher

class AmplificationRound(run, best_score, xs)[source]

Bases: object

One oracle-verified amplification round and its best observed score.

Parameters:
class AmplificationSeeds(search, baseline, permutation, round2_pool)[source]

Bases: object

Concrete independent random streams used by an amplification receipt.

Parameters:

  • search (int)

  • baseline (int)

  • permutation (int)

  • round2_pool (int)

class AmplifyReport(round1, round2, baseline, baseline_p_value, significance, beats_baseline, round2_beats_round1, incumbent_best_score, collapse, student, stopped_early, reason, seeds, round1_effective_n, baseline_effective_n, round2_effective_n)[source]

Bases: object

Receipt for a two-round amplify-and-capture run.

round1 is the initial oracle-driven search. baseline is a random draw matched to round 1’s exact oracle-call budget (not a single draw – see amplify_and_capture), so every oracle call either side spent is accounted for in its own receipted AmplificationRound. baseline_p_value is the one-sided permutation-test p-value for “round 1’s best score exceeds the baseline’s best score by more than chance”, tested against the prespecified significance level to produce beats_baseline. round2 is present only when round 1 clears that gate, and a captured student is used to propose the matched-budget follow-up batch from a FRESH candidate pool (round 1’s own points are never re-offered to round 2, so round2_beats_round1 measures generalization to unseen candidates, not re-verification of already-known ones). incumbent_best_score is bookkeeping only – the best score retained across whichever rounds ran – and, unlike round 1’s era of this report, plays no part in computing round2_beats_round1 or collapse. collapse records the trajectory-level collapse verdict when both rounds run.

Parameters:

  • round1 (AmplificationRound)

  • round2 (AmplificationRound | None)

  • baseline (AmplificationRound)

  • baseline_p_value (float | None)

  • significance (float)

  • beats_baseline (bool)

  • round2_beats_round1 (bool)

  • incumbent_best_score (float | None)

  • collapse (CollapseVerdict | None)

  • student (StudentTeacher | None)

  • stopped_early (bool)

  • reason (str | None)

  • seeds (AmplificationSeeds)

  • round1_effective_n (int)

  • baseline_effective_n (int)

  • round2_effective_n (int | None)

amplify_and_capture(oracle, bounds, *, n_init=5, n_iter=10, candidate_pool_size=200, degree=2, significance=0.05, seed=None)[source]

Round 1: search the oracle for a budget of n_init + n_iter calls (the amplified teacher).

The search must beat a budget-matched random baseline – also n_init + n_iter oracle-verified draws, not one – on a one-sided permutation test of “round 1’s best score exceeds the baseline’s best score by more than a fair reshuffling of the pooled 2 * budget scores would typically produce”, at the prespecified significance level. A single best-of-N-vs-one-draw comparison is a multiple-comparisons trap (see the module docstring): matching the baseline’s budget to the search’s, and testing the two best-of-budget scores against the null distribution obtained by repeatedly reshuffling which of the 2 * budget oracle-verified scores “belong” to round 1 versus the baseline (rather than a bare, un-tested “is my number bigger” check), is what makes beats_baseline a calibrated claim instead of a counter that trends toward “always passes” as the budget grows. If the search does not clear the gate, this returns the explicit stopped_early=True result with nothing distilled.

Otherwise: fit StudentTeacher from round 1’s history; round 2 uses the student to rank a large, FRESH candidate pool (never round 1’s own points – see the round 2 comment below) and spends the same oracle-call budget verifying only the top-ranked fresh candidates – student-guided, not blind, but every accepted score is still oracle-verified. Runs mixle.task.collapse.collapse_monitor() over the two rounds.

Parameters:
Return type:

AmplifyReport