mixle.task.pilot_ladder module

F7: the pilot ladder – capacity_ladder-style GO/NO-GO rung staging for the frontier trainer.

Scope, read this first. The roadmap card this module implements (F7, “(L, calendar-dominant)”) names REAL rungs: 1B/8k-context/8-GPU, then 8B/128k/256-GPU, then 8B/10M-context/1000-GPU, then a headline run whose size/context/MoE-vs-dense choice is made by F5’s scaling-law fit under a fixed compute box. None of that hardware exists in this environment, and pretending otherwise would be dishonest. What this module builds instead – and what is actually real, tested, and runs today – is the ORCHESTRATION MACHINERY: a Rung ladder walker that runs each rung’s training, collects the roadmap’s named artifacts (MFU, loss curve, forgetting curve, decision-journal entry), and GATES progression to the next rung on a real GO/NO-GO check against that rung’s measured training-health receipts. It is exercised here at a tiny simulated scale (a handful of rungs of increasing but still laptop-sized model/data size) standing in for the real 1B -> 8B -> headline progression – the gate/journal/artifact-collection logic is identical to what a real run would drive, only the scale is fake.

Which of the roadmap’s staged sub-pieces this module can actually reach, from this worktree’s base (release/0.7.0) and which it cannot:

  • F4 (training-health + MFU receipts, PR #147) – merged into this worktree’s base. Wired in for real via mixle.utils.parallel.training_health.TrainingHealthMonitor; every rung’s MFU/loss-curve/ restart-continuity artifacts come from that exact machinery, not a re-implementation.

  • F1 (TP/PP/CP atop FSDP2, PR #171) – merged into this worktree’s base. Its own test suite already exercises the parallelism mechanics; this orchestrator does not re-simulate TP/PP/CP sharding (that would just be re-testing F1), it notes F1 as an assumed-healthy dependency of the rung-i shakeout.

  • H2 (MoE block + dense->MoE upcycling, PR #167) – merged into this worktree’s base. Wired in for real: rung ii calls mixle.models.moe.upcycle_dense_to_moe() on the rung’s trained dense block and records the measured MoE-vs-dense output-gap receipt the roadmap calls “the H2 MoE-vs-dense decision”.

  • F9 (muP width transfer, PR #155) – merged into this worktree’s base. Wired in for real: when a rung opts in, its learning rate is transferred (not re-tuned) from a stated base width via mixle.models.mup.transfer_lr(), and the transfer receipt is recorded.

  • F2 (fault-tolerant checkpointing, roadmap card F2) – lives on origin/fault-tolerant-checkpointing, which is NOT an ancestor of this worktree’s base and diverges from it (see that branch’s own diff stat). Pulling its module across would mean vendoring code this PR did not review line-by-line just to claim a checkbox; instead this module documents F2 as unreachable-here and raises NotImplementedError if a caller explicitly opts a rung into fault-injection (Rung.exercise_fault_tolerance=True) rather than silently skipping it.

  • F5 (scaling-law fits, roadmap card F5) – same story, lives on origin/scaling-law-fits, not reachable from this base. Rungs iii/iv nominally depend on F5 to choose the next rung’s size/context; here that choice is a documented manual stand-in, and Rung.exercise_scaling_law_fit=True raises NotImplementedError rather than fabricating a fit.

  • E7 (referee evaluation suite) / E8 (a later long-context item) – neither exists yet anywhere in this repository (see mixle/experimental/README.md: E7 is explicitly “later items on the same roadmap track and don’t exist yet”). Same treatment: documented, NotImplementedError on explicit opt-in.

Every “not reachable” piece above is a named, spelled-out reason attached to the rung’s artifacts (RungArtifacts.skipped_pieces), never a silent no-op.

PILOT_LADDER_UNAVAILABLE_PIECES: dict[str, str] = {'E7': "the E7 referee evaluation suite does not exist yet anywhere in this repository (see mixle/experimental/README.md's graduation rule); this rung ran without an E7 bake-off.", 'E8': "E8 is a later long-context roadmap item that has not been built yet; this rung ran without it. (F1's TP/PP/CP, PR #171, already covers context parallelism as a separate roadmap item and is listed under PILOT_LADDER_ASSUMED_HEALTHY_PIECES below -- it is not the same thing as E8.)", 'F2': "fault-tolerant checkpointing (roadmap F2) lives on origin/fault-tolerant-checkpointing, which is not an ancestor of this worktree's base (release/0.7.0) and diverges from it; this rung ran without fault injection.", 'F5': "scaling-law fits (roadmap F5) live on origin/scaling-law-fits, not reachable from this worktree's base; this rung did not fit a scaling law and used a manually-chosen stand-in configuration instead of one F5 would have chosen."}

roadmap sub-pieces this module cannot reach from this worktree’s base, and exactly why – see the module docstring. Keyed by the name a Rung’s decision_pieces may cite.

PILOT_LADDER_ASSUMED_HEALTHY_PIECES: dict[str, str] = {'F1': "TP/PP/CP atop FSDP2 (roadmap F1, PR #171) is merged into this worktree's base but is not re-simulated here; this rung assumes it healthy and relies on F1's own test suite for that."}

roadmap sub-pieces that ARE merged into this worktree’s base but are treated as an assumed-healthy dependency rather than re-simulated by this orchestrator – their own test suites already exercise them.

class PilotLadderResult(outcomes, halted_at, journal)[source]

Bases: object

The whole ladder’s outcome: every attempted rung, where (if anywhere) it halted, and the journal.

Parameters:

  • outcomes (list[RungOutcome])

  • halted_at (str | None)

  • journal (EpistemicJournal)

class Rung(name, real_target, decision_pieces, vocab=64, d_model=16, n_layer=2, n_head=2, block=8, n_workers=1, steps=40, switch_step=None, batch_size=8, lr=0.01, seed=0, max_final_loss=3.0, max_forgetting_gap=1.5, require_continuity=True, exercise_mup_transfer=False, mup_base_width=None, exercise_moe_decision=False, moe_experts=4, moe_max_relative_diff=1.0, exercise_fault_tolerance=False, exercise_eval_suite=False, exercise_context_parallel=False, exercise_scaling_law_fit=False)[source]

Bases: object

One pilot-ladder rung: a tiny simulated stand-in for a REAL roadmap rung’s size/context/GPU count.

real_target documents the real rung this stands in for (e.g. "1B params / 8k context / 8 GPUs") purely for the record – nothing here can measure that scale, so the actual training below runs at vocab/d_model/n_layer/n_head/block sizes chosen to finish in seconds on a laptop. n_workers is a documented stand-in for the real rung’s GPU count; this module does not spawn n_workers real processes (see the F1 note in the module docstring) but records it as part of the rung’s identity for the decision journal.

Parameters:

  • name (str)

  • real_target (str)

  • decision_pieces (tuple[str, ...])

  • vocab (int)

  • d_model (int)

  • n_layer (int)

  • n_head (int)

  • block (int)

  • n_workers (int)

  • steps (int)

  • switch_step (int | None)

  • batch_size (int)

  • lr (float)

  • seed (int)

  • max_final_loss (float)

  • max_forgetting_gap (float | None)

  • require_continuity (bool)

  • exercise_mup_transfer (bool)

  • mup_base_width (int | None)

  • exercise_moe_decision (bool)

  • moe_experts (int)

  • moe_max_relative_diff (float)

  • exercise_fault_tolerance (bool)

  • exercise_eval_suite (bool)

  • exercise_context_parallel (bool)

  • exercise_scaling_law_fit (bool)

class RungArtifacts(rung, health_report, loss_curve, forgetting_curve, forgetting_gap, final_loss, mfu_mean, skipped_pieces=<factory>, exercised_receipts=<factory>)[source]

Bases: object

The roadmap’s per-rung artifacts: MFU, loss curve, forgetting curve, plus this pilot’s bookkeeping.

Parameters:
class RungOutcome(artifacts, passed, reason, decision_record)[source]

Bases: object

One rung’s full outcome: its artifacts, the GO/NO-GO verdict, why, and its journal entry.

Parameters:

  • artifacts (RungArtifacts)

  • passed (bool)

  • reason (str)

  • decision_record (DecisionRecord)

run_pilot_ladder(rungs, *, peak_flops_per_sec=1.0e12)[source]

Run each of rungs in order, gating progression on a real GO/NO-GO check of its own artifacts.

For every rung: train the rung’s tiny simulated model, collect MFU / loss-curve / forgetting-curve artifacts (reusing mixle.utils.parallel.training_health’s exact machinery), exercise whichever of F9/H2 the rung opted into for real, append one Bayesian decision-journal entry (mixle.epistemic.journal.EpistemicJournal) recording the belief update and the actual GO/NO-GO action taken, and – this is the gate – stop the ladder the first time a rung’s measured receipts fail its own stated criteria. peak_flops_per_sec is an arbitrary stand-in “hardware peak” (no real hardware backs any MFU number this produces); it only needs to be a fixed positive constant for MFU to be comparable across this ladder’s own rungs, which is all the GO/NO-GO gate uses it for.

Parameters:
Return type:

PilotLadderResult