mixle.experimental.long_context_eval module

E7: the long-context referee – one evaluation suite every Track-E mechanism (E1’s baseline and every E2-E6 challenger) is measured against on the same terms. The graduation rule requires matched compute; this module now rejects a comparison that has neither equal measured training FLOPs nor a shared enforced training-compute ceiling.

evaluate(mechanism, ...) drives any ContextMechanism through four kinds of controlled-dependency-distance probes at every requested range:

  • needle – a (key, value) pair planted once near the start; the key recurs distance tokens later and the mechanism must recall the associated value (classic needle-in-a-haystack fact retrieval).

  • copy – a purely positional dependency: the token distance steps back must be reproduced, no key/value indirection (isolates raw positional recall from associative recall).

  • multi-hop – hops independent anchor values scattered across [0, distance) must all be retained and combined (sum mod vocab) to answer a single probe at the end – a dependency that cannot be satisfied by remembering only the most recent anchor.

  • multi-scale perplexity – a fixed, learnable order-1 Markov rule (a random token permutation) is trained and measured at each range, to see whether streaming quality degrades with total length, independent of any single controlled dependency.

Every controlled-dependency probe trains a fresh clone of the same initial mechanism only on the final dependency position, then measures a paired held-out loss_threshold_success_rate. The ContextMechanism protocol returns scalar loss rather than logits, so this module does not call that threshold statistic classification accuracy.

Calibrated forgetting curves (“does it know what it forgot?”). The mechanism’s OWN per-probe loss is its only self-reported signal (the protocol exposes nothing else). evaluate() overlays that signal against the needle threshold-success curve and reports self_knowledge_correlation – the correlation between 1 - loss_threshold_success_rate and probe loss across ranges.

Compute/state protocols. evaluate() accounts for training and evaluation tokens separately and can enforce a shared suite_training_budget_flops across the isolated cells. comparison_table() rejects multi-mechanism output unless seed, data protocol, ranges, vocabulary, compute contract, and state budget match; it also rejects any result that exceeded its state budget. Merely using the same token counts is not described as matched compute.

Length curriculum as a bandit. length_curriculum() (also run internally by evaluate()) uses mixle.task.bandit.ThompsonBernoulli (reused, not reimplemented) with one arm per length bucket in ranges. Reward is the fraction of the maximum possible loss reduction achieved by one training step on that bucket (clip(improvement / chance_loss, 0, 1), so it lives in [0, 1] as ThompsonBernoulli requires), divided by that bucket’s FLOP cost relative to the cheapest bucket – literally “loss improvement per FLOP”, normalized to be dimensionless and bounded. Ultra-long buckets are additionally rationed by a shared FLOP ledger seeded once from compute_budget_flops (split evenly across buckets): an arm whose next pull would exceed its remaining ledger is masked out of selection for the rest of the run, so the policy cannot simply spend the whole compute box on the longest bucket even if its posterior looks best.

Honest scale note (see also this module’s test file): at distance=1e6 a single real training run here is computationally enormous – evaluate’s ranges default matches the roadmap card literally ((1e3, 1e4, 1e5, 1e6)) and the function accepts genuinely large ranges from any caller. The test suite that exercises this module does NOT use those literal values; it uses small stand-in ranges (documented in mixle/tests/long_context_eval_test.py) so the suite runs in a few seconds while exercising the exact same code path a caller would use at card scale.

DEFAULT_VOCAB = 17

Default alphabet size for synthetic suites when mechanism doesn’t expose its own .vocab.

needle_suite(rng, *, distance, vocab)[source]

(key, value) planted at positions 0/1; the key recurs at position distance and the target there is the value – associative recall at a controlled range. Requires distance >= 2.

Parameters:
Return type:

tuple[Any, Any]

copy_suite(rng, *, distance, vocab)[source]

Pure positional recall: the target at position distance is the token that appeared at position 0, with no key/value cue – isolates raw positional memory from associative lookup.

Parameters:
Return type:

tuple[Any, Any]

multi_hop_suite(rng, *, distance, vocab, hops=3)[source]

hops anchor values scattered across [0, distance); the probe at distance must equal their sum mod vocab - 1. Answering correctly requires retaining EVERY anchor, not just the most recent one – a dependency a single-needle test can’t distinguish from short-range recall.

Parameters:
Return type:

tuple[Any, Any]

length_curriculum(mechanism, opt, ranges, *, vocab, n_rounds, compute_budget_flops, seed, perm)[source]

A ThompsonBernoulli arm per length bucket in ranges.

Reward = fraction of the maximum possible loss reduction achieved by one training step on that bucket, divided by the bucket’s FLOP cost relative to the cheapest bucket (“loss improvement per FLOP”, normalized to [0, 1]). A shared FLOP ledger (compute_budget_flops split evenly across buckets) additionally masks out any arm whose next pull would exceed its remaining share – ultra-long buckets are rationed by construction, since each of their pulls costs proportionally more.

Parameters:
Return type:

dict[str, Any]

evaluate(mechanism, *, ranges=(1_000, 10_000, 100_000, 1_000_000), state_budget_bytes, seed, hops=3, n_train_steps=6, n_eval_trials=8, perplexity_steps=6, curriculum_rounds=12, compute_budget_flops=None, suite_training_budget_flops=None)[source]

Run the full E7 referee suite against mechanism end-to-end. See the module docstring for what each piece measures and honestly claims. Requires a torch-trainable mechanism (.parameters() exposed, as every Track-E mechanism in mixle.experimental.context_spine is). The supplied mechanism is not trained or mutated: every range/suite, state probe, and curriculum receives an independent deep copy of its initial weights.

suite_training_budget_flops is a total ceiling split evenly across every range/suite cell. When it is supplied, each cell uses as many of the requested training steps as fit within its share. The separate compute_budget_flops controls only the curriculum and defaults to 20x one pass over the largest range.

Parameters:

  • mechanism (ContextMechanism)

  • ranges (tuple[int, ...])

  • state_budget_bytes (float)

  • seed (int)

  • hops (int)

  • n_train_steps (int)

  • n_eval_trials (int)

  • perplexity_steps (int)

  • curriculum_rounds (int)

  • compute_budget_flops (float | None)

  • suite_training_budget_flops (float | None)

Return type:

dict[str, Any]

comparison_table(results)[source]

Render evaluate() output as a plain-text table. Accepts either a single evaluate() return value, or a {name: evaluate(...)} mapping. Multi-mechanism comparisons fail closed unless their paired-data, compute, and state contracts match.

Parameters:

results (dict[str, Any])

Return type:

str