mixle.experimental.long_context_eval module

E7: the long-context referee – one evaluation suite every Track-E mechanism (E1’s baseline and every E2-E6 challenger) is measured against on the same terms (see mixle/experimental/README.md’s graduation rule: “beats the E1 baseline on the E7 evaluation suite at matched FLOPs” + misfit receipts).

evaluate(mechanism, ...) drives any ContextMechanism through four kinds of controlled-dependency-distance probes at every requested range:

  • needle – a (key, value) pair planted once near the start; the key recurs distance tokens later and the mechanism must recall the associated value (classic needle-in-a-haystack fact retrieval).

  • copy – a purely positional dependency: the token distance steps back must be reproduced, no key/value indirection (isolates raw positional recall from associative recall).

  • multi-hop – hops independent anchor values scattered across [0, distance) must all be retained and combined (sum mod vocab) to answer a single probe at the end – a dependency that cannot be satisfied by remembering only the most recent anchor.

  • multi-scale perplexity – a fixed, learnable order-1 Markov rule (a random token permutation) is trained and measured at each range, to see whether streaming quality degrades with total length, independent of any single controlled dependency.

Every probe trains mechanism briefly (via train_tbptt()) on fresh random instances of its suite, then measures held-out accuracy: the protocol (ContextMechanism) only returns a scalar mean loss per step, not logits, so exact argmax accuracy is unavailable by design – “solved” is instead a chance-normalized loss threshold (probe loss below half the uniform-guess loss 0.5 * ln(vocab)), which is a real, documented proxy rather than a silently-approximate one.

Calibrated forgetting curves (“does it know what it forgot?”). The mechanism’s OWN per-probe loss is its only self-reported signal (the protocol exposes nothing else). evaluate() overlays that signal against the needle accuracy curve and reports self_knowledge_correlation – the correlation between “how much it forgot” (1 - accuracy) and “how surprised it says it was” (its own probe loss) across ranges. A mechanism that is well-calibrated about its own forgetting scores near +1; a mechanism that is confidently wrong scores near 0.

Matched-FLOPs / matched-state-bytes protocols. evaluate() reports, per range, the FLOPs spent (6 * n_params * n_tokens, the same Kaplan/Hoffmann approximation mixle.ppl.scaling_laws uses) and, once, the carried-state byte footprint at the largest tested range against the caller-supplied state_budget_bytes. Two mechanisms compared with comparison_table() on the SAME ranges and state_budget_bytes are, by construction, being compared at matched FLOPs and matched state bytes – that comparison is the caller’s job (pass a {name: evaluate(...)} mapping); this module only makes the numbers honest and side-by-side.

Length curriculum as a bandit. length_curriculum() (also run internally by evaluate()) uses mixle.task.bandit.ThompsonBernoulli (reused, not reimplemented) with one arm per length bucket in ranges. Reward is the fraction of the maximum possible loss reduction achieved by one training step on that bucket (clip(improvement / chance_loss, 0, 1), so it lives in [0, 1] as ThompsonBernoulli requires), divided by that bucket’s FLOP cost relative to the cheapest bucket – literally “loss improvement per FLOP”, normalized to be dimensionless and bounded. Ultra-long buckets are additionally rationed by a shared FLOP ledger seeded once from compute_budget_flops (split evenly across buckets): an arm whose next pull would exceed its remaining ledger is masked out of selection for the rest of the run, so the policy cannot simply spend the whole compute box on the longest bucket even if its posterior looks best.

Honest scale note (see also this module’s test file): at distance=1e6 a single real training run here is computationally enormous – evaluate’s ranges default matches the roadmap card literally ((1e3, 1e4, 1e5, 1e6)) and the function accepts genuinely large ranges from any caller. The test suite that exercises this module does NOT use those literal values; it uses small stand-in ranges (documented in mixle/tests/long_context_eval_test.py) so the suite runs in a few seconds while exercising the exact same code path a caller would use at card scale.

DEFAULT_VOCAB = 17

Default alphabet size for synthetic suites when mechanism doesn’t expose its own .vocab.

needle_suite(rng, *, distance, vocab)[source]

(key, value) planted at positions 0/1; the key recurs at position distance and the target there is the value – associative recall at a controlled range. Requires distance >= 2.

Parameters:
Return type:

tuple[Any, Any]

copy_suite(rng, *, distance, vocab)[source]

Pure positional recall: the target at position distance is the token that appeared at position 0, with no key/value cue – isolates raw positional memory from associative lookup.

Parameters:
Return type:

tuple[Any, Any]

multi_hop_suite(rng, *, distance, vocab, hops=3)[source]

hops anchor values scattered across [0, distance); the probe at distance must equal their sum mod vocab - 1. Answering correctly requires retaining EVERY anchor, not just the most recent one – a dependency a single-needle test can’t distinguish from short-range recall.

Parameters:
Return type:

tuple[Any, Any]

length_curriculum(mechanism, opt, ranges, *, vocab, n_rounds, compute_budget_flops, seed, perm)[source]

A ThompsonBernoulli arm per length bucket in ranges.

Reward = fraction of the maximum possible loss reduction achieved by one training step on that bucket, divided by the bucket’s FLOP cost relative to the cheapest bucket (“loss improvement per FLOP”, normalized to [0, 1]). A shared FLOP ledger (compute_budget_flops split evenly across buckets) additionally masks out any arm whose next pull would exceed its remaining share – ultra-long buckets are rationed by construction, since each of their pulls costs proportionally more.

Parameters:
Return type:

dict[str, Any]

evaluate(mechanism, *, ranges=(1e3, 1e4, 1e5, 1e6), state_budget_bytes, seed, hops=3, n_train_steps=6, n_eval_trials=8, perplexity_steps=6, curriculum_rounds=12, compute_budget_flops=None)[source]

Run the full E7 referee suite against mechanism end-to-end. See the module docstring for what each piece measures and honestly claims. Requires a torch-trainable mechanism (.parameters() exposed, as every Track-E mechanism in mixle.experimental.context_spine is) – trains it IN PLACE via TBPTT, so pass a freshly-initialized instance for a clean baseline measurement.

compute_budget_flops defaults to 20x the FLOP cost of one full pass over the largest range, generous enough that the length-curriculum bandit (length_curriculum()) gets a meaningful number of rounds without the caller having to reason about FLOPs by hand.

Parameters:

  • mechanism (ContextMechanism)

  • ranges (tuple[float, ...])

  • state_budget_bytes (float)

  • seed (int)

  • hops (int)

  • n_train_steps (int)

  • n_eval_trials (int)

  • perplexity_steps (int)

  • curriculum_rounds (int)

  • compute_budget_flops (float | None)

Return type:

dict[str, Any]

comparison_table(results)[source]

Render evaluate() output as a plain-text table. Accepts either a single evaluate() return value, or a {name: evaluate(...)} mapping for a matched-FLOPs / matched-state-bytes side-by-side comparison (e.g. an E2-E6 challenger against the E1 baseline, both evaluated with the same ranges and state_budget_bytes).

Parameters:

results (dict[str, Any])

Return type:

str