mixle.experimental.summary_tree module

E4: hierarchical summary tree + multi-scale objective – see notes/designs/E4.md for the full derivation (persistent frontier tree over evicted tokens, tree-path positional encoding, the predict-the-summary auxiliary loss, the stop-gradient horizon receipt). This module implements that note section-by-section; see the note’s “Implementation notes vs. this design” section for the small, honestly-documented places this module simplifies the note’s scheme for tractability.

What this is. E1’s SlidingWindowSpine keeps an exact but bounded KV window; anything evicted from that window is gone. E4 keeps it: every evicted token is folded, one at a time, into a bounded frontier of learned summaries via mixed-radix carry propagation (the fast-multipole-method structure – near field exact, far field via a bounded number of increasingly coarse representatives – applied to token history). Tree depth grows only as log_fanout(evicted_count), so the far-field attention set stays bounded regardless of how much history has streamed through.

Positional encoding. RoPE’s q . k dependence on i - j is well-conditioned only when i - j is a small, well-scaled number (E1’s window); a far-field summary node represents a range of possibly billions of original positions, and there is no single j to rotate by. E4 replaces RoPE for the far field with (a) a content channel – a level embedding plus a sibling-slot embedding summed into the node’s summary before it’s used as an attention key – and (b) a relative bias channel – a learned scalar indexed by tree distance (lca_depth, see below), the ALiBi/T5-bias shape of mechanism but indexed by tree distance instead of linear offset. Near-field window tokens keep ordinary RoPE unchanged (E1’s _rope_angles/_apply_rope, reused verbatim).

Predict-the-summary auxiliary loss. Every node, the moment it’s finalized, is scored against the exact additive token-id histogram of the leaves it covers via one shared linear head (d_model -> vocab) and cross-entropy against the normalized histogram. This is direct supervision independent of whether any future query ever attends to that node – for a node many levels up the tree that a training run may never query again, this is its compressor’s only gradient.

Stop-gradient horizon (receipted). A frontier node moves from live to archived once H further evicted tokens have arrived after it finalized; .summary is .detach()-ed at that moment and never un-detaches. Archived frontier nodes stay forward-visible but stop receiving gradient. See SummaryTreeSpine.detach() and mixle/tests/summary_tree_test.py for the exact-accounting receipt.

class TreeNode(summary, histogram, path, level, g, finalized_step, finalized_index_within_level, represented_leaves=0, detached=False, detached_at_finalized_count=None, detached_at_evicted_count=None)[source]

Bases: object

One finalized node of the persistent summary tree (E4.md’s TreeNode).

summary carries one entry per layer (each (batch, d_model)) since every layer’s tree is built from that layer’s own (k, v) via that layer’s own qkv projection, but the bookkeeping fields below (histogram/path/level/g) are identical across layers – they describe which evicted tokens this node covers, not any layer-specific content – so they’re stored once.

Parameters:

  • summary (list[Any])

  • histogram (Any)

  • path (tuple[int, ...])

  • level (int)

  • g (int)

  • finalized_step (int)

  • finalized_index_within_level (int)

  • represented_leaves (int)

  • detached (bool)

  • detached_at_finalized_count (int | None)

  • detached_at_evicted_count (int | None)

class SummaryTreeState(window, cached_ids, pending_leaf, pending, live, archived, level_finalized_count, evicted_count=0, batch_size=1, receipt=None)[source]

Bases: object

ContextMechanism carried state: E1’s exact near field plus the persistent far-field tree.

Parameters:
digits_of(n, base)[source]

Base-base digits of n, least-significant first. digits_of(0, base) == (0,).

Both arguments are validated because this is a public export and neither degenerate case has an answer to return (MXR-080-1862): base=1 never terminates – n // 1 == n forever, so the loop appends zeros until memory runs out – base=0 divides by zero, and a negative n skips the loop entirely and silently reports the empty digit sequence, which is not a representation of anything.

Parameters:
Return type:

tuple[int, …]

lca_depth(query_pos, node_level, node_g, fanout, *, max_climb=64)[source]

Tree distance between a query at absolute position query_pos and a node covering the contiguous leaf range [g * fanout**level, (g + 1) * fanout**level) (g = the node’s own 0-based sequential index among nodes finalized at its level – see E4.md’s carry-propagation section). Returns how many levels above node_level the query’s ancestor chain must climb before it lands in the same subtree as the node – see notes/designs/E4.md’s “Implementation notes” section for why this integer recurrence is the exact algebraic equivalent of “matching leading digits of the two paths” without materializing query_pos’s (potentially ~1e9-long) digit expansion. Pure function of (query_pos, node_level, node_g, fanout) – no dependence on how the stream was chunked, which is what makes it stable under re-chunking (Acceptance §3).

Every argument is validated because this is a public export and each degenerate case returns a number that means nothing (MXR-080-1866). fanout=1 makes fanout**m == 1 at every level, so q_anc never moves and the loop runs to max_climb and reports that climb as a tree distance – a base-1 tree has no positional expansion to share a prefix in (the same reason digits_of() refuses it). A negative query_pos floor-divides toward minus infinity, and a negative node_level makes fanout**m a float, so the “integer recurrence” returns a float distance. A negative node_g names no node. None of these are states the tree can reach; each previously returned a plausible-looking integer instead of refusing.

Parameters:

  • query_pos (int)

  • node_level (int)

  • node_g (int)

  • fanout (int)

  • max_climb (int)

Return type:

int