mixle.data.stream_token_source module

A streaming token source: yield (context-window, next-token) micro-batches from an IN-MEMORY token-id array WITHOUT ever materializing the (N, block) window matrix – each batch’s windows are built on the fly and discarded.

This is the data half of the non-buffering streaming estimator. The standard encoder builds and buffers every (window -> next) observation (O(corpus x block) host RAM – the materialization wall); this keeps only the token array resident (O(corpus)) plus O(batch x block) ephemeral per batch. Two honest caveats: the token array must fit in memory (this is not an out-of-core reader), and shuffle=True materializes one full-length O(corpus) permutation for the epoch order (still O(corpus), but a real extra array; shuffle=False avoids it).

The same generator SHAPE extends to a true out-of-core corpus – reading windows from a memory-mapped / sharded token file, where a checkpoint would be just the cursor position – but that out-of-core / resumable-cursor version is not implemented here.

Token ids are validated once at call entry (finite, exact-integer, in a lossless int64 range) and stay int64 all the way through to the yielded batches – context windows are never downcast to float32 internally, since that silently loses integer identity above 2**24 (MXR-080-0062). A model boundary that genuinely wants a float context (e.g. to ride a shared float input path) casts explicitly on its own side; that is a downstream, deliberate choice, not something this data layer should do silently on a caller’s behalf. block/batch_size/epochs/seed are likewise validated eagerly, at call time – before any batch is produced – rather than failing lazily (or silently emitting zero batches) once iteration begins (MXR-080-0063).

stream_token_source(token_ids, block, batch_size, *, epochs=1, shuffle=True, seed=0)[source]

Yield (context_windows (b, block) int64, next_tokens (b,) int64) micro-batches; never buffers windows.

The token array – and, with shuffle=True, one O(len(token_ids)) permutation for the epoch order – is the only resident data; each micro-batch’s windows are built on the fly and discarded.

token_ids is validated once up front (finite, exact-integer, lossless int64 range) and stays int64 through every yielded batch – it is never downcast to float32 here, since that silently loses integer identity for ids at or above 2**24 (MXR-080-0062); a consumer that wants float input casts explicitly on its own side.

block, batch_size, epochs, and seed are all validated eagerly, when this function is CALLED – before the returned iterator yields anything – rather than lazily on first iteration. block/batch_size have no meaningful “zero work” case and must be positive (a zero batch_size previously reached a bare range(..., step=0), and a negative one silently produced zero batches forever). epochs=0 IS a legitimate, explicit no-op (yields nothing, no error) – only a NEGATIVE epoch count is rejected, since that has no sensible meaning and previously also silently did nothing.

Parameters:
Return type:

Iterator[tuple[ndarray, ndarray]]