mixle.data.stream_token_source module¶
A streaming token source: yield (context-window, next-token) micro-batches from a token-id array via a
resumable cursor, WITHOUT ever materializing the (N, block) window matrix or a Python list of observations.
This is the data half of the non-buffering streaming estimator. The standard encoder builds and buffers every
(window -> next) observation (O(corpus x block) host RAM – the materialization wall); this yields them a
micro-batch at a time from the read-only token array (O(corpus) resident + O(batch x block) ephemeral).
For a real out-of-core corpus the same generator shape reads from a memory-mapped / sharded token file; the
cursor is resumable, so a checkpoint is just its position.
- stream_token_source(token_ids, block, batch_size, *, epochs=1, shuffle=True, seed=0)[source]
Yield
(context_windows (b, block) float32, next_tokens (b,) int)micro-batches; never buffers windows.The token array is the only resident data; each micro-batch’s windows are built on the fly and discarded.