mixle.represent.learned_segment module

Learned segmentation – infer WHERE the tokens are, instead of cutting at fixed positions.

A fixed segmenter cuts every N bytes / every patch; but the right boundaries depend on the data. This fits an HMM over a finer atomic stream and cuts where the latent state changes: the boundaries are chosen by maximum likelihood, so the segmentation is inferred – the objective-coupled tokenizer. A run of same-state atoms becomes one variable-length token, pooled into a fixed feature vector for the embedding (a bag-of-symbols histogram for a discrete atomic stream, the mean vector for a continuous one).

LearnedSegmenter wraps any atomic Segmenter, is fit on example raws, and then plugs into the representation pipeline exactly like a fixed segmenter – so tokenization is a model you train, reusing mixle’s HMM inference, not a hand-set rule.

class LearnedSegmenter(atomic, n_states=4, *, max_its=30, seed=0)

Bases: Segmenter

Cut a raw object into variable-length tokens at HMM state changes over its atomic units (pooled to features).

Parameters:

• atomic (Segmenter)

• n_states (int)

• max_its (int)

• seed (int)

fit(raws)

Fit the boundary HMM on example raws – the segmentation is learned to maximize their likelihood.

An all-empty batch stays non-fatal (see the feat fallback below), but it is recorded: the resulting HMM saw zero emissions, so n_fit_atoms is 0 and segment() refuses to invent boundaries from it rather than returning fabricated tokens.

Parameters:

raws (Sequence[Any])

Return type:

LearnedSegmenter

segment(raw)

Segment raw input using fitted HMM state assignments.

An empty input yields ZERO tokens, not one all-zero token. Returning a fabricated unit meant an empty modality handed downstream encoders a full-width feature vector backed by no atoms, no state path, and no inferred boundary – evidence that was never observed.

Raises:

RuntimeError – if fit() was never called, or was called on a batch containing no atomic units at all – an HMM fitted without data cannot locate boundaries, so the no-evidence state is explicit rather than silently productive.

Parameters:

raw (Any)

Return type:

ndarray