mixle.represent.learned_segment module

Learned segmentation – infer WHERE the tokens are, instead of cutting at fixed positions.

A fixed segmenter cuts every N bytes / every patch; but the right boundaries depend on the data. This fits an HMM over a finer atomic stream and cuts where the latent state changes: the boundaries are chosen by maximum likelihood, so the segmentation is inferred – the objective-coupled tokenizer. A run of same-state atoms becomes one variable-length token, pooled into a fixed feature vector for the embedding (a bag-of-symbols histogram for a discrete atomic stream, the mean vector for a continuous one).

LearnedSegmenter wraps any atomic Segmenter, is fit on example raws, and then plugs into the representation pipeline exactly like a fixed segmenter – so tokenization is a model you train, reusing mixle’s HMM inference, not a hand-set rule.

class LearnedSegmenter(atomic, n_states=4, *, max_its=30, seed=0)

Bases: Segmenter

Cut a raw object into variable-length tokens at HMM state changes over its atomic units (pooled to features).

Parameters:

• atomic (Segmenter)

• n_states (int)

• max_its (int)

• seed (int)

discrete: bool = False

hmm: Any

feat: int | None

fit(raws)

Fit the boundary HMM on example raws – the segmentation is learned to maximize their likelihood.

Parameters:

raws (Sequence[Any])

Return type:

LearnedSegmenter

segment(raw)

Parameters:

raw (Any)

Return type:

ndarray