mixle.represent.generative module

Generative objective – fit the embedding (and its codebook) to model the data, so tokenization is inferred.

The self-supervised half of “fit to the objective”: rather than tune the encoder to a label, train it to reconstruct its input – an autoencoder over units. The shared-space vector must retain enough to rebuild the unit, so the representation is a generative one (no collapse). Turn on a VectorQuantizer and it becomes a VQ-VAE: encode -> quantize (straight-through) -> decode, with the codebook periodically refit on the current embeddings. Now the vocabulary is chosen to best reconstruct the data – tokenization inferred under a generative objective, exactly the thing a hardcoded BPE cannot do.

fit_autoencoder returns the trained encoder + decoder (+ codebook) and the reconstruction-loss history. It is modality-agnostic: feed it the unit-feature array from any continuous segmenter (patches, windows, atoms, …).

class AutoencoderResult(encoder, decoder, quantizer, losses=<factory>)[source]

Bases: object

A generatively-trained representation: the encoder, its decoder, an optional codebook, and the loss curve.

Parameters:

  • encoder (FeatureEmbedding)

  • decoder (Any)

  • quantizer (VectorQuantizer | None)

  • losses (list[float])

encoder: FeatureEmbedding
decoder: Any
quantizer: VectorQuantizer | None
losses: list[float]
encode(units)[source]
Parameters:

units (ndarray)

Return type:

ndarray

fit_autoencoder(units, dim, *, hidden=(), quantizer=None, epochs=200, lr=1e-2, refit_codebook_every=25, commitment=0.25, seed=0)[source]

Train an encoder+decoder to reconstruct units (N, in_features); optionally through a VQ bottleneck.

Without quantizer this is a plain autoencoder (the encoder becomes a generative representation). With one, it is a VQ-VAE: the encoder’s vectors are quantized (straight-through) before decoding and the codebook is refit every refit_codebook_every epochs on the current embeddings, so the learned vocabulary adapts to the representation. commitment weights the VQ codebook-commitment term.

Parameters:
Return type:

AutoencoderResult