mixle.represent package

The representation layer – embed any modality into one shared space; discretize it only if (and how) you want.

The tokenizer question, resolved by not fixing a vocabulary. Three separable pieces:

• segment – a Segmenter cuts raw data (text, image, signal, sequence, structure) into units, committing to a decomposition but no vocabulary;

• embed – an Embedding maps each unit into shared R^dim (a lookup table for a discrete unit, a small encoder for a continuous one), sharable/tie-able across models;

• quantize – an optional, learned VectorQuantizer turns the shared vectors into discrete ids in the embedding space, so a codebook (a “vocabulary”) is fit to the data – and can be shared across modalities – rather than guessed.

HeterogeneousEncoder composes per-modality (segmenter, embedding) pairs into one (N, dim) stream a single downstream model consumes – trainable to a generative or a downstream objective, so “the right tokenization” is inferred under the objective rather than fixed upfront.

class PosteriorRetriever(model, corpus, *, evidence_cap=1.0, field_weights=None)

Bases: object

Retrieve over raw heterogeneous records by the fitted mixture’s posterior affinity.

Parameters:

• model (Any)

• corpus (Any)

• evidence_cap (float | None)

• field_weights (Any)

affinity_matrix()

The corpus’s dense (n, n) log-affinity matrix (diagonal -inf).

Return type:

ndarray

retrieve(query, k=5)

Top-k corpus records for one query: [(corpus_index, log_affinity), ...] best first.

Parameters:

• query (Any)

• k (int)

Return type:

list[tuple[int, float]]

retrieve_batch(queries, k=5)

Top-k per query, computed in one joint pass over corpus + queries.

Parameters:

• queries (Any)

• k (int)

Return type:

list[list[tuple[int, float]]]

class AutoencoderResult(encoder, decoder, quantizer, losses=<factory>)

Bases: object

A generatively-trained representation: the encoder, its decoder, an optional codebook, and the loss curve.

Parameters:

• encoder (FeatureEmbedding)

• decoder (Any)

• quantizer (VectorQuantizer | None)

• losses (list[float])

encoder: FeatureEmbedding

decoder: Any

quantizer: VectorQuantizer | None

losses: list[float]

encode(units)

Parameters:

units (ndarray)

Return type:

ndarray

class ByteSegmenter

Bases: Segmenter

A string/bytes -> (n,) byte ids in [0, 256). The vocabulary-free text decomposition.

discrete: bool = True

num_categories = 256

segment(raw)

Parameters:

raw (Any)

Return type:

ndarray

class CategoricalEmbedding(num_categories, dim, *, name=None)

Bases: object

A lazily-built learned embedding of shape (num_categories, dim); every consumer gets the same module.

Parameters:

• num_categories (int)

• dim (int)

• name (str | None)

module()

The underlying nn.Embedding – built on first call, the identical instance thereafter.

Return type:

Any

class Embedder(featurizer, result, kind, corpus_vectors)

Bases: object

A fitted embedding of raw heterogeneous items: transform to vectors, retrieve neighbours.

Parameters:

• featurizer (Any)

• result (AutoencoderResult)

• kind (str)

• corpus_vectors (np.ndarray)

property dim: int

transform(items)

Embed items into the learned space, unit-normalized (so dot = cosine similarity).

Parameters:

items (Any)

Return type:

ndarray

retrieve(query, k=5)

Top-k fitted-corpus neighbours of query as (corpus index, cosine similarity).

Parameters:

• query (Any)

• k (int)

Return type:

list[tuple[int, float]]

save(path)

Parameters:

path (str)

Return type:

str

classmethod load(path)

Parameters:

path (str)

Return type:

Embedder

class ElementSegmenter(alphabet)

Bases: Segmenter

A sequence of hashable symbols (chars, amino acids, k-mers, categories) -> (n,) ids via a fixed alphabet.

Given alphabet (the ordered symbols), each element maps to its index; unknown symbols map to 0. The natural decomposition for proteins/genomes/any categorical sequence, and for characters (alphabet=list(...)).

Parameters:

alphabet (list[Any])

discrete: bool = True

segment(raw)

Parameters:

raw (Any)

Return type:

ndarray

class FeatureEmbedding(in_features, dim, *, hidden=(), name=None)

Bases: object

A continuous unit encoder: (n_units, in_features) -> (n_units, dim) via a linear or small-MLP module.

The continuous analogue of CategoricalEmbedding – same dim / .module() contract, so it shares and trains identically. hidden=() is a single linear projection (a learned patch/ window/element embedding); non-empty hidden inserts ReLU layers.

Parameters:

• in_features (int)

• dim (int)

• hidden (Sequence[int])

• name (str | None)

module()

Return type:

Any

class GraphEmbedding(in_features, dim, *, layers=2, name=None)

Bases: object

A message-passing embedding: (node_features, adjacency) -> (n_nodes, dim) with layers GCN rounds.

Parameters:

• in_features (int)

• dim (int)

• layers (int)

• name (str | None)

module()

Return type:

Any

class GraphEncoder(embedding)

Bases: ModalityEncoder

A structure modality: raw = (node_features (n, f), adjacency (n, n)) -> per-node vectors in shared R^dim.

Parameters:

embedding (GraphEmbedding)

encode(raw)

Torch tensor (n_units, dim) – gradients flow to the embedding (and, via it, train the encoder).

Parameters:

raw (Any)

Return type:

Any

encode_numpy(raw)

Detached (n_units, dim) array – for eval, quantization, or feeding a non-torch model.

Parameters:

raw (Any)

Return type:

ndarray

class HeterogeneousEncoder(dim)

Bases: object

A registry of per-modality encoders sharing one dim space, plus a learned modality-type embedding.

Parameters:

dim (int)

register(modality, segmenter, embedding)

Add a modality’s (segmenter, embedding); the embedding’s dim must match the shared space.

Parameters:

• modality (str)

• segmenter (Any)

• embedding (Any)

Return type:

HeterogeneousEncoder

register_encoder(modality, encoder)

Add a pre-built ModalityEncoder (e.g. a GraphEncoder that owns its own segment+embed path).

Parameters:

• modality (str)

• encoder (ModalityEncoder)

Return type:

HeterogeneousEncoder

encode(record)

A record {modality: raw} -> (stream, modality_ids): one (N, dim) tensor + each unit’s source id.

Each modality’s units are embedded and the modality-tag vector is added, then all are concatenated in registration order – the unified token stream for a downstream model.

Parameters:

record (dict[str, Any])

Return type:

tuple[Any, ndarray]

encode_numpy(record)

Parameters:

record (dict[str, Any])

Return type:

tuple[ndarray, ndarray]

parameters()

Every trainable parameter across all modality encoders + the modality-tag embedding (for an optimizer).

Return type:

list

class LearnedSegmenter(atomic, n_states=4, *, max_its=30, seed=0)

Bases: Segmenter

Cut a raw object into variable-length tokens at HMM state changes over its atomic units (pooled to features).

Parameters:

• atomic (Segmenter)

• n_states (int)

• max_its (int)

• seed (int)

discrete: bool = False

fit(raws)

Fit the boundary HMM on example raws – the segmentation is learned to maximize their likelihood.

Parameters:

raws (Sequence[Any])

Return type:

LearnedSegmenter

segment(raw)

Parameters:

raw (Any)

Return type:

ndarray

class ModalityEncoder(segmenter, embedding)

Bases: object

One modality’s raw -> (n_units, dim) path: a segmenter feeding an embedding into the shared space.

Parameters:

• segmenter (Any)

• embedding (Any)

encode(raw)

Torch tensor (n_units, dim) – gradients flow to the embedding (and, via it, train the encoder).

Parameters:

raw (Any)

Return type:

Any

encode_numpy(raw)

Detached (n_units, dim) array – for eval, quantization, or feeding a non-torch model.

Parameters:

raw (Any)

Return type:

ndarray

class PatchSegmenter(patch=8)

Bases: Segmenter

An image (H, W) or (C, H, W) -> (n_patches, patch_features) float units (ViT-style, no vocab).

Parameters:

patch (int)

discrete: bool = False

segment(raw)

Parameters:

raw (Any)

Return type:

ndarray

unit_features(channels=1)

Parameters:

channels (int)

Return type:

int

class Segmenter

Bases: object

Base: segment(raw) -> np.ndarray. discrete says whether units are ids (vs. float features).

discrete: bool = False

segment(raw)

Parameters:

raw (Any)

Return type:

ndarray

class SetSegmenter

Bases: Segmenter

A set/list of feature vectors -> (n_elements, feat): nodes of a graph, atoms of a molecule, taxa of a section.

The general structured-object decomposition – a scientific structure becomes its set of element features (which a downstream model can further couple with a structure/message-passing embedding).

discrete: bool = False

segment(raw)

Parameters:

raw (Any)

Return type:

ndarray

class VectorQuantizer(num_codes, dim, *, seed=0)

Bases: object

A learned codebook over R^dim: nearest-centroid quantization of embedding vectors into discrete ids.

Parameters:

• num_codes (int)

• dim (int)

• seed (int)

fit(vectors, *, iters=25)

Fit the codebook by k-means (Lloyd) on vectors (n, dim) – the vocabulary is learned, not assumed.

Parameters:

• vectors (ndarray)

• iters (int)

Return type:

VectorQuantizer

quantize(vectors)

Nearest-code id for each vector – the discrete token stream (n,).

Parameters:

vectors (ndarray)

Return type:

ndarray

dequantize(ids)

Codebook vectors for token ids (n,) -> (n, dim) (the reconstruction / de-tokenization).

Parameters:

ids (ndarray)

Return type:

ndarray

reconstruction_error(vectors)

Mean squared quantization error – the codebook’s fidelity (a codebook-size / bitrate knob).

Parameters:

vectors (ndarray)

Return type:

float

straight_through(vectors)

VQ-VAE straight-through estimator: return quantized vectors but pass gradients to vectors unchanged.

Lets the encoders and (with a codebook-commitment loss) the codebook train end to end through the discrete bottleneck. vectors is a torch tensor (n, dim).

Parameters:

vectors (Any)

Return type:

Any

class WholeSegmenter

Bases: Segmenter

A single feature vector -> (1, feat): the object is one unit (a pooled structure descriptor, a record).

discrete: bool = False

segment(raw)

Parameters:

raw (Any)

Return type:

ndarray

class WindowSegmenter(window=64, hop=None)

Bases: Segmenter

A 1-D signal (T,) -> (n_frames, window) float units by a sliding window (seismic/audio/time-series).

Parameters:

• window (int)

• hop (int | None)

discrete: bool = False

segment(raw)

Parameters:

raw (Any)

Return type:

ndarray

fit_autoencoder(units, dim, *, hidden=(), quantizer=None, epochs=200, lr=1e-2, refit_codebook_every=25, commitment=0.25, seed=0)

Train an encoder+decoder to reconstruct units (N, in_features); optionally through a VQ bottleneck.

Without quantizer this is a plain autoencoder (the encoder becomes a generative representation). With one, it is a VQ-VAE: the encoder’s vectors are quantized (straight-through) before decoding and the codebook is refit every refit_codebook_every epochs on the current embeddings, so the learned vocabulary adapts to the representation. commitment weights the VQ codebook-commitment term.

Parameters:

• units (ndarray)

• dim (int)

• hidden (tuple[int, ...])

• quantizer (VectorQuantizer | None)

• epochs (int)

• lr (float)

• refit_codebook_every (int)

• commitment (float)

• seed (int)

Return type:

AutoencoderResult

fit_embedder(data, dim=32, *, kind=None, feature_dim=256, hidden=(64,), epochs=200, lr=1e-2, seed=0)

Fit a learned embedding of raw text or record items and return an Embedder.

Items featurize deterministically (hashing trick; no fitted vocabulary), then an autoencoder learns a dim-dimensional generative representation of the corpus. retrieve works out of the box over the fitted data; transform embeds anything of the same kind.

Parameters:

• data (Any)

• dim (int)

• kind (str | None)

• feature_dim (int)

• hidden (tuple[int, ...])

• epochs (int)

• lr (float)

• seed (int)

Return type:

Embedder

vectorize(item, kind, *, dim=16, embedder=None)

Map a raw item of modality kind to a fixed dim vector (see module docstring).

Parameters:

• item (Any) – the raw item (a string, a record, an image array, a signal array).

• kind (str) – 'text' | 'record' | 'image' | 'signal'.

• dim (int) – output vector dimension.

• embedder (Any) – for text/record, a fitted Embedder to reuse (else a small one is fit on the single item – pass one for consistency across a corpus).

Return type:

ndarray

vectorize_all(items, kind, *, dim=16)

Vectorize a sequence of same-modality items to an (n, dim) array (one shared embedder for text).

Parameters:

• items (Any)

• kind (str)

• dim (int)

Return type:

ndarray

image_features(img, dim=16, *, grid=None)

A fixed dim descriptor of an image: mean intensity over a g x g grid of cells.

img is (H, W) or (H, W, C); channels are averaged. The grid side g is chosen so g*g covers dim (then truncated/padded to exactly dim), giving a coarse spatial-layout vector – enough for an image field to correlate with structured fields in a discovered graph.

Parameters:

• img (Any)

• dim (int)

• grid (int | None)

Return type:

ndarray

signal_features(sig, dim=16, *, windows=None)

A fixed dim descriptor of a 1-D signal: (mean, energy, range) over evenly-spaced windows.

Parameters:

• sig (Any)

• dim (int)

• windows (int | None)

Return type:

ndarray

Submodules

• mixle.represent.api module
• mixle.represent.embed module
• mixle.represent.generative module
• mixle.represent.graph module
• mixle.represent.heterogeneous module
• mixle.represent.learned_segment module
• mixle.represent.modality module
• mixle.represent.posterior module
• mixle.represent.quantize module
• mixle.represent.segment module