mixle.represent.heterogeneous module

Heterogeneous encoding – text, images, signals, sequences, and arbitrary scientific structures in ONE space.

A ModalityEncoder pairs a segmenter (raw -> units) with an embedding (units -> R^dim): one modality’s path into the shared space. A HeterogeneousEncoder is a registry of them plus a learned modality tag (a vector added to each unit so the downstream model knows the source), so a record with several modalities – an image, a caption, a seismic trace, a molecule – becomes a single (N, dim) stream of vectors a single model (a transformer, a mixture, the dependency-structure learner) consumes. Modalities compose because they land in the same space; that is the representation-layer form of mixle’s “compose heterogeneous things into one model”.

Everything is torch modules, so the encoders train end to end – to a generative objective (wrap the stream in a language-model / density leaf) or a downstream one (pool + a task head). Shared embeddings (the same CategoricalEmbedding/FeatureEmbedding instance in two modalities) tie their vectors, as before.

class ModalityEncoder(segmenter, embedding)

Bases: object

One modality’s raw -> (n_units, dim) path: a segmenter feeding an embedding into the shared space.

Parameters:

• segmenter (Any)

• embedding (Any)

encode(raw)

Torch tensor (n_units, dim) – gradients flow to the embedding (and, via it, train the encoder).

Parameters:

raw (Any)

Return type:

Any

encode_numpy(raw)

Detached (n_units, dim) array – for eval, quantization, or feeding a non-torch model.

Parameters:

raw (Any)

Return type:

ndarray

class HeterogeneousEncoder(dim)

Bases: object

A registry of per-modality encoders sharing one dim space, plus a learned modality-type embedding.

Parameters:

dim (int)

register(modality, segmenter, embedding)

Add a modality’s (segmenter, embedding); the embedding’s dim must match the shared space.

Parameters:

• modality (str)

• segmenter (Any)

• embedding (Any)

Return type:

HeterogeneousEncoder

register_encoder(modality, encoder)

Add a pre-built ModalityEncoder (e.g. a GraphEncoder that owns its own segment+embed path).

Parameters:

• modality (str)

• encoder (ModalityEncoder)

Return type:

HeterogeneousEncoder

encode(record)

A record {modality: raw} -> (stream, modality_ids): one (N, dim) tensor + each unit’s source id.

Each modality’s units are embedded and the modality-tag vector is added, then all are concatenated in registration order – the unified token stream for a downstream model.

Parameters:

record (dict[str, Any])

Return type:

tuple[Any, ndarray]

encode_numpy(record)

Parameters:

record (dict[str, Any])

Return type:

tuple[ndarray, ndarray]

parameters()

Every trainable parameter across all modality encoders + the modality-tag embedding (for an optimizer).

Return type:

list