mixle.represent.heterogeneous module

Heterogeneous encoding – text, images, signals, sequences, and arbitrary scientific structures in ONE space.

A ModalityEncoder pairs a segmenter (raw -> units) with an embedding (units -> R^dim): one modality’s path into the shared space. A HeterogeneousEncoder is a registry of them plus a learned modality tag (a vector added to each unit so the downstream model knows the source), so a record with several modalities – an image, a caption, a seismic trace, a molecule – becomes a single (N, dim) stream of vectors a single model (a transformer, a mixture, the dependency-structure learner) consumes. Modalities compose because they land in the same space; that is the representation-layer form of mixle’s “compose heterogeneous things into one model”.

Everything is torch modules, so the encoders train end to end – to a generative objective (wrap the stream in a language-model / density leaf) or a downstream one (pool + a task head). Shared embeddings (the same CategoricalEmbedding/FeatureEmbedding instance in two modalities) tie their vectors, as before.

class ModalityEncoder(segmenter, embedding)

Bases: object

One modality’s raw -> (n_units, dim) path: a segmenter feeding an embedding into the shared space.

Parameters:

• segmenter (Any)

• embedding (Any)

encode(raw)

Torch tensor (n_units, dim) – gradients flow to the embedding (and, via it, train the encoder).

Parameters:

raw (Any)

Return type:

Any

encode_numpy(raw)

Detached (n_units, dim) array – for eval, quantization, or feeding a non-torch model.

Parameters:

raw (Any)

Return type:

ndarray

class HeterogeneousEncoder(dim)

Bases: object

A registry of per-modality encoders sharing one dim space, plus a learned modality-type embedding.

Parameters:

dim (int)

register(modality, segmenter, embedding, *, rebind=False)

Add a modality’s (segmenter, embedding); the embedding’s dim must match the shared space.

Parameters:

• modality (str)

• segmenter (Any)

• embedding (Any)

• rebind (bool)

Return type:

HeterogeneousEncoder

register_encoder(modality, encoder, *, rebind=False)

Add a pre-built ModalityEncoder (e.g. a GraphEncoder that owns its own segment+embed path).

Re-registering an already-registered modality (same name – e.g. swapping its encoder) leaves any already-learned modality-tag embedding untouched, since the modality set did not change. Registering a genuinely new modality grows the tag embedding by one row, preserving already-learned rows for the modalities that were already registered.

Registration is a change to the trainable parameter set. Once parameters() has handed that set to an optimizer, a later registration introduces parameter objects the optimizer does not own – the grown tag embedding is a new module, so its weight receives gradients but is never stepped, and the same holds for the newly registered encoder’s own parameters. That looked exactly like successful dynamic registration while silently freezing every new and replaced tensor. Such a registration now raises unless the caller passes rebind=True, which performs it and invalidates the export, so the optimizer must be rebuilt from a fresh parameters() call before training resumes.

Parameters:

• modality (str)

• encoder (ModalityEncoder)

• rebind (bool)

Return type:

HeterogeneousEncoder

encode(record)

A record {modality: raw} -> (stream, modality_ids): one (N, dim) tensor + each unit’s source id.

Each modality’s units are embedded and the modality-tag vector is added, then all are concatenated in registration order – the order modalities were registered, independent of record’s own key order – the unified token stream for a downstream model. record must carry exactly the registered modalities: a key naming no registered modality, or a registered modality missing from record, raises KeyError.

Parameters:

record (dict[str, Any])

Return type:

tuple[Any, ndarray]

encode_numpy(record)

Encode a heterogeneous record and return NumPy token stream plus modality ids.

Parameters:

record (dict[str, Any])

Return type:

tuple[ndarray, ndarray]

parameters()

Every trainable parameter across all modality encoders + the modality-tag embedding (for an optimizer).

Calling this marks the parameter set as exported: a later register_encoder() that would add trainable state outside the optimizer built from this list is refused (see that method).

Return type:

list