Representation and Model Families

This tutorial sketches a heterogeneous representation pipeline and shows how it connects to model families.

The example record has two modalities:

  • text describing an event;

  • a numeric signal observed around the same event.

Build a Shared Encoder

from mixle.represent import (
    ByteSegmenter,
    CategoricalEmbedding,
    FeatureEmbedding,
    HeterogeneousEncoder,
    WindowSegmenter,
)

encoder = HeterogeneousEncoder(dim=64)
encoder.register("text", ByteSegmenter(), CategoricalEmbedding(256, 64))
encoder.register("signal", WindowSegmenter(window=128, hop=64), FeatureEmbedding(128, 64))

stream, tags = encoder.encode({
    "text": "pump pressure rose quickly",
    "signal": waveform,
})

The stream can be consumed by a neural sequence model, pooled for a task head, or summarized before fitting a classical distribution.

Inspect tags before moving on. A good representation pipeline should make it clear which embedding rows came from which modality and segment. If the alignment metadata is lost here, later model diagnostics will be much harder to explain.

Keep the encoder configuration with the records used to validate it. Segmenter settings, tokenization behavior, numeric scaling, and modality keys define the data contract just as much as the model class does.

Optional Quantization

If the downstream model needs discrete ids, learn a codebook in the shared embedding space.

from mixle.represent import VectorQuantizer

vectors, _ = encoder.encode_numpy(record)

codebook = VectorQuantizer(num_codes=256, dim=64).fit(vectors)
token_ids = codebook.quantize(vectors)

Quantization is optional. It is most useful for compression, enumeration, discrete language-model style training, and stable production artifacts.

Quantization also changes the validation question. Before using token ids as a model input, compare reconstruction or nearest-code error across modalities. Large errors in one modality can make a downstream model look calibrated on average while failing on the modality that was compressed poorly.

If quantized tokens are persisted, store the codebook identity and fitting data window. Reusing an old codebook on shifted inputs can look like a model failure when the representation has actually drifted.

Choose a Model Family

Once the representation exists, choose the model family based on the modeling question. Prefer stable mixle.stats families when they fit; use mixle.models helpers when the applied family is specifically needed and you are ready to validate it.

Question

Candidate

What token or event comes next?

Incubating TransformerLMEstimator or StreamingTransformer.

What continuous response follows from features?

Applied helpers such as GaussianProcessRegressor or RandomForestEstimator.

Are there unknown clusters?

fit_truncated_dpm or a standard MixtureEstimator.

Is there a latent regime over time?

HMMs in HMMs and Latent Structure.

Are event times self-exciting?

Process models in Temporal and Stochastic Processes.

Is the output a structured decision?

Relations in Relations.

The representation layer should not decide the modeling story by itself. It should preserve evidence in a form that lets the right model family use it.

Keep the first model deliberately simple. A pooled Gaussian or categorical baseline is often a better first diagnostic than a large neural sequence model: it reveals whether the representation contains useful signal before the model capacity can hide representation problems.

Move to a larger family only when the simpler baseline has answered its diagnostic question. The goal is not to prove that the baseline is sufficient; it is to catch invalid alignment, missing modalities, or empty signal before training cost obscures the failure.

Practical Checks

Before committing to a representation:

  • verify that each modality produces the expected number of units;

  • check vector dimensions before registering encoders;

  • inspect quantization reconstruction error if a codebook is used;

  • hold out data before comparing representation choices;

  • keep the representation config with the fitted model artifact.

Release or production evidence should record:

  • the modality keys accepted by the encoder;

  • segmenter settings, embedding dimensions, and quantizer settings;

  • the baseline model used to check that the representation carries signal;

  • held-out metrics for each modality or task, not only the aggregate score; and

  • the artifact location where the encoder configuration is stored.

Common Failure Modes

Silent shape drift

A segmenter change alters the number of produced units, but the downstream model still accepts the tensor. Track expected unit counts or add explicit assertions around the representation boundary.

Modality dominance

One modality produces many more units or much larger vector norms, so pooled representations mostly reflect that modality. Normalize deliberately and inspect per-modality contribution.

Unreviewed compression

A codebook improves storage size but destroys rare-event signal. Treat quantization as a modeling choice and validate it against the task that will consume the tokens.

Release Evidence

For release-like representation work, include:

  • a small schema example showing accepted modality keys and value shapes;

  • segmenter and embedding configuration;

  • quantizer configuration and reconstruction or nearest-code diagnostics when a codebook is used;

  • per-modality validation metrics or qualitative checks;

  • the baseline model used to verify signal; and

  • the artifact reference that ties the encoder configuration to the fitted model.

Without that evidence, a representation pipeline is difficult to reproduce and hard to debug when the downstream model changes.