mixle.represent.api module

fit_embedder – point it at raw heterogeneous data, get back vectors and retrieval.

The one-call product surface over the representation layer: text or records (dicts / tuples of mixed fields) featurize through the task layer’s deterministic hashers, and a generatively-trained autoencoder (mixle.represent.generative.fit_autoencoder()) compresses them into a learned dim-space. The returned Embedder transforms new items into that space and retrieves nearest neighbours over the fitted corpus – model-based retrieval over raw data, no upstream tokenizer or external embedding API:

emb = fit_embedder(tickets, dim=16)          # dict/tuple records or strings
emb.transform(new_items)                     # (N, dim)
emb.retrieve(query, k=5)                     # [(corpus index, similarity), ...]
emb.save(path); Embedder.load(path)          # durable artifact

Deterministic given seed. Needs torch only to FIT; a saved Embedder reloads and transforms anywhere.

class Embedder(featurizer, result, kind, corpus_vectors)[source]

Bases: object

A fitted embedding of raw heterogeneous items: transform to vectors, retrieve neighbours.

Parameters:

  • featurizer (Any)

  • result (AutoencoderResult)

  • kind (str)

  • corpus_vectors (np.ndarray)

property corpus_vectors: ndarray

The fitted corpus’s (N, dim) unit embeddings, as a privately owned read-only array.

Retrieval ranks against these vectors, so they are the embedder’s evidence, not a scratch buffer: they are validated, normalized and frozen at construction, and exposed only through this property. Handing back the caller’s own writable array (or letting the attribute be rebound) meant a post-construction corpus_vectors[0, 0] = nan turned retrieve() into an ordinary-looking ranked result whose similarities were all NaN (MXR-080-1785).

property dim: int

Return embedding dimensionality.

property identity: RetrievalIdentity

What this embedder’s (index, similarity) pairs are relative to (MXR-080-1906).

A bare corpus index names nothing without the corpus it indexes, and a cosine similarity is only interpretable against the model that produced both sides of it. Neither was recorded: the save manifest carried kind/dim/n_corpus and no fitted-state identity at all, so two embedders over different corpora returned indistinguishable results.

The digest covers the fitted corpus vectors, which are frozen at construction, so this is stable for the instance’s lifetime and identifies every result it returns. It is computed on each access rather than cached because result and featurizer remain rebindable attributes – see the note in retrieve() about what that does and does not cover.

transform_one(item)[source]

Embed exactly ONE item; always returns a single (dim,) vector.

The unambiguous single-record contract. item is one whole record no matter which container it happens to be, so a list-valued record ([1, 2]) embeds to one vector here exactly as the equivalent tuple or dict does – see transform() for why the container alone cannot decide that.

Parameters:

item (Any)

Return type:

ndarray

transform_batch(items)[source]

Embed a SEQUENCE of items; always returns an (n, dim) block.

The unambiguous batch contract: every element of items is one whole record, never one record’s field.

Parameters:

items (Any)

Return type:

ndarray

transform(items)[source]

Embed one item or a batch of them, unit-normalized (so dot = cosine similarity).

A convenience wrapper over transform_one()/transform_batch() whose one-vs-batch decision is a guess from the outer container. That guess is only sound while a record cannot itself be list-shaped, and for kind="record" it can: _kind_of accepts a list as one record at fit time, so [1, 2] is a legitimate single record while [{...}, {...}] is just as legitimately a batch of two. Rather than silently returning two embedding rows for a single declared record – training and serving disagreeing about the same record type – the genuinely ambiguous shape raises and names the two explicit methods that carry no ambiguity at all.

Raises:

ValueError – If items is a kind="record" sequence whose own elements are not themselves record containers, so it is indistinguishable from one list-shaped record. Call transform_one() or transform_batch() instead.

Parameters:

items (Any)

Return type:

ndarray

retrieve(query, k=5)[source]

Top-k fitted-corpus neighbours of query as (corpus index, cosine similarity).

Indices are positions in corpus_vectors; identity records which corpus that is. Note what identity does NOT cover: result and featurizer are still plain rebindable attributes, so replacing emb.result re-points the query encoder while the frozen corpus vectors still come from the old one. Fixing that is a constructor-ownership change to a class that is pickled by save() and rebuilt field-by-field by load(), which is a wider change than this finding, and it is left deliberately undone rather than half-done.

Raises:

ValueError – If k is not an exact non-negative integer – see mixle.represent.identity.exact_count() for why a Boolean is refused and an integral float is not.

Parameters:
Return type:

list[tuple[int, float]]

save(path)[source]

Persist the embedder and fitted corpus vectors as one atomic, digest-bound envelope.

The artifact is a single file: magic, a canonical JSON manifest, then the pickled state. The manifest’s digest covers the manifest fields AND the body together, and the file is written to a temporary sibling and os.replace-d into position, so a reader can never see a half-written artifact or a manifest describing a different payload than the one beside it.

Parameters:

path (str)

Return type:

str

classmethod load(path, *, trust_code=False)[source]

Load an embedder previously saved with save(), verifying its manifest.

The saved artifact is a pickle of the fitted state, and result can embed a live torch module (see AutoencoderResult) – unpickling it executes arbitrary code from the file, exactly like pickle.load on an untrusted source. Refuses by default; pass trust_code=True for a path whose source you trust (or call from inside mixle.utils.serialization.trusted_deserialization()), matching mixle.inference.production.registry.Registry.get().

Beyond that trust gate, the manifest is now actually enforced rather than ignored (MXR-080-1786). The envelope digest – manifest fields and body together – is checked BEFORE the body is unpickled, so a truncated, corrupted, swapped or manifest-tampered artifact is rejected before any deserialization runs; and the decoded payload’s own type, kind, and corpus shape are checked against what the manifest declared, so state that does not match its description can never load under it. The digest is corruption/mismatch detection, not authentication: it lives in the same file it covers, so it cannot prove the artifact was not replaced wholesale by whoever could already write to path.

Parameters:
Return type:

Embedder

fit_embedder(data, dim=32, *, kind=None, feature_dim=256, hidden=(64,), epochs=200, lr=1e-2, seed=0)[source]

Fit a learned embedding of raw text or record items and return an Embedder.

Items featurize deterministically (hashing trick; no fitted vocabulary), then an autoencoder learns a dim-dimensional generative representation of the corpus. retrieve works out of the box over the fitted data; transform embeds anything of the same kind.

kind is one explicit input schema for the whole corpus, resolved and validated BEFORE any featurizing or fitting happens (MXR-080-1784). An explicit kind must name one of the two supported schemas; an unrecognized value used to select the record featurizer and only surface at the very end of an otherwise complete autoencoder fit. When kind is left to inference, EVERY item is inspected rather than only the first: a corpus mixing text and record items has no single schema to infer, and silently featurizing dict records through str() as text (or the reverse) is a wrong representation, not a usable one. Declare kind explicitly if that coercion is what you want.

Raises:

ValueError – If kind names no supported schema, or if it is left to inference over a corpus whose items are not all the same kind.

Parameters:
Return type:

Embedder