mixle.reason.adapter module

StructuredAdapter for adapting a frozen multimodal encoder while preserving transfer.

For a frozen VLM, the encoder is the expensive component and the trainable bridge on top is the application-specific component. StructuredAdapter uses a low-capacity structured map that adapts to a task while preserving zero-shot transfer to text-specified classes; a full unstructured map can overfit and damage that transfer even with regularization.

The map is a residual, class-agnostic transform of the image embedding:

g(x) = x + (diag ⊙ x) + U Vᵀ x          # identity + diagonal reweight + rank-r correction

Three structural choices matter: (1) it is residual with weight decay, so it stays near the encoder’s alignment; (2) it is class-agnostic: targets enter only as anchor embeddings such as class-text embeddings, so a map fit on some classes still scores classes it never saw at training time; (3) geometry is cosine throughout – fit() L2-normalizes both the adapted embedding and the anchors before the temperature-scaled softmax, and scores() normalizes both the same way before its dot product. Anchor magnitude is therefore never meaningful at either stage: what the training loss optimizes for is exactly what scores() reports at serving time. diag + U Vᵀ is the same diagonal+low-rank structure Mixle uses for structured transition operators, here over a VLM bridge.

The same recipe applies to any frozen encoder that emits comparable embeddings. Torch is imported lazily.

class StructuredAdapter(dim, *, rank=8, weight_decay=1.0, full=False)[source]

Bases: object

A residual diagonal+low-rank adapter over frozen embeddings.

rank sets the low-rank correction’s width; weight_decay pulls the map toward identity (preserve the encoder’s geometry). full=True selects the unstructured baseline. Fit on (embeddings, labels, anchors); score any embeddings against any anchors, including anchors for classes not seen in training.

Parameters:
fit(embeddings, labels, anchors, *, epochs=300, lr=0.01, init_temp=0.07, seed=0)[source]

Train the residual map so g(image) matches its label’s anchor. labels index into anchors.

Both the adapted embedding and the anchors are L2-normalized before the temperature-scaled softmax, so the training objective is pure cosine similarity – exactly what scores() reports at serving time. Anchor magnitude therefore never enters either the loss or the score; only direction does. seed makes the low-rank map’s random initialization reproducible (diag and U start at zero – only V, and the full-matrix baseline’s w, involve randomness, and w starts at zero too, so today only V actually needs the seed).

Validates its inputs and requires at least one real optimization step before mutating any adapter state, so a failed call (bad shapes/labels/epochs, or training that diverges to a non-finite loss) leaves the adapter exactly as it was – still requiring a genuine fit() before transform()/ scores() will work.

Parameters:
Return type:

StructuredAdapter

transform(embeddings)[source]

Apply the learned residual map and L2-normalize – the adapted embedding.

Parameters:

embeddings (ndarray)

Return type:

ndarray

scores(embeddings, anchors)[source]

Cosine similarity of adapted embeddings to anchors; anchors may represent new classes.

Parameters:
Return type:

ndarray

predict(embeddings, anchors)[source]

Return the highest-scoring anchor index for each embedding.

Parameters:
Return type:

ndarray

n_params()[source]

Return the number of learned adapter parameters.

Return type:

int