mixle.reason.adapter module¶
StructuredAdapter for adapting a frozen multimodal encoder while preserving transfer.
For a frozen VLM, the encoder is the expensive component and the trainable bridge
on top is the application-specific component. StructuredAdapter uses a
low-capacity structured map that adapts to a task while preserving zero-shot
transfer to text-specified classes; a full unstructured map can overfit and
damage that transfer even with regularization.
The map is a residual, class-agnostic transform of the image embedding:
g(x) = x + (diag ⊙ x) + U Vᵀ x # identity + diagonal reweight + rank-r correction
Two structural choices matter: (1) it is residual with weight decay, so it stays
near the encoder’s alignment; (2) it is class-agnostic: targets enter only as
anchor embeddings such as class-text embeddings, so a map fit on some classes
still scores classes it never saw at training time. diag + U Vᵀ is the same
diagonal+low-rank structure Mixle uses for structured transition operators,
here over a VLM bridge.
The same recipe applies to any frozen encoder that emits comparable embeddings. Torch is imported lazily.
- class StructuredAdapter(dim, *, rank=8, weight_decay=1.0, full=False)[source]
Bases:
objectA residual diagonal+low-rank adapter over frozen embeddings.
ranksets the low-rank correction’s width;weight_decaypulls the map toward identity (preserve the encoder’s geometry).full=Trueselects the unstructured baseline. Fit on(embeddings, labels, anchors); score any embeddings against any anchors, including anchors for classes not seen in training.- fit(embeddings, labels, anchors, *, epochs=300, lr=0.01, init_temp=0.07)[source]
Train the residual map so
g(image)matches its label’s anchor.labelsindex intoanchors.
- transform(embeddings)[source]
Apply the learned residual map and L2-normalize – the adapted embedding.
- scores(embeddings, anchors)[source]
Cosine similarity of adapted embeddings to
anchors; anchors may represent new classes.
- predict(embeddings, anchors)[source]
Return the highest-scoring anchor index for each embedding.