mixle.utils.hvis.stream module

Streaming HViS: place arriving points into a frozen model-based embedding, honestly.

Classic t-SNE/UMAP cannot stream: they are transductive (no out-of-sample map), their input probabilities are normalized over the whole dataset, and re-running on n+1 points produces an arbitrarily rotated/rearranged layout – the picture “jumps” and a monitoring view is useless.

HViS escapes all three because its affinities come from a FITTED MODEL, not from raw pairwise distances: every observation has a fixed encoder (x -> posterior / field evidence), so a new point’s affinity row against a frozen set of landmarks is self-contained – no other stream data required. That turns streaming into three separable, individually-checkable pieces:

  1. Atlas – embed a landmark reservoir once (htsne(), or any coordinates you supply, e.g. a humap layout). The atlas is FROZEN: existing coordinates never move while streaming, so the view is stable by construction rather than by hope.

  2. Placement – each arriving point gets its perplexity-calibrated affinity row over the landmarks only (reusing the exact factor/calibration machinery of the batch path), then minimizes its OWN row-KL against the frozen atlas under the same heavy-tailed kernel. One moving point per objective (the out-of-sample “transform” trick), O(landmarks) per point, vectorized across the whole batch since placed points do not interact.

  3. Drift – placement is only trustworthy while the model still fits the stream. The model gives a free, principled drift signal: the mean log-density of arrivals versus the landmark reference. When it trips, StreamingHvis.refresh() re-embeds warm-started from the current coordinates and Procrustes-aligns the result back onto the old atlas – and REPORTS the alignment residual, so a genuine geometry change is surfaced instead of being animated away.

With an estimator, the MODEL streams too, closing the loop with mixle’s native sufficient-statistic machinery: each arriving batch is E-stepped once at arrival time under the then-current model (accumulator.seq_update), and StreamingHvis.refresh() performs the M-step over reservoir + accumulated stream statistics before re-embedding – incremental EM (Neal & Hinton 1998), one honest sweep per refresh, never a silent full re-fit.

The one thing this deliberately does not promise: a placement is a projection into the atlas’s geometry. A point unlike anything in the reservoir gets a low-affinity row (and drags the drift score down) rather than a secretly-wrong confident position – check drift_score() before trusting a batch of placements.

class StreamingHvis(mix_model, landmark_data, *, atlas=None, emb_dim=2, alpha=1.0, perplexity=30.0, affinity='balanced', evidence_cap=1.0, field_weights=None, estimator=None, drift_threshold_nats=2.0, seed=None, **htsne_kwargs)[source]

Bases: object

A frozen model-based atlas that arriving points are placed into, with drift accounting.

Parameters:

  • mix_model (Any) – the fitted mixture the affinities come from (any model htsne accepts).

  • landmark_data (list) – the reservoir the atlas is built over. The model’s components make this easy to keep representative – e.g. sample a quota per component.

  • atlas (np.ndarray | None) – optional precomputed (len(landmark_data), emb_dim) coordinates (e.g. a humap layout). When omitted, the atlas is built here with htsne().

  • affinity (str) – any named HViS affinity. Note 'local' learns component-local metrics from the data the factors are built over, which during streaming is landmarks + batch – with a reasonably sized reservoir the landmarks dominate, but 'balanced' (the default) is a pure per-point function of the model and has no such coupling.

  • estimator (Any) – optional ParameterEstimator consistent with mix_model. When given, the MODEL streams too, by incremental EM (Neal & Hinton 1998): every add() batch is E-stepped once, at arrival time, under the model current at that moment, and its sufficient statistics accumulate; refresh() then performs one M-step over the reservoir’s statistics (E-stepped under the current model) combined with the accumulated stream statistics, adopting the re-estimated model before re-embedding. One honest EM sweep per refresh – NOT full-batch EM to convergence; pass refresh(mix_model=...) with your own fully re-fit model when that is what you want (an explicit model always wins, and discards the pending stream statistics).

  • drift_threshold_nats (float) – how far (in nats) the arrivals’ mean log-density may fall below the landmark reference before drifted trips.

  • htsne_kwargs (Any) – forwarded to htsne() for atlas builds and refresh().

  • emb_dim (int)

  • alpha (float)

  • perplexity (float | None)

  • evidence_cap (float | None)

  • seed (int | None)

add(batch, *, max_its=250, eta=None)[source]

Place a batch of arriving observations into the frozen atlas; returns (B, emb_dim).

Landmark coordinates are guaranteed unchanged by this call – stability is structural, not a tuning outcome. Also updates the running drift score from the batch’s log-density.

Parameters:
Return type:

ndarray

extend_landmarks(data, coords=None)[source]

Promote observations into the landmark reservoir (typically recent arrivals), placing them first if coordinates are not supplied. Grows the atlas without moving anything.

Parameters:
Return type:

None

drift_score()[source]

Nats of mean log-density the recent stream sits BELOW the landmark reference (>=0-ish; near zero or negative means the stream fits the model at least as well as the reservoir).

Return type:

float

refresh(mix_model=None)[source]

Re-embed the landmark reservoir (optionally under an updated model), warm-started from the current coordinates and rigidly aligned back onto them.

With an estimator configured and stream statistics pending, the model is re-estimated first (one incremental-EM M-step over reservoir + stream statistics) and the re-embed runs under the NEW model. An explicit mix_model argument always wins and discards the pending stream statistics – passing both a stream-updated posture and an external model would make the vintage of the statistics unaccountable.

Returns {"alignment_residual_rms", "alignment_scale", "atlas_spread", "n_landmarks", "model_updated", "n_stream_obs_consumed"}. A residual small relative to the spread means visual continuity is real; a large one means the embedding geometry genuinely changed and the report says so rather than hiding it in the alignment. Resets the drift accumulator (a refresh is the response to drift, so scoring restarts).

Parameters:

mix_model (Any)

Return type:

dict[str, Any]

place_in_atlas(p_rows, atlas, *, alpha=1.0, max_its=250, eta=None, momentum=0.8, tol=1.0e-7)[source]

Place each row’s point into a FROZEN atlas by minimizing its own row-KL under the t-kernel.

p_rows is (B, L) row-stochastic (each arriving point’s calibrated affinities over the L landmarks); atlas is (L, d). Each point’s objective involves only itself and the frozen landmarks, so the whole batch optimizes as one vectorized gradient descent. Initialized at the affinity-weighted barycenter of landmark coordinates.

Parameters:
Return type:

ndarray