mixle.utils.hvis.distributed module

Distributed / out-of-core construction of the direct layout: map-reduce over data shards.

Everything mixle.utils.hvis.direct.model_map() learns from data is ADDITIVE over shards – posterior masses and overlaps (sum z, sqrt(z)^T sqrt(z), z^T z, triple products for the nerve), per-(field, component) weighted moments behind the whiteners, and per-component weighted moments of the fiber coordinates behind the chart PCA. The one order statistic (the occlusion radius, a 90th percentile) is a cheap gather of one float per dominated point. So the whole layout is a handful of map-reduce passes with EXACT combines – the same shape as mixle’s estimator/accumulator machinery (seq_estimate’s Spark path, the multiprocessing/MPI encoded- data handles), just for the visualization geometry instead of the model parameters:

  1. fiber_stats() per shard, combined with + -> whiteners, vertices, nerve;

  2. score_stats() per shard, combined with + -> chart loadings, spreads, scale (quadratic charts whose base dimension exceeds the lift cap need this pass twice: once for the pre-PCA, once for the lifted moments);

  3. radius_stats() per shard (only when occlusion must resolve overlaps) -> radii percentile;

  4. place() per shard -> coordinates (closed-form, embarrassingly parallel).

distributed_model_map() orchestrates the passes over an in-memory list of shards through any map-shaped mapper (builtin map, multiprocessing.Pool.map, an MPI executor map, …); every stage function is a pure picklable callable and every stats object combines with +, so on Spark the same sequence is rdd.mapPartitions(lambda it: [fiber_stats(model, list(it))]).reduce( lambda a, b: a + b) per pass with the small plan objects broadcast between passes, and coordinates come from broadcasting the finished (geometry-only) map and calling .place per partition.

What does NOT distribute: the t-SNE/UMAP refine pass and embedding goals – global sequential optimizations by construction. At scale, run those on a driver-sized subsample and place the rest (mixle.utils.hvis.stream is the incremental version of that recipe). Model FITTING is the already-solved distributed problem (seq_estimate); this module assumes mix_model arrives fitted and requires it explicitly.

The contract with the single-machine path is equality: distributed_model_map(chunks, model) reproduces model_map(concat(chunks), model) to floating-point summation order (hvis_distributed_test pins it), for any chunking.

class FiberStats(n, mass, sqrt_overlap, co, fields, triple=None)[source]

Bases: object

Pass-1 additive statistics: everything the layout’s GLOBAL geometry needs from data.

Parameters:
class ScoreStats(sw, su, suu, sf, sff, swc, sfc, sffc)[source]

Bases: object

Pass-2 additive statistics: per-component moments of the whitened fiber coordinates u and (when computable in the same pass) of the chart features, plus the clamped-weight moments behind the spread normalization.

Parameters:
distributed_model_map(chunks, mix_model, emb_dim=2, *, spread=0.35, chart='linear', occlusion=True, occlusion_margin=1.05, edge_threshold=0.02, field_weights=None, mapper=None, with_points=True)[source]

model_map() over sharded data (see the module docstring).

chunks is a materialized sequence of data shards (each itself a sequence); every pass maps over it once. mapper is any map-shaped callable – builtin map (default), multiprocessing.Pool.map, an MPI executor’s map – the stage functions and their broadcast plans are picklable. mix_model is required: fitting is the estimator machinery’s distributed problem (seq_estimate), not this one. with_points=False skips the coordinate pass and returns geometry only (empty coords/responsibilities) – broadcast the result and call .place per shard where the data lives. refine and goals are deliberately absent: global sequential optimization does not shard (subsample-and-place instead).

Parameters:
Return type:

ModelMap

fiber_stats(mix_model, chunk, *, nerve_triple=False)[source]

Pass 1 over one shard. Combine shard results with +; order never matters.

nerve_triple=True also accumulates the (K, K, K) triple-overlap tensor so fuzzy_nerve_from_stats() can report triangles (an extra factor-K of work in this pass).

Parameters:

nerve_triple (bool)

Return type:

FiberStats

fuzzy_nerve_from_stats(stats, *, edge_threshold=0.02, triangle_threshold=0.02)[source]

mixle.utils.hvis.topology.fuzzy_nerve() from combined pass-1 statistics – the distributed nerve. Triangles need fiber_stats(..., nerve_triple=True); without the tensor only the 1-skeleton is reported (and nerve_report() would over-count holes, so it raises instead).

Parameters:

  • stats (FiberStats)

  • edge_threshold (float)

  • triangle_threshold (float)

Return type:

dict

radius_stats(plan, chunk)[source]

Pass 3 over one shard: per component, the scaled score norms of the points it dominates.

Parameters:

plan (_ScorePlan)

Return type:

list

score_stats(plan, chunk)[source]

Pass 2 over one shard (pure; combine with +).

Parameters:

plan (_ScorePlan)

Return type:

ScoreStats