mixle.inference.spark_executor module

Spark transport for distributed heterogeneous EM: RDD.treeReduce over the verified combine-tree.

The same sharded-E-step + k-way tree-reduce algorithm as mixle.inference.heterogeneous_executor, run on a Spark cluster: shards become an RDD, each is scored to a fixed-size (count, sufficient-stat) payload by map, and those fold with RDD.treeReduce – the reduction happens IN Spark across O(log W) levels, never a single-root collect to the driver (the OOM fan-in the scaling audit flagged). treeReduce’s combiner runs on freshly-deserialized payloads, so the in-place combine() is safe (the HMM-stat aliasing hazard does not bite).

spark_em_step(sc, estimator, model, data, n_shards=8, depth=2)

One EM step on Spark: parallelize shards, map the E-step, treeReduce the combine, estimate.

Parameters:

• sc (Any)

• estimator (Any)

• model (Any)

• data (Any)

• n_shards (int)

• depth (int)

Return type:

Any

spark_fit(sc, model, data, max_its=10, n_shards=8, depth=2)

Run max_its EM iterations on Spark; the shard RDD is cached once and re-scored each iteration.

Parameters:

• sc (Any)

• model (Any)

• data (Any)

• max_its (int)

• n_shards (int)

• depth (int)

Return type:

Any