mixle.data.sources.spark_source module

Spark-backed sampling helpers for sequence-encodable Mixle distributions.

The functions in this module broadcast distribution objects to Spark workers, draw reproducible partition-level samples, and return RDDs that can feed larger distributed estimation workflows.

take_sample(rdd, with_replacement, n, seed=None)

Take a deterministic-order random sample from a Spark RDD.

Raises ValueError if with_replacement is False and rdd has fewer than n elements: Spark’s own RDD.takeSample(False, n, ...) silently caps the result at the RDD’s true cardinality in that case (returning every element instead of the requested count) rather than raising, so this checks the returned length itself and fails loudly with the shortfall instead of silently handing the caller fewer records than they asked for (MXR-080-0066). with_replacement sampling always returns exactly n elements and is unaffected.

Parameters:

• rdd (Any)

• with_replacement (bool)

• n (int)

• seed (int | None)

sample_seq_as_rdd(sc, dist, seq_len, count_per_split, num_splits, seed=None)

Sample fixed-length sequences from a distribution into a Spark RDD.

sample_rdd(sc, dist, count_per_split, num_splits, seed=None)

Sample independent draws from a distribution into a Spark RDD.