mixle.utils.parallel.ray_data module

Ray-backed encoded-data handle for distributed sufficient-statistic folding (WS-C2).

RayEncodedData plugs Ray into mixle’s encoded-data backend registry (planner.encoded_data(..., backend="ray")). The data is encoded once, split into partitions, and each partition is placed in the Ray object store; the orchestrator-contract methods (pysp_seq_log_density_sum / pysp_seq_estimate / pysp_seq_initialize / pysp_stream_accumulate) map a Ray remote task over the partitions and reduce the per-partition sufficient statistics on the driver – the same map/fold the Spark and dask backends do, on a Ray cluster.

Ray is an optional dependency: this module is imported only when the "ray" backend is requested, so the rest of mixle (and CI without Ray installed) is unaffected.

class RayEncodedData(data, estimator=None, model=None, encoder=None, num_partitions=None, num_workers=None, address=None, **_)[source]

Bases: EncodedDataHandle

Encoded-data handle that folds sufficient statistics over Ray object-store partitions.

Parameters:

  • data (Any)

  • estimator (Any | None)

  • model (Any | None)

  • encoder (DataSequenceEncoder | None)

  • num_partitions (int | None)

  • num_workers (int | None)

  • address (str | None)

  • _ (Any)

pysp_seq_log_density_sum(estimate)[source]

Return (num_observations, summed_log_density) for estimate.

Parameters:

estimate (Any)

Return type:

tuple[float, float]

pysp_seq_estimate(estimator, prev_estimate)[source]

Run one distributed/local sufficient-statistic fold and M-step.

Parameters:

  • estimator (Any)

  • prev_estimate (Any)

Return type:

Any

pysp_seq_initialize(estimator, rng, p)[source]

Initialize a model through the handle’s resident encoded data.

Parameters:
Return type:

Any

pysp_stream_accumulate(estimator, model)[source]

Return folded sufficient statistics for streaming/incremental EM.

Parameters:
Return type:

tuple[float, Any]

property num_chunks: int
close()[source]

No-op: the Ray runtime is left running (shared across handles); the process tears it down.

Return type:

None