Scale-Out Economics: When a Backend Helps

Switching optimize(..., backend=...) from local to a distributed backend (Spark, Dask, Ray, MPI) is not automatically faster. This page states, from measurement, when it is expected to help – and when it is not.

How a distributed fit spends its time

Each EM iteration on a distributed backend does three things:

  1. Compute – every worker folds its data shard into sufficient statistics. This is O(N / workers) per iteration.

  2. Reduce – each worker’s sufficient-statistic payload, exactly pickle.dumps((count, accumulator.value())), is reduced to the root (a gather-and-fold or an O(log W) reduce tree, backend-dependent) and combined. See mixle/utils/parallel/mpi.py.

  3. Broadcast – the root re-estimates the model and sends it back to every worker.

The reduce and broadcast are fixed overhead per iteration; only the compute shrinks as workers are added.

The payload is O(model), not O(data)

The tested invariant is that the sufficient-statistic payload does not grow materially with the dataset size. It is a function of the model, not N. mixle/tests/reduction_payload_telemetry_test.py checks that property for a single Gaussian and a Gaussian mixture, checks that the larger model has the larger payload, and verifies serialization round trips. Exact byte counts and latencies are environment-dependent telemetry; cite them only from a candidate-bound benchmark receipt.

When distribution is expected to help

Because the reduce payload is tiny and fixed while compute scales with N, distribution pays off when per-shard compute time dominates the fixed reduce + fold + broadcast overhead. Concretely:

  • Helps: large N and non-trivial per-row work – a rich model (large mixtures, HMMs, neural leaves), an expensive encoder, or per-row scoring that is not memory-bandwidth-bound. Here more workers cut the dominant compute term while the byte-sized payload is negligible.

  • Does not help: small-to-moderate N with a cheap model (for example a plain Gaussian or a small mixture). The per-row work is so cheap that the local fit finishes before a distributed backend has paid its serialization and scheduling overhead. A small reduction payload buys nothing if the whole local fit already finishes quickly.

  • Backs up the wrong way: a network with high per-message latency penalizes the per-iteration reduce/broadcast on every EM step; iteration count matters, not just data size.

Rule of thumb: distribute when the single-worker fit is compute-bound and takes long enough that cutting compute by the worker count outweighs fixed per-iteration serialization and backend scheduling overhead. If a local fit already finishes quickly, prefer local execution.

Measuring for your own model

The payload and its serialization cost for any estimator/model pair can be measured the same way the telemetry test does: build the accumulator, fold a shard, and size pickle.dumps((count, accumulator.value())). If that payload is small and flat in N (it will be for the classical families), the reduce cost is not your bottleneck, and the decision comes down to per-shard compute versus backend scheduling latency.