mixle.engines.heterogeneous module

Precision-aware planning for distributed EM across heterogeneous compute.

Large worker pools are rarely uniform: some workers may have GPU tensor cores, while others are CPU-only or accuracy-oriented. This module chooses, per worker, how many E-step rows to assign and which precision band to run. The selected precision is the fastest supported band that still satisfies the requested error budget. When NO available precision can satisfy that budget for some worker, planning fails closed by default – it raises InfeasiblePrecisionError quoting the quantified gap – rather than silently returning a plan that violates the accuracy it claims to hold; pass allow_infeasible=True for an explicit best-effort plan instead (see HeterogeneousPlan.is_feasible()).

Meeting a budget and certifying it are different claims, and this module makes only the first: meets_target says the chosen format’s gamma-n roundoff score clears the target at the plan’s op_count, whereas HeterogeneousPlan.is_certified() is False for every constrained plan because the planner has no operation graph, magnitude/conditioning evidence, reduction order, or communication-topology certificate with which to prove a scientific target. A caller wanting numbers ranked need not opt out of anything; a caller wanting a proof will not find one here.

The plan also sizes the 4-way (radix-4) reduction depth so fixed-size sufficient-statistic payloads fold in O(log W) instead of a single-root fan-in.

This module is the pure-Python planning layer. Spark, MPI, or other distributed dispatchers consume the returned plan from the inference layer.

exception InfeasiblePrecisionError[source]

Bases: ValueError

Raised by plan_heterogeneous() when some worker cannot satisfy target_rel_error at any allowed precision and the caller did not opt into a best-effort plan via allow_infeasible=True.

class Worker(name, device, precisions, base_throughput=1.0)[source]

Bases: object

A compute worker: its device, the precisions it can run (any order), and a base throughput.

Validated at construction so a malformed worker can never silently reach planning: device must be a known device, precisions must be nonempty, every declared precision must be a supported (device, precision) pair (present in _THROUGHPUT – planning has no throughput model for an unlisted pair and used to silently default to an arbitrary 0.5x instead of rejecting it), and base_throughput must be finite and strictly positive (a zero or negative throughput could previously divide-by-zero or invert the load balancer’s row split; see MXR-080-0134).

Parameters:
class WorkerAssignment(name, rows, precision, effective_throughput, meets_target=True, achieved_rel_error=None, evidence_kind='unconstrained')[source]

Bases: object

One worker’s row allocation, precision, and effective throughput.

meets_target is a statement about numbers: whether precision’s modeled roundoff at the plan’s op_count actually lands at or below the plan’s target_rel_error (see plan_heterogeneous()), quantified by achieved_rel_error (None when the format’s roundoff isn’t modeled, e.g. fp8 – which is exactly why such a format can never meet a target). Both default to the trivially-satisfied case so callers that never pass a target_rel_error see unconstrained, always-True assignments.

Whether the planner can certify that number is a separate question, answered by evidence_kind and is_certified: a "heuristic" assignment may legitimately report meets_target=True while remaining uncertified.

Parameters:

  • name (str)

  • rows (int)

  • precision (str)

  • effective_throughput (float)

  • meets_target (bool)

  • achieved_rel_error (float | None)

  • evidence_kind (str)

property is_certified: bool

Whether meets_target rests on an accuracy certificate rather than a ranking heuristic.

True only for evidence_kind='unconstrained', where there is no target to certify. For a "heuristic" assignment this is always False, including when meets_target is True: the gamma-n score ranks formats, but the planner holds no operation graph, magnitude/conditioning evidence, reduction order, or communication-topology certificate with which to prove a scientific target.

class HeterogeneousPlan(assignments, reduce_depth)[source]

Bases: object

Assignments and reduction depth for heterogeneous execution.

Parameters:

  • assignments (tuple[WorkerAssignment, ...])

  • reduce_depth (int)

total_rows()[source]

Return total rows assigned across workers.

Return type:

int

is_feasible()[source]

Whether every assignment’s precision actually meets the plan’s requested target_rel_error.

Always True for an unconstrained plan (target_rel_error=None). Can only be False when plan_heterogeneous was called with allow_infeasible=True and at least one worker’s best available precision still could not satisfy the budget – inspect infeasible_assignments() for which workers, and each one’s quantified achieved_rel_error. Without that opt-in the planner raises rather than returning an infeasible plan, so this is True for every plan a default call returns.

Feasible is not certified: see is_certified().

Return type:

bool

is_certified()[source]

Whether the plan’s accuracy claims are certificates rather than format-ranking scores.

True only for an unconstrained plan (target_rel_error=None), which claims nothing about accuracy in the first place. For any constrained plan this is False even when is_feasible() is True: the gamma-n score behind meets_target can rank formats and rule options out, but the planner has no operation graph, magnitude/conditioning evidence, reduction order, or communication-topology certificate with which to prove a scientific target. Supply a workload error certificate elsewhere if the target must be guaranteed.

Return type:

bool

infeasible_assignments()[source]

The subset of assignments whose precision does not meet the plan’s target_rel_error.

Return type:

tuple[WorkerAssignment, …]

plan_heterogeneous(workers, n_rows, allowed_precisions=('fp8', 'bfloat16', 'float16', 'float32', 'float64', 'dd'), target_rel_error=None, op_count=1000, allow_infeasible=False)[source]

Assign rows + a precision band to each worker, balanced by precision-adjusted throughput.

Each worker runs the fastest precision its hardware supports that stays within target_rel_error (None = no accuracy constraint); rows are split proportionally to the resulting throughput so all workers finish together. reduce_depth is the 4-way (radix-4) tree depth for folding the sufficient-statistic payloads (ceil(log4(W)) == ceil(log2(W)/2)), avoiding the single-root fan-in.

Fails closed by default: if target_rel_error is not None and some worker cannot satisfy it at any allowed precision, raises InfeasiblePrecisionError identifying the worker(s) and the quantified gap, instead of silently returning a plan that violates the requested accuracy (MXR-080-0133). Pass allow_infeasible=True to instead receive a best-effort HeterogeneousPlan whose is_feasible() is False and whose offending assignments carry meets_target=False plus a quantified achieved_rel_error.

A target the modeled roundoff DOES clear needs no opt-in: the plan is returned normally, with meets_target=True and is_feasible() – factual statements about the format’s gamma-n score at op_count. What such a plan is not is certified: is_certified() is False for every constrained plan, and evidence_kind stays "heuristic", because ranking formats is not proving a workload’s error. Requiring allow_infeasible=True for these callers was the old behavior; it claimed a target was “not achievable” while quoting an achieved error orders of magnitude below it.

Raises ValueError for malformed input: no workers, a non-int or negative n_rows/op_count, an empty allowed_precisions, or a target_rel_error that isn’t None or a finite positive number (MXR-080-0134). Each Worker validates its own device/precisions/throughput at construction, which is what keeps the row-split arithmetic below free of division-by-zero and negative-row bugs.

Parameters:

  • workers (list[Worker])

  • n_rows (int)

  • allowed_precisions (tuple[str, ...])

  • target_rel_error (float | None)

  • op_count (int)

  • allow_infeasible (bool)

Return type:

HeterogeneousPlan