Benchmark Methodology

The rules a mixle benchmark must follow so a timing measures what its prose claims (worklist B7.2). A speed number that mixes phases, hides a warm-up, or reports alongside a parity warning is worse than no number: it invites a wrong comparison. These rules make the harness’s timings honest; the harness itself is tracked in benchmarks/ (worklist B7.1) and the absolute import/smoke budgets are gated in CI (worklist B7.4).

Separate the phases

A “fit” number that silently includes encoding, or a “score” number that includes a one-time JIT compile, is a category error. Measure and report each phase on its own:

  • initialization – building the estimator/model object and any prior/seed setup;

  • encode – dist_to_encoder().seq_encode(data) (the raw-to-vectorized conversion, paid once);

  • fit – the EM / gradient / closed-form loop over already-encoded data;

  • score – seq_log_density over encoded data;

  • compilation – the first-call cost of a compiled backend (Numba/JAX), which is amortized, not per-call.

A benchmark states which phase(s) its headline number covers, and never folds compilation into a per-call figure.

Warm vs cold

Compiled kernels have a large first-call cost and a small steady-state cost. Report warm and cold as separate panels, never averaged together:

  • cold – first call, includes compilation; the number a user pays once per process.

  • warm – steady state after the kernel is compiled; the number a user pays per batch thereafter.

Averaging one cold call into many warm calls produces a figure that describes neither.

Parity is a hard gate, not a footnote

Before a timing counts, the benchmarked fit must pass a likelihood/parameter parity check against a reference (the serial path, or SciPy / scikit-learn / hmmlearn where applicable) – as an assertion that fails the benchmark, not a printed warning that scrolls by. A fast number for a wrong answer is not a result. Speed is never bought by weakening correctness or parity (worklist B7.4).

Record what actually happened

Every benchmarked fit records, alongside the timing:

  • the actual iterations run and the convergence status (converged / hit max_its / diverged);

  • any failure (a non-finite objective, a raised exception) – reported, never silently dropped from the aggregate;

  • the environment (see External Reproduction) so a number is attributable to a machine and version.

A timing without its iteration count and convergence status is unreproducible and does not belong in a release claim.