Test Tiers

The test suite is organized into tiers by purpose and time budget, so a contributor can run the right subset for the moment and CI can gate each tier at a known cost. The named tiers are derived centrally during collection from the reviewed domain-marker registry. Files do not self-assign execution tiers. The legacy fast alias remains available for local compatibility, while hosted gates use the purpose-named tiers below.

Each tier is a pytest marker declared in pyproject.toml. The latency-bounded local smoke command is python scripts/run_smoke.py. Do not use a broad pytest -m smoke invocation as latency evidence: pytest imports every discovered test module before marker deselection.

Tier	Purpose	Budget	Status
smoke	Import, public-API, and critical-fit-path checks – “is the package fundamentally working?”	<= 30 s local, <= 60 s in CI	Enforced now – 4 tests in an explicit one-file manifest.
core	Stable base-install correctness (no optional extras).	<= 25 min per Python CI job	Populated centrally: quick, non-experimental, non-stochastic tests with no optional-backend marker.
full	All non-optional correctness, including the slow stochastic, integration, and exhaustive tests.	<= 90 min	Populated centrally: every non-optional, non-benchmark correctness test.
optional	Tests that require optional extras or external executables; one job per installed backend group.	per backend group	Usable now. Unlike the other tiers, the collected count depends on which extras are installed, so no single number describes it: this tree collects roughly 1100 with the torch extra present and roughly 340 without it.
numerical	Repeated-seed and numerical-stress tests.	<= 30 min; scheduled/manual, not a per-commit gate	Populated centrally from the stochastic purpose marker.
benchmark	Timing-oriented performance tests. Never mixed with correctness assertions except explicit parity gates.	performance only	Usable now – 7 tests collected.
hardware	Real MPI / multi-process receipts.	<= 20 min; scheduled or manually gated	Populated centrally from MPI and torchrun tests. No GPU result is inferred from this CPU-hosted lane.

Current Status

All named tiers are populated. smoke remains an explicit four-test, collection-light manifest. core and full are derived from the central triage registry, optional includes every optional-backend test, numerical includes stochastic stress tests, and hardware contains the MPI/torchrun multi-process cases. benchmark remains timing-only. The smoke surface is mixle/tests/smoke-manifest.txt and is run without broad collection by scripts/run_smoke.py.

scripts/run_test_tier.py owns hosted marker selection, enforces the documented hard subprocess deadline, binds the command and duration to the full candidate commit, and emits a retained JSON receipt. Job-level deadlines also bound setup or teardown outside pytest. A tier that times out fails; its budget cannot silently become a target.

Guidance

• smoke must stay genuinely fast and dependency-light: it runs on a base install with no optional extras, and its job is to fail loudly and quickly when something is fundamentally broken (an import cycle, a broken public entry point, a critical fit path that no longer converges). Its runner rejects directories, globs, repository escapes, duplicate entries, and budgets above 30 seconds.

• numerical and benchmark tests are deliberately kept out of the per-commit gates. A stochastic assertion that fails one run in fifty does not belong in the signal a contributor reads on every push; it belongs in a nightly tier where a failure is investigated deliberately.

• hardware tests produce real MPI and multi-process receipts. GPU validation requires a separately identified GPU runner and is not inferred from this CPU-hosted tier.

• A test may carry more than one marker (for example smoke and serialization). The tier answers when it runs; the domain markers answer what it covers.

Every executable tier has both a hard subprocess budget and an enclosing hosted job deadline. Receipts are retained as workflow artifacts.