mixle.inference.select module

Verifier-based selection — the generic test-time-compute selector.

Pure orchestration over a user score function: score every candidate, return the best (the largest score, or the smallest when lower_is_better). This is the “best-of-N” / verifier pattern that test-time-compute stacks lean on — generate several candidates, score each with a verifier, keep the winner — with no assumption about what a candidate is (a string, a model, a plan, a sample).

When heuristic_alpha is given, the result also carries a confident flag: whether the winner’s lead over the runner-up clears a normal score-spread heuristic. This is a ranking diagnostic, not a conformal or bootstrap coverage guarantee. The former conformal_alpha spelling remains as a deprecated compatibility alias and produces the same explicitly heuristic result.

class SelectionResult(best, best_index, scores, confident=None, margin=None, band=None, confidence_method=None, coverage_guarantee=False, _extras=<factory>)[source]

Bases: object

Result of a select_best() call.

Parameters:
best

the winning candidate (the actual object, not its index).

Type:

Any

best_index

the position of the winner in the input candidates.

Type:

int

scores

per-candidate scores in input order (a numpy float array).

Type:

numpy.ndarray

confident

whether the winner’s lead clears the optional score-spread heuristic — None when neither heuristic_alpha nor its deprecated alias was supplied.

Type:

bool | None

margin

the winner’s lead over the runner-up (in score units, always >= 0); None when there is a single candidate.

Type:

float | None

band

the heuristic band the margin was compared against — None when no threshold was requested or there is a single candidate.

Type:

float | None

confidence_method

the executed diagnostic, "normal_score_spread_heuristic", or None.

Type:

str | None

coverage_guarantee

always False; one verifier score per selected candidate cannot establish held-out conformal or bootstrap coverage.

Type:

bool

select_best(candidates, *, score, lower_is_better=False, conformal_alpha=None, heuristic_alpha=None)[source]

Score each candidate and return the best, the verifier-based test-time-compute selector.

Parameters:

  • candidates (Any) – an iterable of candidate objects (anything score accepts).

  • score (Callable[[Any], float]) – a verifier score(candidate) -> float; the winner maximizes it (or minimizes it when lower_is_better).

  • lower_is_better (bool) – if True, the winner is the candidate with the smallest score.

  • conformal_alpha (float | None) – deprecated compatibility alias for heuristic_alpha. It does not provide conformal coverage because candidate scores are neither held-out conformity scores nor exchangeable repeated measurements.

  • heuristic_alpha (float | None) – optional normal score-spread threshold level in (0, 1). When given, confident is a heuristic ranking flag only; coverage_guarantee remains False. With exactly TWO candidates the heuristic is undefined – margin and spread are the same number up to sqrt(2), so the flag would be decided by alpha alone (never at 0.05, always at 0.20, whatever the scores) – and confident is returned False with confidence_method naming the degeneracy.

Returns:

A SelectionResult. It is also subscriptable (result["best"]), so callers may treat it as a small dict with keys best, best_index, scores, confident.

Raises:

ValueError – if candidates is empty, or conformal_alpha is outside (0, 1).

Return type:

SelectionResult