mixle.inference.select module

Verifier-based selection — the generic test-time-compute selector.

Pure orchestration over a user score function: score every candidate, return the best (the largest score, or the smallest when lower_is_better). This is the “best-of-N” / verifier pattern that test-time-compute stacks lean on — generate several candidates, score each with a verifier, keep the winner — with no assumption about what a candidate is (a string, a model, a plan, a sample).

When conformal_alpha is given, the result also carries a confident flag: whether the winner’s lead over the runner-up clears a calibration band derived from the spread of the candidate scores (a simple conformal/bootstrap SE band). A clear winner is confident=True; a near-tie is False.

class SelectionResult(best, best_index, scores, confident=None, margin=None, band=None, _extras=<factory>)[source]

Bases: object

Result of a select_best() call.

Parameters:
best

the winning candidate (the actual object, not its index).

Type:

Any

best_index

the position of the winner in the input candidates.

Type:

int

scores

per-candidate scores in input order (a numpy float array).

Type:

numpy.ndarray

confident

whether the winner’s lead clears the conformal band — None when no conformal_alpha was supplied.

Type:

bool | None

margin

the winner’s lead over the runner-up (in score units, always >= 0); None when there is a single candidate.

Type:

float | None

band

the conformal/bootstrap band the margin was compared against — None when no conformal_alpha was supplied or there is a single candidate.

Type:

float | None

best: Any
best_index: int
scores: ndarray
confident: bool | None = None
margin: float | None = None
band: float | None = None
select_best(candidates, *, score, lower_is_better=False, conformal_alpha=None)[source]

Score each candidate and return the best, the verifier-based test-time-compute selector.

Parameters:

  • candidates (Any) – an iterable of candidate objects (anything score accepts).

  • score (Callable[[Any], float]) – a verifier score(candidate) -> float; the winner maximizes it (or minimizes it when lower_is_better).

  • lower_is_better (bool) – if True, the winner is the candidate with the smallest score.

  • conformal_alpha (float | None) – optional miscoverage level in (0, 1). When given, the result’s confident flag reports whether the winner’s lead over the runner-up exceeds a conformal/bootstrap band at confidence 1 - conformal_alpha (a z-scaled estimate of the score spread). None (default) leaves confident unset.

Returns:

A SelectionResult. It is also subscriptable (result["best"]), so callers may treat it as a small dict with keys best, best_index, scores, confident.

Raises:

ValueError – if candidates is empty, or conformal_alpha is outside (0, 1).

Return type:

SelectionResult