mixle.task.collapse module

collapse_monitor – the shared collapse-detection utility for self-improvement loops.

Every self-improvement round claims to be getting better. This shared check evaluates that claim without each loop reimplementing collapse detection: across rounds, the held-out verified score must be non-decreasing, and the proposal diversity must not be shrinking. A loop that improves its score by collapsing onto a few candidates is overfitting to the verifier, not genuinely improving.

verdict = collapse_monitor(history) verdict.ok # True iff enough evidence exists and both checks hold verdict.status # “ok”, “collapse_detected”, or “insufficient_evidence” verdict.reason # None, or the specific failure/insufficiency reason

history is one entry per round: a dict with the round’s held-out verified score and its pool of candidates, or a precomputed diversity number.

class CollapseVerdict(ok, reason, status='ok', scores=<factory>, diversities=<factory>, failed_round=None)

Bases: object

The result of collapse_monitor(): ok plus which check failed, and the raw series.

Parameters:

• ok (bool)

• reason (str | None)

• status (str)

• scores (list[float])

• diversities (list[float])

• failed_round (int | None)

collapse_monitor(history, *, score_key='score', candidates_key='candidates', diversity_fn=distinct_count_diversity, score_tol=0.0, diversity_tol=0.0)

Check a self-improvement round history for collapse: score non-decreasing and diversity not shrinking.

Each entry of history supplies the round’s held-out verified score under score_key and either its candidate pool under candidates_key (diversity computed via diversity_fn) or, when candidates_key is absent, a precomputed diversity number directly under "diversity". score_tol/diversity_tol allow a small, explicitly-named amount of round-to-round noise before a decrease/shrink counts as a real regression (0.0 = strict non-decreasing). At least two rounds are required to make a comparison. Shorter histories return status="insufficient_evidence" and ok=False rather than certifying success. Non-finite evidence is rejected.

Parameters:

• history (Sequence[Mapping[str, Any]])

• score_key (str)

• candidates_key (str)

• diversity_fn (Callable[[Sequence[Any]], float])

• score_tol (float)

• diversity_tol (float)

Return type:

CollapseVerdict

distinct_count_diversity(candidates)

Diversity proxy: the number of distinct candidates (by str identity) in the round’s pool.

Parameters:

candidates (Sequence[Any])

Return type:

float

entropy_diversity(candidates)

Diversity proxy: Shannon entropy (nats) of the candidate-frequency distribution in the round’s pool.

Parameters:

candidates (Sequence[Any])

Return type:

float