mixle.task.active module

Active labeling – spend the teacher’s expensive labels only where they buy the most, not at random.

Labeling is the dominant cost of building a task model: every label is a frontier call or a human minute. This is experimental design applied to the labeling decision (the discrete-pool analogue of mixle.doe active learning): label a small seed, fit a student, then repeatedly query the teacher only for the pool examples the student is most unsure about (and, optionally, most novel), refit, and continue until the budget runs out. Scope of the efficiency claim (STAT-RR22-05): whether ranked labeling reaches a given student quality with fewer labels than random – and by how much – depends on the pool, the model family, and the acquisition; the repository’s own paired evidence is bounded and synthetic (a per-seed win/tie/loss record with an exact sign test in examples/label_economics_demo.py, and a same-budget paired comparison in examples/task_llm_active_example.py). Measure on your own task before booking savings; the per-round labels-vs-agreement log below exists for exactly that.

Acquisitions score the student’s own predictions (uncertainty as a ranking, which needs no calibrated probability) and can blend in the generative density (mixle.task.density.DensityGate) for diversity:

  • margin – smallest gap between the top two class scores (the classic, robust default);

  • entropy – highest predictive entropy;

  • least_confidence – lowest top-class score;

  • random – the baseline this is meant to beat.

active_distill returns the student plus a per-round log of labels-spent vs. agreement, so the labeling efficiency is auditable.

acquisition_scores(student, texts, method='margin')[source]

Informativeness of each unlabeled text under the student (higher = more worth labeling).

Parameters:
Return type:

ndarray

class ActiveResult(model, labels_used, training_labels_used=0, validation_labels_used=0, teacher_queries=0, history=<factory>, labeled_texts=<factory>, labeled_labels=<factory>)[source]

Bases: object

The actively-distilled student plus an audit trail of labels spent vs. quality reached each round.

Parameters:
active_distill(teacher, pool, *, budget, seed_size=20, rounds=5, acquisition='margin', labels=None, recipe=None, val_texts=None, seed=0)[source]

Distill from pool under a labeling budget, querying the teacher only for the most informative items.

Labels a seed_size random seed, then over rounds adds the top-scoring unlabeled examples (by acquisition) until budget labels are spent, refitting the student each round. If val_texts is given, the teacher labels it once and each round’s agreement on it is logged.

Parameters:
Return type:

ActiveResult