mixle.task.bandit module

Multi-armed bandits: pick-an-arm / observe-a-reward loops with posterior policies.

Where this sits in mixle: mixle.doe’s Bayesian-optimization acquisitions (UCB and Thompson draws over a surrogate posterior) are the continuous-design cousins; this module is the small DISCRETE-arm loop for serving-time decisions – which teacher to query, which prompt variant, which data source, which model tier answers when calibrated confidence (mixle.task.router.Router) is unavailable or costs drift and the choice must be LEARNED from observed outcomes instead.

Policies (shared surface: select() -> arm, update(arm, reward), batch_update(arms, rewards), pulls, means):

  • ThompsonBernoulli – Beta-Bernoulli conjugate Thompson sampling; rewards in [0, 1] (clicks, agreement flags, pass/fail, or any fractional credit).

  • ThompsonGaussian – Normal-Inverse-Gamma conjugate Thompson sampling; unbounded real rewards with unknown variance (latencies, margins, log-likelihood gains).

  • UCB1 – the deterministic optimism baseline (Auer et al.); no randomness at all, useful when reproducible selection matters more than Bayesian credit assignment.

  • EstimatorBandit – Thompson sampling with ARBITRARY mixle reward models via the online bootstrap (Poisson(1) replicate weights, Eckles & Kaptein): any estimator/accumulator pair – Gamma service times, categorical outcomes scored by a utility, mixtures for multi-modal rewards – becomes an arm with NO conjugate math, riding the same accumulator machinery the rest of mixle estimates with.

Every policy owns a seeded numpy.random.RandomState and is deterministic given it. Conjugate and UCB1 updates commute, so batch_update (delayed/batched feedback, the streaming pattern used across mixle) is exactly the sequential replay of its pairs.

class EstimatorBandit(estimators, *, n_boot=32, mean_fn=None, mc_draws=64, seed=None)[source]

Bases: _BanditBase

Thompson sampling for ARBITRARY mixle reward models, via the online bootstrap.

Each arm keeps n_boot accumulator replicates of its estimator; update adds the reward to every replicate with an independent Poisson(1) weight (Eckles & Kaptein’s online bootstrap), so the replicate ensemble approximates the sampling distribution of the fitted reward model with no conjugate structure required. select plays each arm once, then draws one non-empty replicate per arm, fits it (estimator.estimate), scores it with mean_fn (default: Monte-Carlo mean of estimate.sampler(...).sample(mc_draws)), and plays the argmax – posterior-sample-then-maximize, exactly Thompson’s rule with a bootstrap posterior.

estimators is one mixle ParameterEstimator per arm (Gamma for waiting times, Gaussian for margins, a mixture for multi-modal rewards – anything with the accumulator contract).

Parameters:

  • n_boot (int)

  • mc_draws (int)

  • seed (int | None)

class ThompsonBernoulli(n_arms, *, alpha=1.0, beta=1.0, seed=None)[source]

Bases: _BanditBase

Beta-Bernoulli Thompson sampling. Rewards live in [0, 1]; fractional rewards contribute fractional pseudo-counts (the standard Bernoulli-moment update).

Parameters:
class ThompsonGaussian(n_arms, *, mu0=0.0, kappa0=1.0e-2, alpha0=0.5, beta0=0.5, seed=None)[source]

Bases: _BanditBase

Normal-Inverse-Gamma Thompson sampling: unknown mean AND variance per arm, so early optimism comes from honest posterior width rather than a tuned exploration constant.

Parameters:
class UCB1(n_arms, *, c=1.0, seed=None)[source]

Bases: _BanditBase

The deterministic optimism baseline: play each arm once, then argmax mean_k + c * sqrt(2 ln t / n_k). Ties break to the lowest index; with no randomness anywhere, two UCB1 runs on the same reward sequence are identical.

Parameters: