mixle.doe.bayesopt module

Gaussian-process Bayesian optimization over a bounded input space (WS-E).

A sequential model-based optimization loop: fit a GP surrogate to the observed points, score Latin-hypercube candidates with an acquisition function, and evaluate the best candidate next. Reuses mixle.models.gaussian_process.GaussianProcessRegressor (torch) as the surrogate; the acquisition functions themselves are torch-free numpy.

Acquisitions are looked up through a small registry (register_acquisition / acq= name) – the same “register, don’t branch” pattern as the engines and encoded-data backends – so a new acquisition plugs in without editing the proposal loop. Built in: expected improvement ("ei"), probability of improvement ("pi"), and the upper/lower confidence bound ("ucb"). Each takes (mean, std, best, *, maximize, **params) and returns a merit array that is maximized over the candidate set. Batch proposals (propose_batch) use the kriging-believer heuristic: fantasize the posterior mean at each pick, refit, and repeat, giving a spatially diverse batch.

class Acquisition(*args, **kwargs)[source]

Bases: Protocol

The acquisition-function contract registered via register_acquisition (EI/PI/UCB).

An acquisition scores candidate points from their surrogate posterior moments and returns a merit array that the proposal loop maximizes over the candidate set. It is called as fn(mean, std, best, *, maximize, **params) where mean / std are the predictive mean and standard deviation at the candidates, best is the incumbent objective value, maximize selects the optimization sense, and **params carries per-acquisition knobs (e.g. xi for EI/PI, kappa for UCB). Built-ins mixle.doe.bayesopt.expected_improvement(), probability_of_improvement(), and upper_confidence_bound() satisfy this contract.

class Surrogate(*args, **kwargs)[source]

Bases: Protocol

The GP-surrogate contract passed as gp= to the Bayesian-optimization loops.

A surrogate is fit to the observed (x, y) and queried for the posterior predictive moments at new candidate points. The call convention is the one used by mixle.models.gaussian_process.GaussianProcessRegressor: fit trains in place (it may return diagnostics, which the loops ignore), and predict takes the training data alongside the query points, returning the posterior mean (return_cov=False) or (mean, cov) pair (return_cov=True).

fit(x, y, **kwargs)[source]

Fit or update the surrogate from observed design points x and responses y.

Parameters:
Return type:

Any

predict(x_train, y_train, x_new, return_cov=...)[source]

Return predictive moments at x_new using the observed training data.

Parameters:
Return type:

Any

expected_improvement(mean, std, best, xi=0.0, *, maximize=False, **_)[source]

Return the expected-improvement acquisition at points with surrogate mean and std.

For minimization the improvement over the incumbent best is best - mean - xi; for maximization it is mean - best - xi. xi >= 0 trades exploration for exploitation. Points with (near-)zero predictive std get the DETERMINISTIC LIMIT max(improve, 0) – a point-mass posterior has no uncertainty to average over, so the “expected” improvement is exactly the guaranteed one, not 0. Higher is better (maximized over candidates).

Raises ValueError if std is negative/non-finite, mean/std shapes disagree, mean/best are non-finite, or xi is negative/non-finite.

Parameters:
Return type:

ndarray

log_expected_improvement(mean, std, best, xi=0.0, *, maximize=False, **_)[source]

Return the log expected-improvement acquisition – a numerically stable EI (Ament et al. 2023).

Mathematically log(EI), but computed so it stays finite and informative deep in the no-improvement tail where EI itself underflows to 0 (and log EI to -inf), keeping the optimizer’s ordering and gradients usable. Same argmax as expected_improvement(); points with (near-)zero predictive std get the deterministic limit log(max(improve, 0)) – -inf when the point-mass outcome does not improve on best, and the log of the guaranteed improvement when it does (see expected_improvement()). Higher is better. The z >= 0 branch is the direct well-conditioned form; the z < 0 tail uses the scaled complementary error function erfcx (Phi(z)/phi(z) = sqrt(pi/2) erfcx(-z/sqrt2)), which is bounded there and so never under/overflows.

Raises ValueError if std is negative/non-finite, mean/std shapes disagree, mean/best are non-finite, or xi is negative/non-finite.

Parameters:
Return type:

ndarray

probability_of_improvement(mean, std, best, *, maximize=False, xi=0.0, **_)[source]

Return the probability-of-improvement acquisition.

The probability that a candidate improves on the incumbent best by at least xi: P(f < best - xi) for minimization, P(f > best + xi) for maximization. Where the predictive std is zero the improvement is deterministic (1.0 if it improves, else 0.0). A NEGATIVE std is rejected outright (see _validate_acquisition_inputs()) rather than treated as this same deterministic case – it is an invalid input, not a zero-variance one. Higher is better.

Raises ValueError if std is negative/non-finite, mean/std shapes disagree, mean/best are non-finite, or xi is negative/non-finite.

Parameters:
Return type:

ndarray

upper_confidence_bound(mean, std, best=0.0, *, maximize=False, kappa=1.96, **_)[source]

Return the confidence-bound acquisition (UCB for maximization, LCB for minimization).

Returns a merit that is maximized over candidates: the optimistic bound mean + kappa * std when maximizing, and kappa * std - mean when minimizing (so picking the largest merit selects the most promising low-objective point). kappa >= 0 trades exploration for exploitation; best is ignored. Higher is better.

Raises ValueError if std is negative/non-finite, mean/std shapes disagree, mean/best are non-finite, or kappa is negative/non-finite.

Parameters:
Return type:

ndarray

thompson_sampling(mean, std, best=0.0, *, maximize=False, rng=None, **_)[source]

Thompson-sampling acquisition: one marginal posterior draw N(mean, std) per candidate.

Returns a merit (maximized over candidates) equal to the drawn value when maximizing and its negation when minimizing, so the selected point is the optimum of the sampled objective. A randomized, exploration-aware acquisition – repeated proposals explore competing optima in proportion to posterior probability, with no exploration knob to tune. This is the low-cost marginal variant (independent per-candidate draws, ignoring the GP’s cross-candidate correlation). The proposal loop (_propose_one(), and so propose_next() / propose_batch() / BayesianOptimizer) automatically threads its own candidate-generation rng through to this acquisition, so a seeded caller gets reproducible draws with no extra wiring; pass rng explicitly via acq_kwargs only to override that with a different generator. Called directly with no rng at all, this draws from a fresh, unseeded generator (matching _as_rng’s own None convention) – reproducible only through the proposal loop’s threading above. best is ignored.

Raises ValueError if std is negative/non-finite, mean/std shapes disagree, mean/best are non-finite, or rng is neither None nor a numpy.random RandomState/Generator.

Parameters:
Return type:

ndarray

knowledge_gradient(mean, cov, noise=1.0e-6)[source]

Knowledge-gradient acquisition value of one observation at each candidate (Frazier et al. 2009).

Given the Gaussian-process posterior mean and joint cov over a candidate set, returns, for each candidate x, the expected increase in the best posterior mean after fantasizing one (noisy) observation there: KG(x) = E_y[max_x' mu_{n+1}(x')] - max_x' mu_n(x'). Maximizing KG is the one-step Bayes-optimal, look-ahead choice (it values information, not just immediate improvement), so it explores where an observation would most change the believed optimum. Assumes a maximization objective; negate mean for minimization. Computed exactly via the piecewise-linear epigraph of the fantasized posterior means, and >= 0 by construction.

Parameters:
Return type:

ndarray

propose_knowledge_gradient(x, y, bounds, *, maximize=False, n_candidates=512, seed=None, gp=None, fit_kwargs=None, noise=1.0e-6)[source]

Propose the next evaluation point by maximizing the knowledge gradient over a candidate set.

Fits the GP surrogate to (x, y), draws n_candidates Latin-hypercube points, evaluates the joint posterior, and returns the candidate with the largest knowledge_gradient() – the look-ahead Bayesian-optimization proposal. maximize selects the objective sense (the mean is negated for minimization).

Raises ValueError if the surrogate’s predicted mean/covariance don’t match the candidate contract (wrong shape, non-finite, asymmetric covariance) or if every candidate’s knowledge-gradient value is non-finite (see _validate_prediction() / _select_index()).

Parameters:
Return type:

ndarray

register_acquisition(name, fn, aliases=())[source]

Register an acquisition fn under name (and any aliases).

fn is called as fn(mean, std, best, *, maximize, **params) and must return a merit array that the proposal loop maximizes over candidates. This is the extension point for new acquisitions – registering is all that is needed, no edits to propose_next/minimize.

Parameters:

  • name (str)

  • fn (Acquisition)

  • aliases (tuple[str, ...])

Return type:

None

available_acquisitions()[source]

Return the sorted names (and aliases) of all registered acquisitions.

Return type:

list[str]

propose_next(x, y, bounds, n_candidates=512, seed=None, *, maximize=False, xi=0.0, acq='ei', acq_kwargs=None, gp=None, fit_kwargs=None, return_acquisition=False)[source]

Propose the next point to evaluate by maximizing an acquisition function.

Fits a GP to (x, y), scores n_candidates Latin-hypercube points by the acq acquisition ("ei" / "pi" / "ucb" or any registered name / callable), and returns the best candidate (a (d,) array), optionally with its merit. xi is forwarded to acquisitions that use it (EI, PI); per-acquisition parameters such as kappa go in acq_kwargs.

Raises ValueError if the surrogate’s predicted mean/covariance don’t match the candidate contract (wrong shape, non-finite, asymmetric covariance) or if every candidate’s acquisition merit is non-finite (validated at the shared surrogate boundary, see _propose_one()).

Parameters:
Return type:

ndarray | tuple[ndarray, float]

propose_batch(x, y, bounds, q, n_candidates=512, seed=None, *, maximize=False, xi=0.0, acq='ei', acq_kwargs=None, fit_kwargs=None)[source]

Propose a batch of q points to evaluate together, via the kriging-believer heuristic.

Each pick maximizes the acquisition; the GP posterior mean at the chosen point is then appended as a fantasized observation and the surrogate refit, so the next pick is steered away from it. Returns a (q, d) array. This needs no true objective evaluations between picks, so it suits parallel/asynchronous experiment campaigns.

Raises ValueError if the surrogate’s predicted mean/covariance or fantasized single-point prediction don’t match the candidate contract, or if every candidate’s acquisition merit is non-finite at some step (see _validate_prediction() / _select_index()).

Parameters:
Return type:

ndarray

minimize(objective, bounds, n_init=5, n_iter=15, seed=None, *, maximize=False, xi=0.0, acq='ei', acq_kwargs=None, n_candidates=512, fit_kwargs=None)[source]

Run sequential GP Bayesian optimization of a scalar objective over bounds.

Seeds with an n_init-point Latin-hypercube design, then runs n_iter acquisition-driven steps using acq ("ei" by default; also "pi" / "ucb" or any registered acquisition). Minimizes by default; set maximize=True to maximize. objective takes a (d,) point and returns a float.

A non-finite objective response is recorded in failed_evaluations and stops the run with stopped_reason='objective_failed'. It is never admitted to the fitted x/y history.

The GP surrogate needs the optional torch extra (pip install "mixle[torch]"). On a base install that is checked up front, before the first objective evaluation – never after the n_init design has already spent expensive black-box calls.

Parameters:
Return type:

BayesOptResult

class OptimizationResult(x, y)[source]

Bases: object

Common outcome of a model-based optimization run: the full evaluation history.

x is the (N, d) matrix of evaluated points and y the corresponding objective values (an (N,) vector for single-objective runs, an (N, M) matrix for multi-objective). Concrete result types extend this with their best-point / Pareto-front fields.

Parameters:
class BayesOptResult(x, y, best_x, best_y, n_evaluations, failed_evaluations, stopped_reason)[source]

Bases: OptimizationResult

Outcome of a Bayesian-optimization run.

Parameters: