mixle.doe.bayesopt module

Gaussian-process Bayesian optimization over a bounded input space (WS-E).

A sequential model-based optimization loop: fit a GP surrogate to the observed points, score Latin-hypercube candidates with an acquisition function, and evaluate the best candidate next. Reuses mixle.models.gaussian_process.GaussianProcessRegressor (torch) as the surrogate; the acquisition functions themselves are torch-free numpy.

Acquisitions are looked up through a small registry (register_acquisition / acq= name) – the same “register, don’t branch” pattern as the engines and encoded-data backends – so a new acquisition plugs in without editing the proposal loop. Built in: expected improvement ("ei"), probability of improvement ("pi"), and the upper/lower confidence bound ("ucb"). Each takes (mean, std, best, *, maximize, **params) and returns a merit array that is maximized over the candidate set. Batch proposals (propose_batch) use the kriging-believer heuristic: fantasize the posterior mean at each pick, refit, and repeat, giving a spatially diverse batch.

class Acquisition(*args, **kwargs)[source]

Bases: Protocol

The acquisition-function contract registered via register_acquisition (EI/PI/UCB).

An acquisition scores candidate points from their surrogate posterior moments and returns a merit array that the proposal loop maximizes over the candidate set. It is called as fn(mean, std, best, *, maximize, **params) where mean / std are the predictive mean and standard deviation at the candidates, best is the incumbent objective value, maximize selects the optimization sense, and **params carries per-acquisition knobs (e.g. xi for EI/PI, kappa for UCB). Built-ins mixle.doe.bayesopt.expected_improvement(), probability_of_improvement(), and upper_confidence_bound() satisfy this contract.

class Surrogate(*args, **kwargs)[source]

Bases: Protocol

The GP-surrogate contract passed as gp= to the Bayesian-optimization loops.

A surrogate is fit to the observed (x, y) and queried for the posterior predictive moments at new candidate points. The call convention is the one used by mixle.models.gaussian_process.GaussianProcessRegressor: fit trains in place (it may return diagnostics, which the loops ignore), and predict takes the training data alongside the query points, returning the posterior mean (return_cov=False) or (mean, cov) pair (return_cov=True).

fit(x, y, **kwargs)[source]
Parameters:
Return type:

Any

predict(x_train, y_train, x_new, return_cov=...)[source]
Parameters:
Return type:

Any

expected_improvement(mean, std, best, xi=0.0, *, maximize=False, **_)[source]

Return the expected-improvement acquisition at points with surrogate mean and std.

For minimization the improvement over the incumbent best is best - mean - xi; for maximization it is mean - best - xi. xi >= 0 trades exploration for exploitation. Points with zero predictive std get zero EI. Higher is better (maximized over candidates).

Parameters:
Return type:

ndarray

log_expected_improvement(mean, std, best, xi=0.0, *, maximize=False, **_)[source]

Return the log expected-improvement acquisition – a numerically stable EI (Ament et al. 2023).

Mathematically log(EI), but computed so it stays finite and informative deep in the no-improvement tail where EI itself underflows to 0 (and log EI to -inf), keeping the optimizer’s ordering and gradients usable. Same argmax as expected_improvement(); points with zero predictive std get -inf. Higher is better. The z >= 0 branch is the direct well-conditioned form; the z < 0 tail uses the scaled complementary error function erfcx (Phi(z)/phi(z) = sqrt(pi/2) erfcx(-z/sqrt2)), which is bounded there and so never under/overflows.

Parameters:
Return type:

ndarray

probability_of_improvement(mean, std, best, *, maximize=False, xi=0.0, **_)[source]

Return the probability-of-improvement acquisition.

The probability that a candidate improves on the incumbent best by at least xi: P(f < best - xi) for minimization, P(f > best + xi) for maximization. Where the predictive std is zero the improvement is deterministic (1.0 if it improves, else 0.0). Higher is better.

Parameters:
Return type:

ndarray

upper_confidence_bound(mean, std, best=0.0, *, maximize=False, kappa=1.96, **_)[source]

Return the confidence-bound acquisition (UCB for maximization, LCB for minimization).

Returns a merit that is maximized over candidates: the optimistic bound mean + kappa * std when maximizing, and kappa * std - mean when minimizing (so picking the largest merit selects the most promising low-objective point). kappa >= 0 trades exploration for exploitation; best is ignored. Higher is better.

Parameters:
Return type:

ndarray

thompson_sampling(mean, std, best=0.0, *, maximize=False, rng=None, **_)[source]

Thompson-sampling acquisition: one marginal posterior draw N(mean, std) per candidate.

Returns a merit (maximized over candidates) equal to the drawn value when maximizing and its negation when minimizing, so the selected point is the optimum of the sampled objective. A randomized, exploration-aware acquisition – repeated proposals explore competing optima in proportion to posterior probability, with no exploration knob to tune. This is the cheap marginal variant (independent per-candidate draws, ignoring the GP’s cross-candidate correlation); pass an rng (via acq_kwargs) for reproducible proposals. best is ignored.

Parameters:
Return type:

ndarray

knowledge_gradient(mean, cov, noise=1.0e-6)[source]

Knowledge-gradient acquisition value of one observation at each candidate (Frazier et al. 2009).

Given the Gaussian-process posterior mean and joint cov over a candidate set, returns, for each candidate x, the expected increase in the best posterior mean after fantasizing one (noisy) observation there: KG(x) = E_y[max_x' mu_{n+1}(x')] - max_x' mu_n(x'). Maximizing KG is the one-step Bayes-optimal, look-ahead choice (it values information, not just immediate improvement), so it explores where an observation would most change the believed optimum. Assumes a maximization objective; negate mean for minimization. Computed exactly via the piecewise-linear epigraph of the fantasized posterior means, and >= 0 by construction.

Parameters:
Return type:

ndarray

propose_knowledge_gradient(x, y, bounds, *, maximize=False, n_candidates=512, seed=None, gp=None, fit_kwargs=None, noise=1.0e-6)[source]

Propose the next evaluation point by maximizing the knowledge gradient over a candidate set.

Fits the GP surrogate to (x, y), draws n_candidates Latin-hypercube points, evaluates the joint posterior, and returns the candidate with the largest knowledge_gradient() – the look-ahead Bayesian-optimization proposal. maximize selects the objective sense (the mean is negated for minimization).

Parameters:
Return type:

ndarray

register_acquisition(name, fn, aliases=())[source]

Register an acquisition fn under name (and any aliases).

fn is called as fn(mean, std, best, *, maximize, **params) and must return a merit array that the proposal loop maximizes over candidates. This is the extension point for new acquisitions – registering is all that is needed, no edits to propose_next/minimize.

Parameters:

  • name (str)

  • fn (Acquisition)

  • aliases (tuple[str, ...])

Return type:

None

available_acquisitions()[source]

Return the sorted names (and aliases) of all registered acquisitions.

Return type:

list[str]

propose_next(x, y, bounds, n_candidates=512, seed=None, *, maximize=False, xi=0.0, acq='ei', acq_kwargs=None, gp=None, fit_kwargs=None, return_acquisition=False)[source]

Propose the next point to evaluate by maximizing an acquisition function.

Fits a GP to (x, y), scores n_candidates Latin-hypercube points by the acq acquisition ("ei" / "pi" / "ucb" or any registered name / callable), and returns the best candidate (a (d,) array), optionally with its merit. xi is forwarded to acquisitions that use it (EI, PI); per-acquisition parameters such as kappa go in acq_kwargs.

Parameters:
Return type:

ndarray | tuple[ndarray, float]

propose_batch(x, y, bounds, q, n_candidates=512, seed=None, *, maximize=False, xi=0.0, acq='ei', acq_kwargs=None, fit_kwargs=None)[source]

Propose a batch of q points to evaluate together, via the kriging-believer heuristic.

Each pick maximizes the acquisition; the GP posterior mean at the chosen point is then appended as a fantasized observation and the surrogate refit, so the next pick is steered away from it. Returns a (q, d) array. This needs no true objective evaluations between picks, so it suits parallel/asynchronous experiment campaigns.

Parameters:
Return type:

ndarray

minimize(objective, bounds, n_init=5, n_iter=15, seed=None, *, maximize=False, xi=0.0, acq='ei', acq_kwargs=None, n_candidates=512, fit_kwargs=None)[source]

Run sequential GP Bayesian optimization of a scalar objective over bounds.

Seeds with an n_init-point Latin-hypercube design, then runs n_iter acquisition-driven steps using acq ("ei" by default; also "pi" / "ucb" or any registered acquisition). Minimizes by default; set maximize=True to maximize. objective takes a (d,) point and returns a float.

Parameters:
Return type:

BayesOptResult

class OptimizationResult(x, y)[source]

Bases: object

Common outcome of a model-based optimization run: the full evaluation history.

x is the (N, d) matrix of evaluated points and y the corresponding objective values (an (N,) vector for single-objective runs, an (N, M) matrix for multi-objective). Concrete result types extend this with their best-point / Pareto-front fields.

Parameters:
x: ndarray
y: ndarray
class BayesOptResult(x, y, best_x, best_y)[source]

Bases: OptimizationResult

Outcome of a Bayesian-optimization run.

Parameters:
best_x: ndarray
best_y: float