mixle.inference.project module

Closed-form variational projections — compress a structured teacher onto a smaller student exactly.

mixle.ops.project already does the general forward-KL (M-)projection by SAMPLING the source and fitting the target by maximum likelihood. That is universal but approximate and slow. This module adds the cases where the projection has a closed form and needs no samples and no iteration:

  • collapse_mixture() — moment-match a mixture onto a single component. For Gaussians this is the law of total variance, exact to machine precision (the M-projection of the mixture onto one Gaussian).

  • reduce_mixture() — Runnalls’ KL-greedy mixture reduction: repeatedly merge the pair of components whose merge costs the least KL, until the target count is reached. Each merge is a closed-form moment match, and the merge cost is Runnalls’ analytic dissimilarity — so the whole reduction is closed form. Every merge is moment-preserving, so the reduced mixture keeps the original’s overall mean and covariance exactly (a strong, checkable invariant).

  • gaussian_kl() — the analytic KL between two Gaussians (the metric the above use).

Why Gaussians first: the mixtures that actually need compressing at frontier scale are Gaussian-ish – mixture-of-experts routed in a latent, Kalman/SSM belief states, VLM latent fusion, GP posteriors. The same moment-matching idea extends to any exponential family (via mixle.stats.compute.exp_family.ExponentialFamilyForm.mean_parameters); those are added as their closed-form moment algebra is filled in. Non-Gaussian sources fall back to the sampling projection with a clear pointer, rather than silently pretending to be exact.

References: Runnalls, “Kullback-Leibler approach to Gaussian mixture reduction” (IEEE T-AES 2007); the moment-matching M-projection is the standard forward-KL result (Minka; Bishop PRML §10.7).

gaussian_kl(p, q)[source]

KL(p || q) between two Gaussians (uni- or multivariate), in nats. Accepts distributions.

0.5[ tr(Σq⁻¹ Σp) + (μq-μp)ᵀ Σq⁻¹ (μq-μp) - d + ln(det Σq / det Σp) ] – the analytic Gaussian KL.

Parameters:
Return type:

float

collapse_mixture(mixture)[source]

Moment-match a mixture onto a single distribution, in closed form (exact for Gaussians).

Returns the Gaussian minimizing KL(mixture || Gaussian) – its mean and covariance are the mixture’s overall mean and covariance (law of total variance). This is the exact, sample-free counterpart to mixle.ops.project(mixture, GaussianDistribution(...).estimator()).

Parameters:

mixture (Any)

Return type:

Any

reduce_mixture(mixture, n_components, *, method='runnalls')[source]

Reduce a Gaussian mixture to n_components by greedily merging the least-costly pair (Runnalls).

Repeatedly merges the two components whose merge costs the least KL (_runnalls_cost()) until n_components remain. Every merge is moment-preserving, so the reduced mixture has the same overall mean and covariance as the original – only higher moments are lost. Returns a mixture of the same kind as the input (a GaussianMixtureDistribution for multivariate input).

Parameters:

  • mixture (Any) – a Gaussian mixture (GaussianMixtureDistribution or a mixture of Gaussian components).

  • n_components (int) – target number of components (>= 1); no-op if already <= the current count.

  • method (str) – only "runnalls" for now.

Return type:

Any

moment_project(teacher, target=None, *, exact=True, **sampling_kw)[source]

Project teacher onto a smaller student – exactly when possible, else by sampling.

If teacher is a Gaussian mixture and target is None (or a single Gaussian family), the projection is the closed-form collapse_mixture() – no samples, machine-precision. Otherwise (or when exact=False) it delegates to mixle.ops.project(), the sampling M-projection onto target’s family. This gives one entry point that is exact where the structure allows and honest (sampling, clearly) where it does not.

Parameters:
Return type:

Any

fisher_merge(estimates, fishers=None)[source]

Fisher-weighted merge of parameter estimates – the closed-form Laplace-posterior combination.

Given estimates θ_i (each a flat parameter vector) and their Fisher information F_i, returns θ* = (Σ F_i)⁻¹ (Σ F_i θ_i) – the point that maximizes the sum of the local Laplace log-posteriors Σ_i -½(θ-θ_i)ᵀ F_i (θ-θ_i). This is Matena & Raffel Fisher merging (diagonal F) and, in general, the precision-weighted mean; for Gaussians it coincides with the product-of-experts mean. No gradient steps – a single linear solve.

Parameters:

  • estimates (Any) – sequence of parameter vectors θ_i (each shape (p,)), or a stack (m, p).

  • fishers (Any) – per-estimate Fisher information. Each may be a scalar/1-D vector (diagonal Fisher, per-coordinate precision) or a (p, p) matrix (full Fisher). None uses unit Fisher (a plain average). A single value is broadcast to every estimate.

Returns:

The merged parameter vector θ* of shape (p,).

Return type:

ndarray