mixle.stats.latent.lda module¶

Create, estimate, and sample from an integer latent Dirichlet allocation model (LDA).

Defines the LDADistribution, LDASampler, LDAAccumulatorFactory, LDAEstimatorAccumulator, LDAEstimator, and the LDADataEncoder classes for use with mixle.

LDA is a generative model for producing draws from multinomial distribution. The process for generating a document of length N from an LDA with L topics is given as follows:

Draw theta ~ Dirichlet(alpha) (alpha is L dimensional)

Draw topic-counts z_1,….,z_L ~ Multinomial(N, theta)

From each topic l = 1,2,…,L draw z_l words w_{i,l}, w_{i+1,l},…,w_{z_l,l} ~ Categorical(beta_l), where each topic has its own Categorical distribution parameterized by beta_l.

A document is then given by the bag of words produced from this sampling process. Note that a length distribtion is used to sample the number of words in a given document.

class LDADistribution(topics, alpha, len_dist=NullDistribution(), gamma_threshold=1.0e-8, max_gamma_iter=100)[source]

Bases: SequenceEncodableProbabilityDistribution

Latent Dirichlet allocation model for documents given as bags of weighted values.

Data type: Sequence[Tuple[T, float]], where T is the data type of the topic distributions and each (value, count) pair gives the count of a value in the document.

Parameters:

topics (Sequence[SequenceEncodableProbabilityDistribution])
alpha (Sequence[float] | ndarray)
len_dist (SequenceEncodableProbabilityDistribution | None)
gamma_threshold (float)
max_gamma_iter (int)

compute_capabilities()[source]: Return backend capability metadata for this concrete LDA instance.

compute_declaration()[source]

density(x)[source]

Evaluate the density of a single LDA document.

See log_density() for details.

Parameters:: x (Sequence[Tuple[int, float]]) – A document given as (value, count) pairs.
Returns:: Density evaluated at x.
Return type:: float

density_semantics()[source]

What log_density returns relative to the true log-density (default: exact).

Override to declare that this distribution’s log_density is a variational lower bound (ELBO), an upper bound, or an approximation rather than the exact log p(x). This is surfaced as the ExactDensity capability and noted in mixle.describe(), so code that needs an exact likelihood can require(x, ExactDensity) instead of silently trusting a bound.

log_density(x)[source]

Evaluate the log-density of a single LDA document.

Note: The returned value is the variational lower bound (ELBO) on the marginal document log-likelihood obtained from the standard LDA mean-field approximation, not the exact (intractable) marginal log-likelihood.

Parameters:: x (Sequence[Tuple[int, float]]) – A document given as (value, count) pairs.
Returns:: Variational lower bound on the log-density evaluated at x.
Return type:: float

seq_log_density(x)[source]

Vectorized evaluation of the document log-densities for an encoded corpus x.

Encoded sequence ‘x’ is a Tuple of length 5 containing:: x[0] (int): Number of documents in corpus. x[1] (np.ndarray): Document id for flattened array of values. x[2] (np.ndarray): Flattened array of counts for each value in each document. x[3] (Optional[np.ndarray]): Optional warm-start gammas (defaults to None). x[4] (E0): Sequence encoded flattened values.

Note: Returns the per-document variational lower bound (ELBO); see log_density(). If a document-length distribution ‘len_dist’ is set, its log-density of the total token count of each document is added to the returned values.

Parameters:: x (tuple[int, ndarray, ndarray, ndarray | None, E0]) – Encoded corpus of LDA documents (see LDADataEncoder.seq_encode()).
Returns:: Numpy array of log-density (ELBO) values, one entry per document.
Return type:: ndarray

backend_seq_log_density(x, engine)[source]

Backend-neutral LDA variational lower-bound scoring.

Parameters:

x (tuple[int, ndarray, ndarray, ndarray | None, E0])
engine (Any)

Return type:

Any

seq_component_log_density(x)[source]

Vectorized evaluation of the per-topic log-density of each document in encoded corpus x.

Parameters:: x (tuple[int, ndarray, ndarray, ndarray | None, E0]) – Encoded corpus of LDA documents (see LDADataEncoder.seq_encode()).
Returns:: 2-d numpy array with shape (number of documents, n_topics), where entry (i, l) is the log-density of document i evaluated entirely under topic l.
Return type:: ndarray

backend_seq_component_log_density(x, engine)[source]

Backend-neutral per-topic document scores.

Parameters:

x (tuple[int, ndarray, ndarray, ndarray | None, E0])
engine (Any)

Return type:

Any

seq_posterior(x)[source]

Vectorized evaluation of the posterior topic proportions for each document in encoded corpus x.

The variational gammas are computed for each document and normalized to sum to one.

Parameters:: x (tuple[int, ndarray, ndarray, ndarray | None, E0]) – Encoded corpus of LDA documents (see LDADataEncoder.seq_encode()).
Returns:: 2-d numpy array with shape (number of documents, n_topics) containing posterior topic proportions for each document.
Return type:: ndarray

latent_posterior(doc)[source]

Return the mean-field variational posterior q(theta, z) for a single document.

Runs the per-document Blei-Ng-Jordan variational fixed point and returns a MeanFieldLDAPosterior: .topic_proportions() (the document-topic mix E[theta]), .marginals() (per-word topic responsibilities phi), .sample(rng) (theta, z), .mode() (MAP topic per word), or .entropy().

Parameters:: doc (Sequence[tuple[int, float]])
Return type:: MeanFieldLDAPosterior

posterior_predictive(doc, n_words, seed=None)[source]

Draw n_words new words conditioned on the document doc.

Sample the document-topic mix theta ~ q(theta) = Dir(gamma) from the variational posterior, then generate each new word by drawing a topic ~ theta and a word from that topic – “given this document, generate more words from its inferred topic mixture”.

Parameters:

doc (Sequence[tuple[int, float]])
n_words (int)
seed (int | None)

Return type:

list[Any]

sampler(seed=None)[source]

Create an LDASampler object for sampling documents from this distribution.

Parameters:: seed (Optional[int]) – Seed for the random number generator used in sampling.
Returns:: LDASampler object.
Return type:: LDASampler

estimator(pseudo_count=None)[source]

Create an LDAEstimator object from the topics of this distribution.

Parameters:: pseudo_count (Optional[float]) – If passed, used to re-weight sufficient statistics during estimation.
Returns:: LDAEstimator object.
Return type:: LDAEstimator

dist_to_encoder()[source]

Return an LDADataEncoder object for encoding sequences of iid LDA documents.

Return type:: LDADataEncoder

enumerator()[source]

LDA does not support enumeration.

The document log-density is a variational lower bound (ELBO) over latent topic assignments rather than an exact density, so an enumeration satisfying log_prob == log_density over a well-defined support cannot be constructed.

Raises:: EnumerationError – Always.
Return type:: DistributionEnumerator

class LDASampler(dist, seed=None)[source]

Bases: DistributionSampler

LDASampler object for sampling documents from an LDADistribution.

Parameters:

dist (LDADistribution)
seed (int | None)

sample(size=None, *, batched=True)[source]

Draw one or ‘size’ documents from the LDA model.

Note: Sample return value is not counted by value! Each document is returned as a flat list of sampled topic values (use mixle.utils.optsutil.count_by_value to obtain (value, count) pairs).

With batched=True (default), when size is not None the per-document lengths, Dirichlet proportions and topic-count multinomials are drawn first, then every token across the whole batch is grouped by topic and each topic sampler is invoked once. Because the topic samplers are consumed in topic order rather than per-document order, the token draws are statistically equivalent but NOT byte-identical to batched=False. The length, Dirichlet and multinomial draws are byte-identical (same order). Set batched=False to reproduce the exact legacy per-document output for a given seed.

Parameters:

size (Optional[int]) – Number of documents to sample. If None, a single document is returned.
batched (bool) – Vectorize token draws across documents (default); set False for the legacy per-document loop.

Returns:

A single document (list of values) if size is None, else a list of ‘size’ documents.

Return type:

Sequence[Any] | Any

class LDAEstimatorAccumulator(accumulators, len_accumulator=NullAccumulator(), keys=(None, None), prev_alpha=None)[source]

Bases: SequenceEncodableStatisticAccumulator

LDAEstimatorAccumulator object for aggregating sufficient statistics of observed LDA documents.

Parameters:

accumulators (Sequence[SequenceEncodableStatisticAccumulator])
len_accumulator (SequenceEncodableStatisticAccumulator | None)
keys (tuple[str | None, str | None] | None)
prev_alpha (ndarray | None)

update(x, weight, estimate)[source]

Update sufficient statistics with a single weighted LDA document.

Encodes the single observation and delegates to seq_update() so that the scalar and vectorized estimation paths agree.

Parameters:

x (Sequence[Tuple[Any, float]]) – A document given as (value, count) pairs.
weight (float) – Weight for the observation.
estimate (LDADistribution) – Previous estimate of the LDA model.

Returns:

None.

Return type:

None

seq_initialize(x, weights, rng)[source]

Vectorized initialization of sufficient statistics from an encoded corpus x.

Topic assignments are drawn at random from a Dirichlet draw of topic proportions for each document.

Parameters:

x (tuple[int, ndarray, ndarray, ndarray | None, E0]) – Encoded corpus of LDA documents (see LDADataEncoder.seq_encode()).
weights (np.ndarray) – Weights for each document.
rng (np.random.RandomState) – RandomState object used to seed member RandomState objects.

Returns:

None.

Return type:

None

initialize(x, weight, rng)[source]

Initialize sufficient statistics with a single weighted LDA document.

Parameters:

x (Sequence[Tuple[Any, float]]) – A document given as (value, count) pairs.
weight (float) – Weight for the observation.
rng (np.random.RandomState) – RandomState object used to seed member RandomState objects.

Returns:

None.

Return type:

None

seq_update(x, weights, estimate)[source]

Vectorized update of sufficient statistics from an encoded corpus x.

Computes the variational posterior over topic assignments for each document under the previous estimate, then aggregates per-topic statistics, expected log topic proportions, and document counts.

Parameters:

x (tuple[int, ndarray, ndarray, ndarray | None, E0]) – Encoded corpus of LDA documents (see LDADataEncoder.seq_encode()).
weights (np.ndarray) – Weights for each document.
estimate (LDADistribution) – Previous estimate of the LDA model.

Returns:

None.

Return type:

None

seq_update_engine(x, weights, estimate, engine)[source]

Engine-resident LDA E-step.

The variational gamma loop (_backend_seq_posterior), the expected log topic proportions, and the topic-count aggregations all run on the active engine (numpy or torch); per-item topic responsibilities are produced on the engine and fed to the child topic accumulators. Matches host seq_update.

Parameters:

x (tuple[int, ndarray, ndarray, ndarray | None, E0])
weights (ndarray)
estimate (LDADistribution)
engine (Any)

Return type:

None

combine(suff_stat)[source]

Combine the sufficient statistics of suff_stat with this accumulator.

Arg suff_stat is a Tuple of length 6 containing:: suff_stat[0] (Optional[np.ndarray]): Previous Dirichlet parameter estimate. suff_stat[1] (np.ndarray): Aggregated expected log topic proportions. suff_stat[2] (float): Aggregated weighted document count. suff_stat[3] (np.ndarray): Aggregated weighted per-topic value counts. suff_stat[4] (Sequence[SS0]): Sufficient statistics for each topic. suff_stat[5] (Optional[Any]): Sufficient statistics for the document-length distribution.

Parameters:: suff_stat (tuple[ndarray | None, ndarray, float, ndarray, Sequence[SS0], Any | None]) – See above for details.
Returns:: LDAEstimatorAccumulator object.
Return type:: LDAEstimatorAccumulator

value()[source]

Returns sufficient statistics as a Tuple (see combine() for entry details).

Return type:: tuple[ndarray | None, ndarray, float, ndarray, Sequence[Any], Any | None]

from_value(x)[source]

Set the sufficient statistics of this accumulator to x.

Parameters:: x (tuple[ndarray | None, ndarray, float, ndarray, Sequence[SS0], Any | None]) – Sufficient statistic Tuple (see combine() for entry details).
Returns:: LDAEstimatorAccumulator object.
Return type:: LDAEstimatorAccumulator

scale(c)[source]

Scale linear variational sufficient statistics while preserving previous alpha metadata.

Parameters:: c (float)
Return type:: LDAEstimatorAccumulator

key_merge(stats_dict)[source]

Merge sufficient statistics of object instance with matching keys in stats_dict.

Merges alpha sufficient statistics if alpha_key is set, and topic accumulators if topics_key is set.

Parameters:: stats_dict (Dict[str, Any]) – Dictionary mapping keys to corresponding sufficient statistics.
Returns:: None.
Return type:: None

key_replace(stats_dict)[source]

Replace sufficient statistics of object instance with those of matching keys in stats_dict.

Parameters:: stats_dict (Dict[str, Any]) – Dictionary mapping keys to corresponding sufficient statistics.
Returns:: None.
Return type:: None

acc_to_encoder()[source]

Return an LDADataEncoder object for encoding sequences of iid LDA documents.

Return type:: LDADataEncoder

class LDAEstimatorAccumulatorFactory(factories, dim, len_factory=NullAccumulatorFactory(), keys=(None, None), prev_alpha=None)[source]

Bases: StatisticAccumulatorFactory

LDAEstimatorAccumulatorFactory object for creating LDAEstimatorAccumulator objects.

Parameters:

factories (Sequence[StatisticAccumulatorFactory])
dim (int)
len_factory (StatisticAccumulatorFactory)
keys (tuple[str | None, str | None] | None)
prev_alpha (ndarray | None)

make()[source]

Returns an LDAEstimatorAccumulator object from attribute variables.

Return type:: LDAEstimatorAccumulator

class LDAEstimator(estimators, len_estimator=NullEstimator(), suff_stat=None, pseudo_count=None, keys=(None, None), fixed_alpha=None, gamma_threshold=1.0e-8, alpha_threshold=1.0e-8, max_gamma_iter=100)[source]

Bases: ParameterEstimator

LDAEstimator object for estimating an LDADistribution from aggregated sufficient statistics.

Parameters:

estimators (Sequence[ParameterEstimator])
len_estimator (ParameterEstimator | None)
suff_stat (Any | None)
pseudo_count (tuple[float, float] | None)
keys (tuple[str | None, str | None] | None)
fixed_alpha (ndarray | None)
gamma_threshold (float)
alpha_threshold (float)
max_gamma_iter (int)

accumulator_factory()[source]

Returns an LDAEstimatorAccumulatorFactory object from attribute variables.

Return type:: LDAEstimatorAccumulatorFactory

estimate(nobs, suff_stat)[source]

Estimate an LDADistribution from aggregated sufficient statistics.

Arg suff_stat is a Tuple of length 6 containing:: suff_stat[0] (Optional[np.ndarray]): Previous Dirichlet parameter estimate. suff_stat[1] (np.ndarray): Aggregated expected log topic proportions. suff_stat[2] (float): Aggregated weighted document count. suff_stat[3] (np.ndarray): Aggregated weighted per-topic value counts. suff_stat[4] (Sequence[SS0]): Sufficient statistics for each topic. suff_stat[5] (Optional[Any]): Sufficient statistics for the document-length distribution.

Parameters:

nobs (Optional[float]) – Weighted number of observations used in aggregation of suff_stat.
suff_stat – See above for details.

Returns:

LDADistribution object.

Return type:

LDADistribution

class LDADataEncoder(encoder)[source]

Bases: DataSequenceEncoder

LDADataEncoder object for encoding sequences of iid LDA documents.

Parameters:: encoder (DataSequenceEncoder)

seq_encode(x)[source]

Encode a sequence of iid LDA observations for vectorized functions.

Return value ‘rv’ is a Tuple containing:: rv[0] (int): Number of documents in corpus. rv[1] (np.ndarray): Document id for flattened array of values. rv[2] (np.ndarray): Flattened array of counts for each value in each document. rv[3] (Optional[np.ndarray]): Currently default to None rv[4] (E0): Sequence encoded flattened values.

Parameters:: x (Sequence[Sequence[Tuple[int, float]]]) – Sequence of LDA documents.
Returns:: See above for details.
Return type:: tuple[int, ndarray, ndarray, Any | None, Any]

update_alpha(alpha_curr, mean_log_p, alpha_threshold)[source]

Fixed-point update of the Dirichlet parameter alpha given mean expected log proportions.

Parameters:

alpha_curr (np.ndarray) – Current alpha estimate.
mean_log_p (np.ndarray) – Mean expected log topic proportions across documents.
alpha_threshold (float) – Convergence threshold for the fixed-point iteration.

Returns:

Tuple of (updated alpha, number of iterations performed).

Return type:

tuple[ndarray, int]

mpe_update(x_mat, y, min_size=2)[source]

Single minimal polynomial extrapolation (MPE) step for fixed-point sequence acceleration.

Parameters:

x_mat (ndarray | None)
y (ndarray)
min_size (int)

Return type:

tuple[ndarray, ndarray]

mpe(x0, f, eps)[source]

Minimal polynomial extrapolation of the fixed point of f starting from x0.

Parameters:

x0 – Initial point of the fixed-point iteration.
f – Fixed-point map.
eps (float) – Convergence threshold on successive extrapolants.

Returns:

Tuple of (extrapolated fixed point, number of iterations performed).

Return type:

tuple[ndarray, int]

alpha_seq_lambda(mean_log_p)[source]

Returns the alpha fixed-point map for a given mean expected log topic proportion.

Parameters:: mean_log_p (float)
Return type:: Callable[[ndarray], float]

find_alpha(current_alpha, mlp, thresh)[source]

Find the alpha fixed point via MPE acceleration (see update_alpha for the plain iteration).

Parameters:

current_alpha (ndarray)
mlp (float)
thresh (float)

seq_posterior2(estimate, x)[source]

C-extension variant of seq_posterior(). Requires the optional mixle.c_ext module.

Parameters:

estimate (LDADistribution)
x (tuple[int, ndarray, ndarray, Any | None, E0])

seq_posterior(estimate, x)[source]

Run the per-document variational (gamma) fixed-point iteration for an encoded corpus.

Parameters:

estimate (LDADistribution) – LDA model under which the posterior is computed.
x (tuple[int, ndarray, ndarray, Any | None, E0]) – Encoded corpus of LDA documents (see LDADataEncoder.seq_encode()).

Returns:

Tuple of (log_density_gamma, final_gammas, per_topic_log_densities), where log_density_gamma has a row per flattened value with the expected topic-assignment weights (scaled by counts), final_gammas has a row per document with the converged variational Dirichlet parameters, and per_topic_log_densities has a row per flattened value with each topic’s log-density.

LDAAccumulator: alias of LDAEstimatorAccumulator

LDAAccumulatorFactory: alias of LDAEstimatorAccumulatorFactory