mixle.stats.compute.sequence module

Vectorized sequence-driver primitives over the pdist protocol.

The module-level seq_* API — encode a dataset, score it, run a vectorized EM E-step, initialize an accumulator — dispatching over the (encoder, estimator, distribution) contracts in mixle.stats.compute.pdist. These were defined inline in mixle.stats.__init__; they live here so the fitting machinery (mixle.inference) can import them WITHOUT importing the whole mixle.stats package — which previously forced mixle.inference to resolve lazily to dodge a half-initialized mixle.stats. mixle.stats re-exports them, so the public mixle.stats.seq_estimate / seq_log_density_sum / … API is unchanged.

seq_encode(data, encoder=None, estimator=None, model=None, num_chunks=1, chunk_size=None)[source]

Sequence encode a sequence of iid observations from a distribution corresponding to ‘encoder’.

Takes data of type Union[Sequence[T], pyspark.rdd.RDD], where the data type of the DataSequenceEncoder object’s corresponding distribution is type T.

If not RDD, returns a List[Tuple[int, T1]], with each list entry being a tuple containing the number of observations in the sequence (chunk_size), and an encoded sequence of the observations having type T1. The list has length num_chunks.

RDD version with receive the Tuple of chunk_size and encoded data of type T1 for each corresponding node.

Parameters:

  • data (Union[Sequence[T], pyspark.rdd.RDD]) – Sequence of iid observations of data type consistent with ‘encoder’.

  • encoder (Optional[DataSequenceEncoder]) – A DataSequenceEncoder object for sequence encoding iid sequences.

  • estimator (Optional[ParameterEstimator]) – An estimator to create DataSequenceEncoder from.

  • model (Optional[SequenceEncodableProbabilityDistribution]) – A distribution to create DataSequenceEncoder from.

  • num_chunks (int) – Number of chunks to split the data into. Useful for distributed data sets.

  • chunk_size (Optional[int]) – Approximate size of chunks to determine num_chunks above.

Returns:

Sequence encoded data for use with ‘seq_’ functions.

Return type:

pyspark.rdd.RDD | list[tuple[int, Any]]

seq_log_density_sum(enc_data, estimate)[source]
Vectorized evaluation of the sum of log_density values for a given SequenceEncodableProbabilityDistribution

over encoded data.

Returns a Tuple containing the sum of all observations in enc_data, and the sum of the log_density evaluated at all encoded data observations in enc_data. This is a fully vectorized evaluation.

Parameters:

  • enc_data (Union[List[Tuple[int, T]], 'pyspark.rdd.RDD']) – Sequence encoded data of format matching output of seq_encode() function.

  • estimate (SequenceEncodableProbabilityDistribution) – Distribution to use for log_density evaluations. Must be consistent with enc_data.

Returns:

Tuple of sum of total obs, and sum of log_density of estimate at all encoded data observations.

Return type:

tuple[float, float]

seq_log_density(enc_data, estimate)[source]

Vectorized evaluation of ‘estimate’ log-density for each observation in enc_data.

If ‘estimate’ is input as a List of numpy arrays. Each list entry corresponds to the seq_log_density calls of all the encoded data for each List entry of estimate.

If ‘estimate’ is a single SequenceEncodableProbabilityDistribution instance. The log_density of every observation in the ‘enc_data’ data set is returned as a list.

Parameters:

  • enc_data (Union[List[Tuple[int, T]], 'pyspark.rdd.RDD']) – Sequence encoded data of format matching output of seq_encode() function.

  • estimate (SequenceEncodableProbabilityDistribution) – Distribution to use for log_density evaluations. Must be consistent with enc_data.

Returns:

List[np.ndarray[float]] or List[float] depending on input.

Return type:

list[np.ndarray]

log_density(data, model)[source]

Per-observation log-density of ‘model’ over raw (unencoded) ‘data’.

Convenience wrapper that encodes ‘data’ with the model’s own encoder, evaluates the vectorized seq_log_density, and returns a single flat numpy array aligned to the input order – the common need that otherwise requires the seq_encode / seq_log_density / np.concatenate boilerplate. For a distributed RDD the densities are collected to the driver in partition order.

Parameters:

  • data (Union[Sequence[T], pyspark.rdd.RDD]) – Raw iid observations of data type consistent with ‘model’.

  • model (SequenceEncodableProbabilityDistribution) – Distribution to score the observations under.

Returns:

np.ndarray of per-observation log-densities.

Return type:

np.ndarray

density(data, model)[source]

Per-observation density of ‘model’ over raw (unencoded) ‘data’.

Exponentiated companion to log_density(); returns a flat numpy array of densities aligned to the input order.

Parameters:

  • data (Union[Sequence[T], pyspark.rdd.RDD]) – Raw iid observations of data type consistent with ‘model’.

  • model (SequenceEncodableProbabilityDistribution) – Distribution to score the observations under.

Returns:

np.ndarray of per-observation densities.

Return type:

np.ndarray

seq_estimate(enc_data, estimator, prev_estimate)[source]

Perform vectorized E-step in EM algorithm for encoded sequence of observations in ‘enc_data’.

Arg estimator must be consistent with prev_estimate. That is, prev_estimate must be an estimate that could be obtained from estimator.

Arg enc_data must type consistent with estimator and prev_estimate (result of seq_encode() call).

Returns the next iteration of EM algorithm with vectorized calls to “seq_update()” of the corresponding SequenceEncodableStatsiticAccumulator objects.

Parameters:

  • enc_data (Union[List[Tuple[int, T]], 'pyspark.rdd.RDD']) – Sequence encoded data of format matching output of seq_encode() function.

  • estimator (ParameterEstimator) – Model to be estimated from ‘enc_data’.

  • prev_estimate (SequenceEncodableProbabilityDistribution) – Previous estimate of EM algorithm.

Returns:

SequenceEncodableProbabilityDistribution object.

Return type:

T_D

seq_initialize(enc_data, estimator, rng, p=0.1)[source]
Vectorized initialization of a model corresponding to ParameterEstimator for encoded sequences of iid data

observations.

Arg enc_data must type consistent with estimator (result of seq_encode() call). Arg estimator must be of data type consistent with encoded sequence data type in ‘enc_data’.

Vectorized initialization of SequenceEncodableProbabilityDistribution corresponding to ‘estimator’ from enc_data. Observations in the encoded sequence enc_data are kept with probability p.

This functions relies on calls to SequenceEncodableStatisticAccumulator.seq_initialize(), which is a vectorized initialization of the SequenceEncodableStatisticAccumulator object.

This method should produce the same initialized model as a call to initialize() if the data sets are the same.

Parameters:

  • enc_data (Union[List[Tuple[int, T]], 'pyspark.rdd.RDD']) – Sequence encoded data of format matching output of seq_encode() function.

  • estimator (ParameterEstimator) – Model to be estimated from ‘enc_data’.

  • rng (RandomState) – RandomState object for setting seed.

  • p (float) – Proportion of data to randomly sample for initializing model.

Returns:

SequenceEncodableProbabilityDistribution object consistent with ‘estimator’.

Return type:

SequenceEncodableProbabilityDistribution

initialize(data, estimator, rng, p=0.1)[source]

Randomly initialize a model corresponding to ParameterEstimator for iid observations data.

Note: ParameterEstimator must be of data type T, matching the input data.

This function sequentially iterates over the entire data set ‘data’, repeatedly calling initialize() method of the SequenceEncodableStatisticAccumulator object created from ‘estimator’. Data points are weighted 0 or 1 with probability p.

Seq_initialize() is much more efficient, and should produce the same initialized model for the same data sets.

Parameters:

  • data (Union[Sequence[T], pyspark.rdd.RDD]) – Set of iid observations compatible with ‘estimator’.

  • estimator (ParameterEstimator) – ParameterEstimator object for desired model to be estimated from data.

  • rng (RandomState) – RandomState object for setting seed.

  • p (float) – Proportion of data to randomly sample for initializing model.

Returns:

SequenceEncodableProbabilityDistribution object consistent with ‘estimator’.

Return type:

SequenceEncodableProbabilityDistribution

estimate(data, estimator, prev_estimate=None)[source]

Perform E-step in EM algorithm by iterating over all observations in ‘data’.

Arg estimator must be consistent with prev_estimate. That is, prev_estimate must be an estimate that could be obtained from estimator.

Data must type consistent with estimator and prev_estimate.

Returns the next iteration of EM algorithm by iterating over each observation of data. See seq_estimate() for a more computationally efficient implementation.

Parameters:

  • data (Union[Sequence[T], pyspark.rdd.RDD]) – Sequence of iid observations of data type consistent with ‘estimator’ and/or ‘prev_estimate’.

  • estimator (ParameterEstimator) – Model to be estimated from ‘data’.

  • prev_estimate (Optional[SequenceEncodableProbabilityDistribution]) – Previous estimate of EM algorithm. Must be included for distributions that require initialization.

Returns:

SequenceEncodableProbabilityDistribution object.

Return type:

SequenceEncodableProbabilityDistribution