mixle.utils.evaluation module

Held-out evaluation and data-partitioning utilities.

Empirical KL divergence between a fitted model and data, plus index/data partitioning helpers (k-fold split, proportional split) used for validation and cross-validation.

empirical_kl_divergence(dist1, dist2, enc_data)

Computes the emirical KL-divergence between two densities.

Compute the KL-divergence between dist1 and dist2, for encoded sequence of data. Dists must both have the same encodings.

Parameters:

• dist1 (SequenceEncodableProbabilityDistribution) – Distribution compatible with enc_data.

• dist2 (SequenceEncodableProbabilityDistribution) – Distribution compatible with enc_data.

• enc_data (List[Tuple[int, Any]]) – List of Tuple containing chunk size and encoded sequence for chunked data.

Returns:

Tuple of KL-div estiamte, number of ‘bad’ likelihood values for dist1, ‘bad’ likelihood values for dist2.

Return type:

tuple[float, float, float]

ks_test(data, dist)

One-sample Kolmogorov-Smirnov goodness-of-fit test of data against dist.

Returns (D, p_value) where D = sup_x |F_n(x) - dist.cdf(x)| is the KS statistic and the two-sided p-value is the exact Kolmogorov distribution P(D_n >= D) (scipy.stats.kstwo). dist must expose a scalar cdf (the HasCDF capability). A small p-value is evidence that data is not distributed as dist – continuous goodness-of-fit / model checking.

Parameters:

• data (Sequence[float])

• dist (Any)

Return type:

tuple[float, float]

chi_square_test(data, dist, *, lo=None, hi=None)

Pearson chi-square goodness-of-fit test for a discrete dist against integer data.

Bins the observations over each value in [lo, hi] plus a single combined tail bin for everything outside that window (so the expected cell probabilities sum to 1, using dist.cdf for the tail). Returns (chi2, dof, p_value) with dof = #cells - 1 and the upper-tail chi-square p-value; lo/hi default to the data’s min/max. A small p-value is evidence of misfit.

Parameters:

• data (Sequence[int])

• dist (Any)

• lo (int | None)

• hi (int | None)

Return type:

tuple[float, int, float]

k_fold_split_index(sz, k, rng)

Returns integer numpy index vector for k-fold split. Entry j is the fold-id for the j^{th} data point.

Parameters:

• sz (int) – Integer length of data points in data set.

• k (int) – Integer number of folds for k-folds.

• rng (RandomState) – RandomState for setting seed.

Returns:

1-d np.ndarray[int] of indices for each data points fold-id.

Return type:

ndarray

partition_data_index(sz, pvec, rng)

Returns List of np.ndarray[int] containing integers indexes for data partitions proportional to pvec.

Parameters:

• sz (int) – Integer value of total number of data observations.

• pvec (Union[List[float], np.ndarray]) – Vector of proportions for each partition.

• rng (RandomState) – RandomState for setting seed of random partitioning.

Returns:

List of numpy arrays containing indexes of each partition.

Return type:

list[ndarray]

partition_data(data, pvec, rng)

Partitions List of data into partitions, each with size equal to the proportion of pvec.

Parameters:

• data (Sequence[T]) – Sequence of data observations, each entry of type T.

• pvec (Union[List[float], np.ndarray]) – List of length n, containing proportion of data to be held in each data partition.

• rng (RandomState) – RandomState for setting seed on random partitioning of data.

Returns:

List of List containing data partitions of proportion equal to pvec.

Return type:

list[list[T]]