mixle.inference.cross_validation module

Cross-validation fold generators, including the dependence-aware schemes.

Random k-fold silently assumes the rows are exchangeable. When they are not – a time series, a spatial field, repeated measures on the same subject – a random split leaks information from neighbours of a test point into the training set, so the held-out score overstates real out-of-sample skill. This is the single most common silent error in evaluating models on autocorrelated data. The fix is to make the fold boundaries respect the dependence:

• i.i.d. – kfold(), leave_one_out(), stratified_kfold() (preserve class balance).

• grouped (repeated measures) – group_kfold(), leave_one_group_out(): never split a group across train and test.

• temporal – blocked_kfold() (contiguous blocks), time_series_split() (forward- chaining, train always precedes test), purged_kfold() (an embargo buffer around each test block removed from train).

• spatial – spatial_block_kfold(): hold out contiguous spatial blocks, not scattered points.

• nested – nested_kfold(): an inner CV inside each outer fold for honest tuned-model evaluation.

Every generator returns a list of (train_index, test_index) integer-array pairs, so they are interchangeable wherever a list of folds is expected (a pluggable fold source).

kfold(n, n_splits=5, *, shuffle=False, seed=0)

Standard k-fold split of n rows.

Parameters:

• n (int) – number of observations.

• n_splits (int) – number of folds k (each row is in exactly one test fold).

• shuffle (bool) – shuffle the row order before splitting (use for i.i.d. data; leave False to keep contiguous folds, i.e. blocked_kfold()).

• seed (int | RandomState | None) – RNG seed when shuffle is True.

Returns:

k (train_index, test_index) pairs.

Return type:

list[tuple[ndarray, ndarray]]

blocked_kfold(n, n_splits=5)

Contiguous-block k-fold (no shuffle) for serially dependent data.

Identical to kfold() with shuffle=False; named separately because keeping blocks contiguous is the point for time series, not an incidental default.

Parameters:

• n (int)

• n_splits (int)

Return type:

list[tuple[ndarray, ndarray]]

leave_one_out(n)

Leave-one-out CV: n folds, each holding out a single observation.

Parameters:

n (int)

Return type:

list[tuple[ndarray, ndarray]]

stratified_kfold(y, n_splits=5, *, shuffle=True, seed=0)

Stratified k-fold preserving class proportions in every fold.

Distributes each class’s indices across the folds so that class frequencies in each test fold mirror the overall frequencies – important for imbalanced classification.

Parameters:

• y (ndarray) – (n,) class labels.

• n_splits (int) – number of folds.

• shuffle (bool) – shuffle within each class before round-robin assignment.

• seed (int | RandomState | None) – RNG seed when shuffle is True.

Returns:

n_splits (train_index, test_index) pairs.

Return type:

list[tuple[ndarray, ndarray]]

leave_one_group_out(groups)

Leave-one-group-out CV: one fold per distinct group held out entirely.

Parameters:

groups (ndarray) – (n,) group labels (e.g. subject / site / genus id).

Returns:

One (train_index, test_index) pair per unique group.

Return type:

list[tuple[ndarray, ndarray]]

group_kfold(groups, n_splits=5)

Group k-fold: partition groups into k folds so no group spans train and test.

Greedily assigns whole groups (largest first) to the currently-smallest fold, balancing fold sizes while keeping each group intact – the right scheme for repeated measures when there are more groups than folds.

Parameters:

• groups (ndarray)

• n_splits (int)

Return type:

list[tuple[ndarray, ndarray]]

time_series_split(n, n_splits=5, *, gap=0, max_train_size=None)

Forward-chaining time-series CV: train always precedes test (expanding window).

The series is cut into n_splits + 1 contiguous blocks; fold i tests on block i+1 and trains on everything strictly before it. A gap (buffer) drops the gap points just before each test block from the training set so leakage through short-range autocorrelation is avoided.

Parameters:

• n (int) – number of time-ordered observations.

• n_splits (int) – number of train/test splits.

• gap (int) – number of observations to drop between the train and test segments.

• max_train_size (int | None) – cap on the training-window length (sliding window); None keeps it expanding.

Returns:

n_splits (train_index, test_index) pairs.

Return type:

list[tuple[ndarray, ndarray]]

purged_kfold(n, n_splits=5, *, embargo=0)

Purged/embargoed blocked k-fold (a.k.a. buffered CV).

Contiguous test blocks, but the embargo observations on each side of a test block are removed from the training set, so no training point sits within the autocorrelation reach of a test point. The standard guard for serially dependent data when you still want every block to serve as a test fold (cf. de Prado 2018).

Parameters:

• n (int) – number of time-ordered observations.

• n_splits (int) – number of contiguous folds.

• embargo (int) – buffer width removed from train on both sides of each test block.

Returns:

n_splits (train_index, test_index) pairs.

Return type:

list[tuple[ndarray, ndarray]]

spatial_block_kfold(coords, n_splits=5, *, block_size=None, n_side=None, seed=0)

Spatial-block k-fold: hold out contiguous spatial blocks, not scattered points.

Partitions space into a regular grid of blocks and assigns whole blocks to folds at random, so a test point’s spatial neighbours are held out with it rather than leaking into training. This is the right scheme for geostatistical / spatially autocorrelated data.

Parameters:

• coords (ndarray) – (n, d) spatial coordinates (typically d = 2).

• n_splits (int) – number of folds.

• block_size (float | None) – grid-cell width in coordinate units; if None it is derived from n_side.

• n_side (int | None) – number of grid cells per axis; defaults so there are comfortably more blocks than folds.

• seed (int | RandomState | None) – RNG seed for assigning blocks to folds.

Returns:

n_splits (train_index, test_index) pairs.

Return type:

list[tuple[ndarray, ndarray]]

nested_kfold(n, *, outer_splits=5, inner_splits=4, shuffle=False, seed=0)

Nested k-fold: an inner CV (for tuning) inside each outer fold (for evaluation).

The outer loop estimates generalisation; the inner loop selects hyper-parameters using only the outer-training data, so the reported score is not optimistically biased by tuning on the test set.

Parameters:

• n (int) – number of observations.

• outer_splits (int) – number of outer folds.

• inner_splits (int) – number of inner folds within each outer training set.

• shuffle (bool) – shuffle before splitting (i.i.d. data only).

• seed (int | RandomState | None) – RNG seed.

Returns:

A list of NestedFold; inner folds index the original array.

Return type:

list[NestedFold]

class NestedFold(train, test, inner)

Bases: object

One outer fold of nested_kfold().

Parameters:

• train (ndarray)

• test (ndarray)

• inner (list[tuple[ndarray, ndarray]])

train

outer training indices (used to build the inner CV).

Type:

numpy.ndarray

test

outer test indices (held out for the final, untuned-on evaluation).

Type:

numpy.ndarray

inner

inner (train_index, test_index) folds, indexing into the original array (not into train), so they can be applied directly.

Type:

list[tuple[numpy.ndarray, numpy.ndarray]]

train: ndarray

test: ndarray

inner: list[tuple[ndarray, ndarray]]