mixle.data.sources.array_source module

Scientific-array data sources – zarr / HDF5 / numpy-memmap volumes, plus a PatchSampler.

mixle’s data model is records/tuples/sequences; a 40 GB array on disk has no first-class answer without this module. Each array-store connector is lazy: constructing it opens the store’s metadata (shape, dtype, chunking) but never reads the full array into memory – only requested slices are pulled off disk.

PatchSampler wraps any of these (or any N-D array-like object exposing .shape/__getitem__) and yields (patch, coords) records – fixed-size N-D patches at deterministic, seeded locations – without ever materializing the underlying volume. Because it also implements __len__/__getitem__ by index, it plugs directly into MPEncodedData: the driver shards by index (data[j] for j in range(i, n, num_workers)) and only those patches are read and pickled to worker i – the full volume is never touched by the driver process.

Optional: zarr and h5py are guarded behind mixle.utils.optional_deps (pip install mixle[arrays]); numpy-memmap needs no extra dependency. Every connector still constructs even when its dependency is uninstalled – it defers the require(...) error to first use (matching sql_source/mongo_source), except memmap, which never has a missing dependency to guard.

class ZarrArraySource(path, component=None, *, structure=EXCHANGEABLE, schema=None)[source]

Bases: _ArrayVolumeSource

Lazy connector over a zarr array (or a named array within a zarr group/store).

Optional: requires zarr (pip install mixle[arrays]). Opening a zarr store reads only its metadata; self.array is the live zarr Array – slicing it (row iteration here, or arbitrary N-D slices via PatchSampler) reads and decompresses only the requested chunks.

Parameters:

  • path (str)

  • component (str | None)

  • structure (SampleStructure)

  • schema (Schema | None)

class HDF5ArraySource(path, dataset, *, structure=EXCHANGEABLE, schema=None)[source]

Bases: _ArrayVolumeSource

Lazy connector over an HDF5 dataset within a file.

Optional: requires h5py (pip install mixle[arrays]). The file handle is opened read-only and kept resident (h5py datasets are themselves lazy: indexing reads only the requested slice); call close() (or use as a context manager) to release it.

Parameters:

  • path (str)

  • dataset (str)

  • structure (SampleStructure)

  • schema (Schema | None)

close()[source]

Close the underlying HDF5 file handle. Idempotent.

Return type:

None

class MemmapArraySource(path, dtype, shape, *, mode='r', structure=EXCHANGEABLE, schema=None)[source]

Bases: _ArrayVolumeSource

Lazy connector over a numpy memmap volume (no optional dependency: pure numpy).

np.memmap maps the file into the process’s address space and only pages in the slices that are actually indexed, so this never reads the full volume into resident memory either.

Parameters:

  • path (str)

  • dtype (Any)

  • shape (tuple[int, ...])

  • mode (str)

  • structure (SampleStructure)

  • schema (Schema | None)

class PatchSampler(array, patch_size, num_patches, *, seed=0, stride=None, structure=EXCHANGEABLE, schema=None)[source]

Bases: object

Yield (patch, coords) records – fixed-size N-D patches sampled from an array – lazily.

Wraps any N-D array-like object with a .shape and N-D __getitem__ (a ZarrArraySource/HDF5ArraySource/MemmapArraySource’s .array, a raw zarr/h5py/ memmap object, or a plain numpy.ndarray). Patch placement is a deterministic function of seed: the top-left corner of each patch is drawn from a seeded numpy.random.Generator once, up front (cheap – it is only num_patches integer tuples), and never touches the underlying array until a patch is actually indexed. Iterating/indexing therefore reads only the requested patches off disk, never the full volume.

Parameters:

  • array (Any) – N-D array-like source (anything supporting .shape and N-D __getitem__).

  • patch_size (Sequence[int]) – the N-D patch extent; must have the same rank as array.shape.

  • num_patches (int) – how many (patch, coords) records to sample.

  • seed (int) – seeds the corner-placement RNG; identical seed -> identical patch sequence.

  • stride (int | None) – if given, corners are snapped to this stride along every axis (useful for aligning patches to a chunk grid); default None samples arbitrary integer corners.

  • structure (SampleStructure) – the SampleStructure of the patch stream (patches are i.i.d. draws from the volume by default, hence EXCHANGEABLE).

  • schema (Schema | None)

coords()[source]

Return the full, deterministic list of sampled patch corners (cheap – no array I/O).

Return type:

list[tuple[int, …]]

read_zarr(path, component=None, *, structure=EXCHANGEABLE, schema=None)[source]

Open a zarr array/store lazily as a DataSource.

Parameters:

  • path (str)

  • component (str | None)

  • structure (SampleStructure)

  • schema (Schema | None)

Return type:

ZarrArraySource

read_hdf5(path, dataset, *, structure=EXCHANGEABLE, schema=None)[source]

Open an HDF5 dataset lazily as a DataSource.

Parameters:

  • path (str)

  • dataset (str)

  • structure (SampleStructure)

  • schema (Schema | None)

Return type:

HDF5ArraySource

read_memmap(path, dtype, shape, *, mode='r', structure=EXCHANGEABLE, schema=None)[source]

Open a numpy-memmap volume lazily as a DataSource.

Parameters:

  • path (str)

  • dtype (Any)

  • shape (tuple[int, ...])

  • mode (str)

  • structure (SampleStructure)

  • schema (Schema | None)

Return type:

MemmapArraySource