mixle.data.sources.arrow_source module¶
Apache Arrow connectors – Parquet / Arrow / Feather into a DataSource.
Optional: requires pyarrow (pip install mixle[arrow]); reading is columnar and zero-copy where
Arrow allows. When schema is omitted, it is inferred from the file’s own Arrow schema – a
footer/metadata-only read of field types and nullability, not the row data itself, which stays
deferred to first use like every other LazySource connector. Passing an
explicit schema= skips inference entirely (and keeps even that metadata read lazy).
The inferred mapping covers Arrow’s common scalar types: signed integers -> Integer, unsigned
integers -> Count,
float16/float32/float64 -> Real, bool -> Boolean, string/large_string ->
Text, dictionary-encoded columns -> Categorical (values only – the fixed category set itself
is not reconstructed), and a nullable Arrow field -> Optional(...). Arrow types with no Mixle
counterpart (nested/list/struct, timestamp/date, binary, decimal, …) raise TypeError during
inference rather than silently guessing a type or dropping schema-wide to unvalidated – pass an
explicit schema= for tables with such columns.
- read_parquet(path, columns=None, *, structure=EXCHANGEABLE, schema=None)[source]
Read a Parquet file/dataset into a lazy DataSource of scalar/tuple records.
When
schemais omitted, it is inferred from the file/dataset’s own Arrow schema viapyarrow.dataset(a footer-only read that resolves a directory of partition files the same waypyarrow.parquet.read_tabledoes; the row data itself is still not read until the source is materialized). Pass an explicitschemato skip inference – required for files whose columns fall outside the supported mapping (see_arrow_field_type()).
- read_feather(path, columns=None, *, structure=EXCHANGEABLE, schema=None)[source]
Read an Arrow/Feather file into a lazy DataSource of scalar/tuple records.
When
schemais omitted, it is inferred from the file’s own Arrow schema viapyarrow.ipc(a footer-only read; the row data itself is still not read until the source is materialized). Pass an explicitschemato skip inference – required for files whose columns fall outside the supported mapping (see_arrow_field_type()).