mixle.data.sources.hadoop_source module¶
Remote-filesystem connector – HDFS / S3 / GCS files via fsspec, composing with the text/arrow readers.
Optional: requires fsspec (and its backend, e.g. s3fs/gcsfs/pyarrow for HDFS) –
pip install mixle[hadoop]. read_remote(path, format=...) opens the object store path and
delegates to the matching local reader, so a Parquet/Feather/CSV/JSON/JSONL file on S3 reads exactly
like a local one – including columns projection, applied identically to the local connectors
(arrow_source / text_source) rather than ignored.
fmt values outside this set raise ValueError immediately, before any I/O.
- read_remote(path, fmt, columns=None, *, structure=EXCHANGEABLE, schema=None, **storage_options)[source]
Read a remote (
s3:///gcs:///hdfs://) file byfmtvia fsspec + the local reader.