mixle.data.sources.hadoop_source module

Remote-filesystem connector – HDFS / S3 / GCS files via fsspec, composing with the text/arrow readers.

Optional: requires fsspec (and its backend, e.g. s3fs/gcsfs/pyarrow for HDFS) – pip install mixle[hadoop]. read_remote(path, format=...) opens the object store path and delegates to the matching local reader, so a Parquet/CSV/JSONL file on S3 reads exactly like a local one.

read_remote(path, fmt, columns=None, *, structure=EXCHANGEABLE, schema=None, **storage_options)[source]

Read a remote (s3:// / gcs:// / hdfs://) file by fmt via fsspec + the local reader.

Parameters:

  • path (str)

  • fmt (str)

  • columns (list[str] | None)

  • structure (SampleStructure)

  • schema (Schema | None)

  • storage_options (Any)