mixle.data.encoded_io module

Serialize encoded (seq_encode) data to disk, so an expensive encode is done once and reused.

seq_encode turns raw records into an encoder-specific payload (nested tuples of NumPy arrays). Fitting re-encodes every call; for large datasets that is the dominant cost. save_encoded/load_encoded persist the encoded payload (with a content digest and the encoder’s identity) so subsequent fits load it directly. The body is pickle (the payloads are internal numeric structures with no safe-JSON registry representation); the header carrying the digest and the encoder’s structural signature (str(encoder)) is plain JSON, deliberately never pickle, so reading it cannot itself execute code – only the digest-verified body is ever unpickled, and only after its digest is checked.

The digest covers the whole envelope – every header field (currently just the encoder signature) plus the body – not the body alone. An earlier version hashed only the body, so the header’s encoder field could be edited on disk without touching the body or the stored digest, and load_encoded would still accept the payload under the forged identity (MXR-080-0052). Header fields are folded into the digest via mixle.data.hashing._canonical()’s self-delimiting canonical encoding – the same helper dataset_hash/model_hash use – so on-disk JSON key order or whitespace can never produce a false “corrupt” mismatch, and the header/body split can never be ambiguous with a different split producing the same bytes. The current on-disk magic is PSPENC3; it adds mandatory, versioned encoder binding and rejects older unbound artifacts rather than treating an opaque payload as checked reusable data. A file written by the older, narrower-digest code is rejected outright as an unrecognized format rather than failing the new digest check in a way that reads like corruption.

This digest is corruption-detection, not authentication: it is computed from and stored inside the same file, so it catches truncation/bit-rot/header-tampering but cannot prove the file was not replaced wholesale by whoever could already write to path. A forged artifact carries a valid digest for its own contents, a matching encoder signature, and a body that executes on load. load_encoded therefore requires an explicit trusted=True (MXR-080-1873): the decision belongs at the call site, where the provenance of path is known, not in a docstring the caller may never read. This mirrors what mixle.lifecycle.Model.load() documents for its own pickle-format artifacts.

save_encoded(encoded, path, *, encoder=None)[source]

Write encoded (the output of encoder.seq_encode(...)) to path; return its hex digest.

encoder is required and records the encoder’s versioned structural signature – not just its class name – so a load against a structurally different encoder is flagged. Comparing class names alone would let e.g. a one-field CompositeDataEncoder be accepted for a two-field CompositeDataEncoder request, since both share the same class name. The returned digest (also stored in the header) covers this signature as well as the body – see _envelope_digest().

Parameters:
Return type:

str

load_encoded(path, *, encoder=None, trusted=None)[source]

Load encoded data written by save_encoded(), verifying its integrity digest.

The header is parsed as JSON (never pickle) and the envelope digest – header fields and body together, see _envelope_digest() – is checked BEFORE the body is unpickled, so a truncated, corrupted, or header-tampered file is rejected before any deserialization runs. The required encoder argument’s structural signature must match the one recorded at save time (else ValueError). This deliberately checks more than the encoder’s class: a DataSequenceEncoder subclass is expected to make __str__ structural (mirroring what its __eq__ checks) – e.g. CompositeDataEncoder.__str__ recurses into its component encoders and so reflects field count, and DiagonalGaussianDataEncoder.__str__ includes its dim – so a shape mismatch under the same class name (e.g. a one-field vs. two-field CompositeDataEncoder) is caught, not just a mismatch in class.

trusted=True is required, and is the caller’s statement that path is under their control. The body is pickle, so loading it is equivalent to executing whatever is in the file, and NONE of the checks above change that: the digest is computed from and stored inside the same file, so it detects truncation and header tampering but is worthless against a file replaced wholesale – a forged artifact carries a valid digest for its own contents and a matching encoder signature, and executes on load (MXR-080-1873). The module docstring said as much and the parameter did not exist, so the warning was addressed to whoever read the source rather than to whoever called the function. Making it an argument puts the decision at the call site, where the provenance of path is actually known.

Parameters:
Return type:

Any