mixle.substrate.ingest module

Ingest adapters: pull the stores the ecosystem already has into the knowledge substrate.

The substrate does not want copies of everything – it wants TYPED, PROVENANCED, RETRIEVABLE entries pointing at what already exists. These adapters turn the three stores mixle already keeps into SubstrateItem s:

  • ingest_documents – raw text / passages -> kind="text" items (the RAG corpus).

  • ingest_artifacts – a registry directory of deployed model/dataset artifacts (each a manifest.json) -> kind="artifact" items whose text surface is the manifest summary and whose payload references the artifact path (so lineage + retrieval work without copying weights).

  • ingest_traces – a harvested .jsonl (the /feedback / agent-trace format) -> kind="trace" items (input->answer pairs for retrieval and curriculum).

Every item carries provenance (source path, kind, ingest time) so the reasoner can cite where a piece of knowledge came from.

Integrity, identity, and failure visibility (MXR-080-0268/0269): every function below returns an IngestReceipt rather than a bare list[str] of successful ids. Three problems this fixes together, because they share one root cause – nothing observed a per-input failure or recorded what would let a LATER reader detect one:

  • Missing integrity data. When a doc/record/trace’s source (or an artifact’s manifest.json) names a real, readable file, its provenance now carries a full, algorithm-labelled content_hash – byte-for-byte the same digest content_hash() produces, so check_freshness() can actually detect a later byte change instead of silently having nothing recorded to compare against.

  • Duplicate records instead of revisions. Every item’s id is now derived deterministically from (kind, source, position) (see _stable_id()) rather than a fresh random UUID every call. Re-ingesting the SAME source reuses the SAME ids, so put()’s replace-by-id semantics turn a repeat ingestion into a REVISION of the existing records (a bumped provenance["revision"] counter, see _put_revisioned()) instead of an ever-growing pile of duplicates that all claim to be the current copy.

  • Silent data loss. A malformed line, a manifest that is valid JSON of the wrong shape (a list where an object was expected), or any other per-input failure used to be swallowed by a bare continue (or, worse, allowed to raise and abort the WHOLE batch after earlier inputs had already been stored – MXR-080-0269’s partial-write-then-crash). Every ingest_* function now validates each input’s shape BEFORE constructing/storing anything for it, and wraps that input’s handling so nothing it does can abort its siblings: one bad input becomes one named entry on the returned receipt’s .failed, and every other input in the same batch still lands. A caller can always tell “N succeeded, M silently failed” apart from “N+M succeeded” – and can always tell exactly what happened to every single input, never just “the call didn’t raise.”

class IngestFailure(index, source, error)[source]

Bases: object

One batch input that did NOT make it into the substrate (MXR-080-0268/0269): its position in the batch, the source it came from, and why. Every ingest_* function reports these on its returned IngestReceipt instead of silently skipping the input – “N succeeded, M silently failed” must never be indistinguishable from “N+M succeeded”.

Parameters:
class IngestReceipt(ids=(), failed=())[source]

Bases: list

The ids of every item actually ingested, in order – so len(receipt), receipt[0], if receipt:, receipt == [...], and for item_id in receipt: all mean exactly what they meant back when every ingest_* function returned a bare list[str]. That backward compatibility is deliberate, not an oversight: unlike a fresh @dataclass receipt (e.g. FactualityReceipt), this one has EXISTING external callers that already index/measure it as a plain list of ids – mixle.reason.receipt._ingest_posterior()’s ids[0] if ids else "", mixle.scientist.Scientist.learn()’s len(ingest_documents(...)) – and both keep working completely unchanged against this type.

PLUS the structured per-input accounting a bare list could never carry (MXR-080-0268): .failed names every input that did NOT make it in, and why (see IngestFailure); .ok is True iff nothing failed. Before this fix, a malformed line or manifest was silently continue-d past (or, for a wrong-shaped-but-valid JSON value, allowed to raise and abort everything after it – MXR-080-0269) and only the ids of what DID succeed were ever returned. A caller could not tell “3 succeeded, 2 silently failed” from “5 succeeded, nothing lost.” Now it always can, without losing any of the old call sites’ simplicity.

Parameters:

  • ids (Sequence[str])

  • failed (Sequence[IngestFailure])

property ok: bool

True iff every input in this batch was ingested – no failures at all.

merge(other)[source]

Combine two receipts (ingest_file()’s .jsonl branch fans one file out to both ingest_documents() and ingest_records()): ids from both, in order, AND both .failed lists. Deliberately not done via +/+= – plain list concatenation only extends the id storage, silently dropping whichever side’s .failed entries were not self – exactly the kind of silent loss this whole fix exists to prevent.

Parameters:

other (IngestReceipt)

Return type:

IngestReceipt

ingest_documents(substrate, docs, *, source='documents', scope='local', indices=None)[source]

Add text passages to the substrate as kind="text" items. Returns an IngestReceipt (the new/revised item ids, plus any per-doc failures – see its docstring; behaves exactly like the historical bare list[str] for every existing caller).

Each doc is a string, or a {"text": ..., "tags": [...], "payload": {...}} dict for metadata. Anything else is reported on .failed rather than crashing the whole batch (MXR-080-0269).

Stable, revisioned identity and integrity (MXR-080-0268): each doc’s id is derived from (source, its position) – indices[i] when the caller supplies one (so ingest_file() can pass through a .jsonl file’s true line numbers even after splitting mixed content into docs/records), else the plain enumerate position. Re-ingesting the SAME source reuses the SAME ids, turning a repeat call into a revision of the existing records (see _put_revisioned()) rather than a fresh random-id duplicate every time. When source names a real, readable file, every doc’s provenance also carries that file’s full, algorithm-labelled content_hash (the exact digest check_freshness() later compares against) – source is often just a label with nothing to hash (e.g. the "documents" default), so no content_hash is recorded for those.

Parameters:
Return type:

IngestReceipt

ingest_artifacts(substrate, registry_root, *, scope='local')[source]

Index every deployed artifact under registry_root (dirs containing a manifest.json). Returns an IngestReceipt (behaves exactly like the historical bare list[str]).

The item’s text surface is a human summary of the manifest (kind, io, meta); its payload REFERENCES the artifact directory ({"ref": path}) rather than copying it – unchanged, since other readers (e.g. mixle.reason.receipt.decision_receipt()) depend on ref naming the directory, not the manifest file – and provenance carries the manifest’s lineage fields when present.

Integrity, identity, and validation (MXR-080-0268/0269): provenance also carries the manifest FILE’s full, algorithm-labelled content_hash (content_hash()) and its own manifest_path, for anyone who wants to verify it directly – note that because payload["ref"] (a directory) is what check_freshness() actually re-hashes, an artifact item’s freshness verdict is UNVERIFIABLE by construction (a directory cannot be read as file bytes), which is the documented, intentional behavior of that check, not a gap in this one; recording the hash here is still correct and useful (forensic verification, and a future reader can compare it against a fresh hash of the same manifest file by hand). Each artifact’s id is derived from its directory path (see _stable_id()), so re-ingesting the SAME artifact directory revises the existing record instead of duplicating it. A manifest that fails to read, fails to parse, or parses to valid JSON of the wrong shape (anything other than a JSON object) is reported on .failed – checked BEFORE any .get() is called on it and BEFORE anything is stored for it – rather than raising and aborting every artifact discovered after it (MXR-080-0269).

Parameters:

  • substrate (Substrate)

  • registry_root (str)

  • scope (str)

Return type:

IngestReceipt

ingest_traces(substrate, jsonl_path, *, source=None, scope='local')[source]

Index a harvested .jsonl of {"input": ..., "answer"/"label"/"call": ...} rows as traces. Returns an IngestReceipt (behaves exactly like the historical bare list[str]).

Integrity, identity, and validation (MXR-080-0268/0269): every row’s provenance carries the source file’s full, algorithm-labelled content_hash (one read of the whole file up front, reused for every row – all rows from the same file share the same recorded hash, since a byte anywhere in the file invalidates every row’s “as ingested” guarantee equally). Each row’s id is derived from (source, its line number), so re-ingesting the SAME file revises the existing per-row records instead of duplicating them. A line that fails to parse, or parses to valid JSON that is not a JSON object (the documented row shape requires .get()-able fields), is reported on .failed – checked BEFORE any .get() is called on it – rather than raising and aborting every row after it in the file (MXR-080-0269, the exact “trace code assumes every row is a mapping” bug).

Parameters:

  • substrate (Substrate)

  • jsonl_path (str)

  • source (str | None)

  • scope (str)

Return type:

IngestReceipt

ingest_records(substrate, records, *, source='records', scope='local', text_fields=None, indices=None)[source]

Add a sequence of records (dicts or tuples) to the substrate as kind="record" items. Returns an IngestReceipt (behaves exactly like the historical bare list[str]).

Each record’s payload is stored structured; its retrievable text surface is the text_fields (for dict records) joined, else the whole serialized record – so records are searchable by content.

Stable, revisioned identity and integrity (MXR-080-0268), mirroring ingest_documents(): each record’s id is derived from (source, its position) – indices[i] when supplied (see ingest_file()), else the plain position – so re-ingesting the SAME source revises the existing records instead of duplicating them, and a real, readable source file’s content_hash is recorded on every record’s provenance. One record’s storage failure (e.g. a rejected secret under secret_policy="reject") is reported on .failed rather than aborting the rest of the batch.

Parameters:
Return type:

IngestReceipt

ingest_file(substrate, path, *, kind=None, source=None, scope='local')[source]

Ingest a data file into the substrate. Format inferred from the extension unless kind forces it. Returns an IngestReceipt (behaves exactly like the historical bare list[str]).

.txt/.md -> one text item per non-empty line; .jsonl -> one item per JSON line (a string / {"text": ...} becomes a text item, any other object a record item); .csv -> one record item per row keyed by the header. source defaults to the file path, so every resulting item picks up that file’s content_hash and a stable, revisioned identity (see ingest_documents()/ingest_records()) automatically.

A malformed .jsonl line is reported on the returned receipt’s .failed (naming its 1-based line number) rather than silently dropped (MXR-080-0268) or allowed to abort every line after it (MXR-080-0269); the .jsonl branch preserves each row’s TRUE original line number as its stable identity even after splitting mixed content into a docs sub-batch and a records sub-batch (via IngestReceipt.merge(), which – unlike plain list concatenation – keeps both sub-batches’ .failed entries instead of silently dropping one side’s).

Parameters:

  • substrate (Substrate)

  • path (str)

  • kind (str | None)

  • source (str | None)

  • scope (str)

Return type:

IngestReceipt