mixle.substrate.ingest module

Ingest adapters: pull the stores the ecosystem already has into the knowledge substrate.

The substrate does not want copies of everything – it wants TYPED, PROVENANCED, RETRIEVABLE entries pointing at what already exists. These adapters turn the three stores mixle already keeps into SubstrateItem s:

  • ingest_documents – raw text / passages -> kind="text" items (the RAG corpus).

  • ingest_artifacts – a registry directory of deployed model/dataset artifacts (each a manifest.json) -> kind="artifact" items whose text surface is the manifest summary and whose payload references the artifact path (so lineage + retrieval work without copying weights).

  • ingest_traces – a harvested .jsonl (the /feedback / agent-trace format) -> kind="trace" items (input->answer pairs for retrieval and curriculum).

Every item carries provenance (source path, kind, ingest time) so the reasoner can cite where a piece of knowledge came from.

ingest_documents(substrate, docs, *, source='documents', scope='local')[source]

Add text passages to the substrate as kind="text" items. Returns the new item ids.

Each doc is a string, or a {"text": ..., "tags": [...], "payload": {...}} dict for metadata.

Parameters:
Return type:

list[str]

ingest_artifacts(substrate, registry_root, *, scope='local')[source]

Index every deployed artifact under registry_root (dirs containing a manifest.json).

The item’s text surface is a human summary of the manifest (kind, io, meta); its payload REFERENCES the artifact directory ({"ref": path}) rather than copying it, and provenance carries the manifest’s lineage fields when present.

Parameters:

  • substrate (Substrate)

  • registry_root (str)

  • scope (str)

Return type:

list[str]

ingest_traces(substrate, jsonl_path, *, source=None, scope='local')[source]

Index a harvested .jsonl of {"input": ..., "answer"/"label"/"call": ...} rows as traces.

Parameters:

  • substrate (Substrate)

  • jsonl_path (str)

  • source (str | None)

  • scope (str)

Return type:

list[str]

ingest_records(substrate, records, *, source='records', scope='local', text_fields=None)[source]

Add a sequence of records (dicts or tuples) to the substrate as kind="record" items.

Each record’s payload is stored structured; its retrievable text surface is the text_fields (for dict records) joined, else the whole serialized record – so records are searchable by content.

Parameters:
Return type:

list[str]

ingest_file(substrate, path, *, kind=None, source=None, scope='local')[source]

Ingest a data file into the substrate. Format inferred from the extension unless kind forces it.

.txt/.md -> one text item per non-empty line; .jsonl -> one item per JSON line (a string / {"text": ...} becomes a text item, any other object a record item); .csv -> one record item per row keyed by the header. source defaults to the file path.

Parameters:

  • substrate (Substrate)

  • path (str)

  • kind (str | None)

  • source (str | None)

  • scope (str)

Return type:

list[str]