mixle.substrate.core module

Core substrate store and item model.

The substrate is a filesystem-backed, queryable surface for typed items with provenance, scope, freshness metadata, tags, and links. Raw data, documents, model artifacts, traces, simulation outputs, ontology triples, and context packets can all be represented as SubstrateItem records.

Text and document items rank by cosine similarity over a learned embedding when available. Structured records and other items fall back to lexical, tag, and provenance matching. Higher-level retrieval and context assembly build on this single local store.

class SubstrateItem(kind, text='', payload=<factory>, provenance=<factory>, scope='local', tags=<factory>, links=<factory>, derived_from=<factory>, id=<factory>, created_at=<factory>)[source]

Bases: object

One typed, provenanced, scoped item in the substrate.

Edge typing (MXR-080-0261): links and derived_from are both lists of item ids, but they are NOT interchangeable. links is the generic, untyped KG-relation surface – “related to”, “mentions”, “co-occurs with”, anything kg_rag/multihop want to associate two items by. derived_from is narrower and load-bearing: it is the ONLY place genuine provenance/ancestry (“this item was derived from that one”) is recorded, and it is the only field verify_lineage() ever traverses. Before this field existed, lineage verification walked links itself, so a merely-related entity was indistinguishable from a true derivation parent – putting a citation-worthy relation into links silently made it certifiable ancestry. Keeping the two lists separate (rather than tagging each entry with a type) is a deliberate minimal-diff choice: every existing reader of links (kg_rag, multihop, freshness, context, governance, spaces) keeps reading exactly the same untyped list it always has, with no change to its shape or meaning.

Parameters:
to_json()[source]

Return this item as a JSON-serializable dictionary.

Return type:

dict[str, Any]

classmethod from_json(d)[source]

Build a substrate item from its serialized dictionary form.

Parameters:

d (dict[str, Any])

Return type:

SubstrateItem

class Substrate(root=None, *, secret_policy='redact')[source]

Bases: object

A local shard of the knowledge substrate: a filesystem-backed store with typed retrieval.

put / get / remove / all manage items; search retrieves the k most relevant items for a query, filtered by kind and scope, ranking text items semantically (a learned embedding over the current text corpus) and everything else lexically. save / load persist the shard as one items.jsonl under root.

Immutability contract: every item that crosses the store boundary is a defensive copy. put copies the object you hand it before storing it; get / all (and therefore search) return copies of what is stored. Mutating an object you obtained from – or are about to pass to – this store never affects it: the store’s own state, and everything derived from it (the semantic index), is independent of what any caller does with its own reference afterward. The only way to change a stored item is put() (replace wholesale) or update() (change named fields); both revalidate the result and correctly invalidate the semantic index, exactly like a fresh write.

Scope isolation contract: search(..., scope=X) ranks using an embedding index fit ONLY on scope X’s own text items (see reindex()) – a scoped query’s results, and the model that produced them, never depend on any other scope’s content, so the presence or content of items in a scope you cannot pass never leaks through a scope you can. scope=None (the default) is the one deliberate exception: it asks for every item across every scope, so its index is fit over everything, same as asking for all of it directly through all()/get() would show you.

Secret-handling contract (MXR-080-0262): every write – put(), and update(), which routes through it – scans item.text, item.payload (recursively), and item.tags for known secret shapes (mixle.substrate.security) using exactly the surface _lexical_score() serializes for search, BEFORE anything reaches self._items or either index. This is unconditional: there is no path into the store that skips it, so the redact-before-store guard safe_text() always offered is no longer opt-in. secret_policy (set at construction, or overridden per call) picks what happens when a secret is found: "redact" (the default) masks it in place so only the masked form is ever stored, embedded, or served; "reject" raises SecretPolicyError and stores nothing.

Parameters:

  • root (str | None)

  • secret_policy (SecretPolicy)

put(item, *, secret_policy=None)[source]

Add or replace an item; returns its id and schedules semantic-index rebuilds for text items.

Stores a defensive copy of item (see the class docstring’s immutability contract) – the caller’s own object is never aliased into the store, so mutating it after this call has no effect. Dirty whenever this put could change what _text_items() (the embedding index’s real inclusion rule – ANY kind with a truthy .text, not just a fixed subset) would return: the new item itself carries text, OR it replaces an item that used to. That second half matters as much as the first – clearing an existing item’s text (or replacing it with a no-text item) shrinks the indexed corpus exactly as much as adding text grows it, and without it the stale embedding for the old text would keep matching queries after the text is gone.

Secret-handling (MXR-080-0262, see the class docstring’s secret-handling contract): before anything is stored or indexed, this scans item.text, item.payload (recursively), and item.tags – the same surface _lexical_score() serializes for search – for known secret shapes, and acts per secret_policy (this call’s override, else self.secret_policy, default "redact"): "redact" masks detected secrets in place before storing; "reject" raises SecretPolicyError and stores nothing. This is unconditional – there is no way to reach self._items that skips it.

Parameters:

  • item (SubstrateItem)

  • secret_policy (Literal['redact', 'reject'] | None)

Return type:

str

add(kind, text='', **kw)[source]

Convenience: build a SubstrateItem and put() it.

Parameters:
Return type:

str

get(item_id)[source]

Return a defensive copy of the item stored as item_id, or None when it is absent.

Mutating the returned object does not change what is stored – call update() (or put() a full replacement) to make a change land, so index invalidation stays correct.

Parameters:

item_id (str)

Return type:

SubstrateItem | None

remove(item_id)[source]

Remove an item by id and return whether anything was deleted.

Parameters:

item_id (str)

Return type:

bool

all(*, kind=None, scope=None)[source]

Return defensive copies of stored items, optionally filtered by kind and scope.

As with get(), mutating a returned item never affects the store; use update().

Parameters:

  • kind (str | None)

  • scope (str | None)

Return type:

list[SubstrateItem]

update(item_id, *, secret_policy=None, **fields)[source]

Change named fields on the stored item item_id, the supported way to edit in place.

Mutating an object returned by get()/all() never reaches the store – both return defensive copies, so the store’s own state (and anything derived from it, like the semantic index) is untouched by that. update is how a change actually lands: it rebuilds the stored item with fields applied, re-runs SubstrateItem’s own validation (__post_init__, e.g. rejecting an unknown kind) via dataclasses.replace(), and routes the result back through put()’s storage path so the same dirty/index-invalidation bookkeeping AND the same secret-handling guard (see the class docstring’s secret-handling contract; secret_policy here overrides self.secret_policy for this call only) fire as they would for any other write – a change that affects what _text_items() covers (new text, cleared text, a different scope for a text item) correctly schedules a reindex instead of leaving the semantic index silently stale, and a change that introduces a secret into any field is masked (or rejected) exactly like a fresh put() would.

Returns a defensive copy of the item as actually stored (post-redaction, consistent with get()/all() – NOT the raw pre-policy fields-applied object).

Raises:

  • KeyError – item_id is not stored.

  • ValueError – fields tries to change id – update changes an item’s content, not its identity; put a new item and remove the old one for that.

  • SecretPolicyError – secret_policy="reject" (or self.secret_policy is) and the update introduces a secret anywhere in the surface – nothing is changed.

Parameters:

  • item_id (str)

  • secret_policy (Literal['redact', 'reject'] | None)

  • fields (Any)

Return type:

SubstrateItem

reindex()[source]

(Re)fit one embedding index PER visibility domain. Idempotent, lazy-called.

MXR-080-0237: a single embedder fit over every scope’s text pooled together makes a scoped query’s transform and ranking depend on inaccessible scopes’ content – a membership side channel (whether some distinctive text exists in a scope you can never read is inferable from how it shifts your OWN scoped query’s scores). Fixed by fitting one embedder per distinct scope value present, each ONLY on that scope’s own text items, plus one unrestricted index (key None) fit over everything for callers that pass scope=None – that caller is asking for every scope by construction, so pooling there asserts no isolation boundary to violate. search() picks the index matching its own scope argument, so a scope’s results and the model that produced them never see a single byte of another scope’s text.

Return type:

None

search(query, k=5, *, kind=None, scope=None)[source]

The k most relevant items to query as (item, score), filtered by kind/scope.

Text-bearing items rank by cosine similarity in the learned embedding space fit for THIS scope alone (see reindex()); when there are too few same-scope items to learn one (or for a non-text query), ranking falls back to a lexical token overlap. Structured items with no text always rank lexically over their serialized payload + tags.

k must be an exact, non-negative int (MXR-080-0236, see _require_count()): “return up to k results,” never a negative-indexing slice trick. Checked before reindex() (a potentially expensive rebuild) ever runs, so an invalid k fails fast. retrieve()’s diversified and flat merge paths, and top(), all validate through this same function, so a result count means exactly the same thing – and an invalid one is rejected exactly the same way – at every entry point across the retrieval surface.

Parameters:
Return type:

list[tuple[SubstrateItem, float]]

save(root=None)[source]

Persist the shard to {root}/items.jsonl (one item per line), atomically.

The full snapshot is written to a sibling temp file in target first, fsynced, then published with a single os.replace – mirrors mixle.task.artifact._atomic_json_dump() and mixle.system.registry.Registry._write_index()’s temp-file-plus-os.replace convention. A plain open(path, "w") truncates items.jsonl BEFORE the new content is written, so any failure partway through this method (a crash, a disk-full write, a non-serializable item further down the store) would destroy the last good shard instead of just failing the save; staging the new snapshot off to the side first means the write is all-or-nothing and the previous items.jsonl is untouched by a failed attempt.

Parameters:

root (str | None)

Return type:

str

load(root=None)[source]

Load items from {root}/items.jsonl into this shard, all-or-nothing.

Every row is parsed into a private, freshly-built dict first; only once the ENTIRE file has parsed and validated cleanly does that dict become self._items. A malformed or invalid row anywhere in the file raises – naming the file, the 1-based line number, and a preview of the offending row – and leaves whatever this shard held before the call completely untouched: never a partial store holding only the rows that happened to precede the bad one.

Parameters:

root (str | None)

Return type:

None