mixle.substrate.security module

Secret detection and redaction for substrate text and structured data.

Knowledge flows into the substrate from documents, traces, and tool outputs – exactly the places a leaked credential hides, and not only as free text: a payload dict (a harvested trace, a manifest, a record) carries one just as easily, and lexical retrieval serializes payload + tags into the same searchable surface as .text (see mixle.substrate.core._lexical_score). detect_secrets() scans a string for well-known secret shapes (API keys, bearer tokens, AWS keys, private-key blocks, credentials embedded in URLs, and key=value assignments of sensitive names); redact_secrets() masks them in place; item_surface() builds the same text+payload+tags surface lexical search indexes, so scan_item() / scan_substrate() sweep everything reachable through retrieval, not just .text; redact_value() is redact_secrets() recursed over a whole JSON-like structure; safe_text() is the redact-before-store guard for free text, and enforce_secret_policy() is the same guard for a whole item – the store-boundary choke point Substrate.put() / .update() route every write through, so the guard is mandatory rather than something a caller opts into.

The patterns are deliberately conservative and named – each finding says which rule matched, so a false positive is inspectable rather than mysterious. This is detection, not a vault: it catches the common leaks (a pasted key, a token in a log line) so they don’t get indexed and served, and it flags the rest for review. Redaction preserves a short prefix so a human can still recognize which key it was without exposing the secret.

class SecretFinding(rule, start, end, preview)

Bases: object

One detected secret: which rule matched, where, and a safe preview (the value stays masked).

Parameters:

• rule (str)

• start (int)

• end (int)

• preview (str)

class SecretScan(findings=<factory>)

Bases: object

The result of scanning a text: whether anything leaked and every finding.

Parameters:

findings (list[SecretFinding])

property clean: bool

Whether the scan found no secrets.

rules()

Return the sorted names of triggered secret-detection rules.

Return type:

list[str]

as_dict()

Return a JSON-serializable scan summary.

Return type:

dict[str, Any]

detect_secrets(text)

Scan text for well-known secret shapes; return a SecretScan naming each finding.

Parameters:

text (str)

Return type:

SecretScan

redact_secrets(text, *, mask='[REDACTED:{rule}]', keep_prefix=0)

Return text with every detected secret replaced by a rule-labelled mask (destructive to secrets).

keep_prefix leaves that many leading characters of the secret visible (0 = fully masked) so a reader can still tell which credential it was without recovering it.

Applied to a FIXED POINT, because one pass did not deliver what this docstring promises (MXR-080-1882). A rule may match only the credential and leave its assignment context in place: token=ghp_... masks to token=[REDACTED:github_token], which still matches the sensitive_assignment rule, so the “redacted” text was still detected as carrying a secret. That only became visible once the store began re-scanning what it was about to write; before that, the single pass simply returned text nobody checked again.

Iteration is bounded and monotone in practice – each pass replaces at least one match with a mask that is not itself that rule’s shape – but the bound is explicit rather than assumed, and a text that has not converged is refused instead of returned dirty.

Parameters:

• text (str)

• mask (str)

• keep_prefix (int)

Return type:

str

safe_text(text)

Redact-before-store guard: mask any secrets so they are never indexed or served.

Parameters:

text (str)

Return type:

str

item_surface(item)

The full text a substrate item exposes to indexed/serialized surfaces: text + payload + tags.

Mirrors mixle.substrate.core._lexical_score’s surface construction exactly (same join, same json.dumps(payload)), so this scans precisely what lexical retrieval makes reachable – a secret embedded anywhere in payload (however deeply nested; json.dumps flattens it into the same string lexical search tokenizes) or in tags is caught, not just a secret sitting in .text. Falls back to str(payload) when payload is not JSON-serializable, so a scan can never raise on data that would otherwise be storable (search’s own lexical path would raise on that same payload lazily, at query time; scanning must not be the thing that raises earlier at write time).

Parameters:

item (Any)

Return type:

str

redact_value(value, *, mask='[REDACTED:{rule}]', keep_prefix=0)

Recursively redact secrets from a stored value, over the whole surface item_surface() reads.

Redaction has to cover exactly what the scanner covers, and it did not (MXR-080-1882). item_surface serializes payload with json.dumps – which writes dictionary KEYS as well as values – and falls back to str(payload) when that fails, which stringifies sets and any opaque object’s repr. This function reached only dict values, lists and tuples, so three reachable places were detected by the scan and then stored in the clear:

• a dictionary key, e.g. {"api_key=abcdefghi": "v"}. Detected, stored intact.

• a set/frozenset, which fell to the untouched-passthrough at the bottom.

• an opaque object whose str/repr carries the secret, likewise.

Keys are redacted too, and a redaction that would collapse two distinct keys onto one masked string raises rather than silently dropping an entry – losing a payload field to sanitization is a different failure from masking one. An opaque leaf is replaced by its redacted string form only when its own string carries a secret; otherwise it passes through unchanged, since converting every opaque value to text would destroy payloads that were never at risk.

Parameters:

• value (Any)

• mask (str)

• keep_prefix (int)

Return type:

Any

scan_item(item)

Scan a substrate item’s full indexed/serialized surface (text + payload + tags) for secrets.

Parameters:

item (Any)

Return type:

SecretScan

scan_substrate(substrate, *, scope=None)

Sweep a substrate for leaked secrets and report which stored items triggered rules.

Returns {n_items, n_dirty, dirty: [{item_id, rules}]} for compatibility with existing callers; entries in dirty are the items that matched one or more secret-detection rules.

Parameters:

• substrate (Any)

• scope (str | None)

Return type:

dict[str, Any]

exception SecretPolicyError(scan, *, item_id=None)

Bases: ValueError

Raised when secret_policy="reject" rejects a write because a secret was detected.

Nothing is written when this is raised – the caller’s item never reaches the store.

Parameters:

• scan (SecretScan)

• item_id (str | None)

Return type:

None

enforce_secret_policy(item, *, policy='redact')

The store-boundary secret guard: scan every indexed/serialized field of item – text, payload (recursively), and tags, i.e. exactly item_surface() – and act on what’s found.

This is the single choke point Substrate.put() / .update() route every write through, so the redact-before-store guard is no longer optional: previously safe_text() existed but nothing called it, and scanning (scan_item() / scan_substrate()) only ever looked at .text, silently missing a secret embedded in payload (reachable through lexical retrieval’s own json.dumps(payload) serialization – see mixle.substrate.core._lexical_score) or in tags.

Parameters:

• item (Any) – a substrate item – read via .text / .payload / .tags and, when a secret is found under policy="redact", rebuilt with dataclasses.replace(), so any dataclass with those three fields works, not only SubstrateItem.

• policy (Literal['redact', 'reject']) – "redact" (default) returns a sanitized copy with every detected secret masked in place (text via safe_text(), payload via redact_value(), tags via safe_text() per entry) – so a secret can be written, embedded, and served back only in masked form, never in the clear. "reject" raises SecretPolicyError instead of returning anything – nothing is written.

Returns:

(item_to_store, scan) where scan is the pre-redaction SecretScan (what was found before any masking, empty when nothing triggered) so a caller can log/report what fired even when policy="redact" already fixed it up. When nothing is found, item_to_store is item (no copy is made) and scan.clean is True.

Raises:

• SecretPolicyError – policy="reject" and a secret was detected anywhere in the surface.

• ValueError – policy is not one of SECRET_POLICIES.

Return type:

tuple[Any, SecretScan]