mixle.inference.production.registry module

A versioned model registry: store fitted models + their provenance, list versions, promote/swap.

A filesystem-backed store so a production system can register every fitted model (with its Header), list and load any version, and promote a chosen version to an alias (e.g. "production") – the swap point a serving layer reads from. Models serialize through mixle.utils.serialization (the safe registry-keyed JSON); headers are plain JSON dicts.

class Registry(root)[source]

Bases: object

A directory of named models, each with numbered versions and movable aliases.

Parameters:

root (str)

names()[source]

Registered model names.

Return type:

list[str]

versions(name)[source]

Version ids for name in registration order (v1, v2, …).

Parameters:

name (str)

Return type:

list[str]

register(model, name, *, header=None, metadata=None, expected_tip=_UNSPECIFIED)[source]

Store model under name as a new version; return its version id.

expected_tip makes the write conditional: the current latest version must equal it (or the name must be empty when it is None) at the moment of allocation, checked under the same lock that allocates the version. A caller extending a lineage passes the tip it adopted, and a concurrent writer that got there first makes this call refuse instead of persisting a sibling successor.

header defaults to model.header if present. The model is serialized with the safe mixle registry; the header (a Header or dict) and metadata are stored alongside.

Allocating the next version number is a read-modify-write over versions(name): two concurrent callers (two threads, or two processes) can both read the same existing versions, both compute the same “next” number, and both write it – one write silently clobbers the other, with no error to either caller. To prevent that, the number allocation and the write are serialized per model name with an fcntl.flock on a lock file in the model’s directory (held for both the read and the write, so a second writer waits and is correctly given the NEXT free number rather than racing for the same one). The write itself additionally uses O_CREAT | O_EXCL as an independent, belt-and-suspenders guard: if a file for that version exists anyway (e.g. flock is a no-op on the underlying filesystem), it raises instead of silently overwriting the earlier registration.

The version file is also published atomically: the payload is written to a private temp file in the model’s directory and fsynced before it is ever linked to the final <version>.json path (via os.link, after which the temp name is dropped). A failure anywhere in between – serialization raising partway through json.dump, the disk filling up, the process crashing – therefore never leaves a truncated or corrupt version file on disk: the file only ever exists fully-formed or not at all. This mirrors the temp-file-plus-atomic-publish idiom used by mixle.task.artifact._atomic_json_dump() and mixle.system.registry’s _write_index, except the publish step uses os.link rather than os.replace: this write must still refuse to clobber an existing (or concurrently-claimed) version file, and os.replace would silently overwrite one – reopening the bug the O_CREAT | O_EXCL guard above fixes. Unlike writing directly into the O_CREAT | O_EXCL-opened file, a failed attempt here also leaves nothing behind at path, so the same version number stays cleanly retriable instead of being permanently stuck behind a corrupt file that neither parses nor can be overwritten.

Parameters:
Return type:

str

checkpointer(name, *, every=1, resume=True, trust_code=False)[source]

Return an optimize(on_step=...) callback that snapshots the model under name every every iterations (recording the iteration + log-density in the version metadata).

Each checkpoint records its model model_hash and the previous checkpoint’s parent_hash, so the saved snapshots form a verifiable chain (see verify_chain()). Resume an interrupted run from the latest checkpoint:

reg = Registry("ckpts")
optimize(data, est, on_step=reg.checkpointer("run", every=5))
model, _ = reg.get("run")              # latest checkpoint
optimize(data, est, prev_estimate=model)   # continue training

A checkpointer created for a name that already holds a VERIFIABLE chain resumes it: it adopts that chain’s run identity, links its first new checkpoint to the existing tip, and continues the iteration count, so appending recovered work leaves verify_chain true. Pass resume=False to deliberately root a new lineage instead. trust_code is only used to re-load models while checking whether the existing chain is adoptable, and carries the same meaning as in get().

Parameters:
Return type:

Callable[[Any], None]

get(name, version='latest', *, trust_code=False)[source]

Load (model, header) for a version ("latest" = highest-numbered).

A registered model containing a NeuralLeaf-family component embeds its weights as a pickle blob (see mixle.models._neural_serial); deserializing that executes code, so get requires trust_code=True for such an entry – trust the registry root, not just the JSON extension. A pure-statistical entry loads either way.

Parameters:
Return type:

tuple[Any, dict | None]

header(name, version='latest')[source]

Just the provenance header of a version (no model deserialization).

Parameters:
Return type:

dict | None

metadata(name, version='latest')[source]

Just the metadata of a version (no model deserialization) – e.g. a checkpoint’s iteration.

Parameters:
Return type:

dict

record_digest(name, version='latest')[source]

Content digest of the exact persisted version envelope, re-verified on load.

A content digest, not a signature: it detects corruption and out-of-band edits of the model alone, not a writer with directory access who rewrites the model and every digest together (the same limit mixle.lifecycle.Model.load() states); integrity against that threat needs a signature or a copy of the digests kept outside the directory.

Parameters:
Return type:

str

promote(name, version, alias='production')[source]

Point alias (e.g. "production") at version – the atomic model swap.

Written via a temp file + os.replace in the same directory (same filesystem, so the rename is atomic): a concurrent reader of current() either sees the old alias target or the new one, never a truncated/partial write, and a crash mid-write leaves the old alias file untouched rather than corrupted.

Parameters:
Return type:

None

current(name, alias=None, *, trust_code=False)[source]

Load the model an alias points at.

Two deliberately different behaviours, split on whether the caller NAMED an alias:

  • current(name) – no alias requested. Resolves the default "production" alias if it exists and falls back to latest if it does not. This is the bootstrap path: a registry with registrations but no promotion yet still serves something.

  • current(name, "production") – an alias was explicitly requested. It must exist; an absent alias raises KeyError. A caller that names an alias is asserting “serve the version promoted to this alias”, and quietly substituting latest answers a different question with an UNPROMOTED model – which also silently defeats a rollback, since rolling an alias back to a known-good version has no effect on a caller whose alias name is misspelled or was never created (SYS-05).

See get() – trust_code is required in the same way and for the same reason.

Parameters:
Return type:

tuple[Any, dict | None]

verify_chain(name, *, trust_code=False)[source]

Verify the persisted checkpoint lineage for name (see checkpointer()).

Requires every version to carry a complete digest-chained lineage record. Checks exact parent versions, record digests, model hashes, transition digests, run identity, and iteration order, and re-hashes each loaded model. Missing or partial lineage is unverified and returns False.

Every digest is an unkeyed content digest recomputed by this module’s own helpers, so the chain proves that the stored versions are consistent with each other and unchanged since they were written, not who wrote them: a writer with directory access can replace a model and recompute every digest, and the chain verifies. Guard against that threat with a signature or a copy of the tip digest kept outside the directory.

See get() – trust_code is required in the same way and for the same reason: a chain that contains a NeuralLeaf-family checkpoint must be loaded to be re-hashed, so verifying it is subject to the same code-execution trust gate as loading it directly.

Parameters:
Return type:

bool