mixle.inference.production.serving module

Production scoring with activity + computation logging and health/problem reporting.

A Service wraps a fitted model (loaded directly or from a Registry alias) and scores production batches, recording every computation – record count, wall time, mean log-likelihood, how many records were unscorable (outside the model’s support), and how many were unavailable (the model could not be reached to ask) – to an in-memory activity log (and optionally a JSONL file). health() summarizes recent activity so problems (rising unscorable rate, model unavailability, falling log-likelihood, slow batches) are visible; with a reference sample set it can also flag drift.

Unscorable and unavailable are deliberately kept distinct. A record legitimately outside the model’s support scores as a non-finite log-density that the model itself computes and returns, no exception involved. A configured availability exception raised while asking the model – by default a failed network call or backend timeout – means the model was unavailable: an outage, not a scored observation. Service.score() isolates only that explicit failure mode as "model_unavailable" so an outage is never silently folded into an ordinary -inf indistinguishable from a genuine one. Other exceptions propagate as implementation or request errors instead of being masked by compatibility retries.

The batch call failing is itself a distinct, separately-tracked degradation (mode "batch_unavailable"). When seq_log_density raises a configured availability exception, every record in that call is retried individually – and a record can recover (score normally through the per-record path) even though the batch path failed for it. Left untracked, that retry would be invisible on precisely the calls where it matters most: a batch endpoint that fails on every call but whose per-record fallback always succeeds would leave no trace in the activity log or health() at all, since every record still ends up scored and finite – nothing looks unscorable or unavailable, even though every call is silently paying for a full per-record retry. n_batch_unavailable / batch_unavailable_rate count records exposed to that fallback regardless of whether the retry itself succeeded, so a degrading batch endpoint is visible before it starts actually losing data – which is what unavailable_rate covers.

class Service(model, *, name=None, reference=None, log_path=None, keep=1000, availability_errors=(ConnectionError, TimeoutError))[source]

Bases: object

A deployed model that scores batches and logs each computation for monitoring.

Parameters:
classmethod from_registry(registry, name, *, alias=None, **kw)[source]

Load the model an alias points at in registry and serve it (carrying its provenance header).

alias follows Registry.current() exactly: naming one requires it to exist (an absent or misspelled alias raises rather than serving an unpromoted version), while omitting it takes the promoted production model if there is one and the latest registration otherwise.

Parameters:
Return type:

Service

score(records)[source]

Return per-record log-densities and log the computation (timing, mean log-lik, unscorable count, unavailable count, batch-unavailable count).

Records affected by a model_unavailable failure (see module docstring) still score -inf in the returned array – preserving its shape/dtype contract for callers – but are counted separately as n_unavailable in the logged event and surfaced as unavailable_rate by health(), rather than being indistinguishable from a genuinely-scored -inf.

The batch call itself failing is tracked separately again, as batch_unavailable: every record in a batch whose seq_log_density call raised is retried individually, and is counted in n_batch_unavailable / batch_unavailable_rate regardless of whether that per-record retry recovers a real score or itself fails into n_unavailable – so a batch endpoint that is failing over to the (much slower) per-record path on every call remains visible even when every individual retry happens to succeed and no record is actually lost.

Raises:

ValueError – if the model’s batch output does not have exactly one log-density per input record – a cardinality mismatch is a contract violation to reject, not a shape to silently accept. Raised only for a batch call that succeeded with the wrong shape; it is not itself treated as a reason to retry per-record.

Parameters:

records (Any)

Return type:

ndarray

check_drift(records)[source]

Drift of records versus the service’s reference sample (requires a reference).

Parameters:

records (Any)

Return type:

Any

health(*, window=100)[source]

Summary of the most recent window scoring events – throughput, mean log-likelihood, the unscorable rate (the production problem signal), the unavailable rate (the subset of unscorable records caused by the model being unreachable – an outage – rather than legitimately scored), and the batch-unavailable rate (records whose batch call failed and had to be retried per-record, whether or not that retry itself recovered a score – always >= the unavailable rate; see score()).

mean_loglik is the mean over RECORDS, not over events: it is the total finite log-density divided by scored_records, so it does not move when the same scores arrive in differently sized batches. Averaging each event’s own batch mean made one record with log-likelihood 0 followed by 100 records averaging -10 report -5.0 instead of the true -9.901, which let batch shape alone manufacture or mask a degradation. Unscorable records are excluded from both the numerator and scored_records (they are counted by unscorable_rate instead); an all- unscorable window reports mean_loglik=None rather than a number derived from no evidence.

Parameters:

window (int) – how many of the most recent scoring events to summarize; an exact positive integer. 0 used to select the whole history ([-0:] is a full slice) and a negative value selected a surprising suffix, both silently.

Raises:

  • TypeError – if window is not an exact integer (bool included).

  • ValueError – if window is not positive.

Return type:

dict