mixle.substrate.interop module

External model interop for reasoner delegation.

ExternalModel wraps a generate callable from an external model, agent, hosted LLM, or remote tool. It can estimate semantic uncertainty by sampling multiple answers and clustering them through an equivalence function. The returned answer is always one of the assessed samples, never a separate, unassessed call (MXR-080-0270) – the verdict genuinely describes the text the caller actually receives.

external_action() adapts the wrapper into a reasoner Action. When the external model is not CONFIDENT – genuinely uncertain, or never calibrated at all – the action contributes no evidence, allowing the reasoner to continue or abstain.

class Confidence(*values)[source]

Bases: StrEnum

The three-state verdict on an external answer’s semantic-entropy assessment (MXR-080-0270) – a closed vocabulary, never a bare boolean: a bare confident=True cannot distinguish “assessed against a calibrated cutoff and found consistent” from “no calibration policy was ever configured, so there was nothing to compare the entropy against” – exactly the fail-open overclaim this finding fixes (mirrors FreshnessState).

CONFIDENT: a calibration policy (calibration_prompts or an explicit max_entropy) is

configured, and the returned answer’s own measured semantic entropy is at or below the cutoff.

UNCERTAIN: a calibration policy is configured, but the measured semantic entropy exceeds the

cutoff – the sampled batch (which the returned answer is itself drawn from) disagrees with itself more than the calibrated policy tolerates.

UNCALIBRATED: no calibration policy is configured at all – there is no cutoff to compare the

entropy against, so the verdict fails closed rather than silently treating a missing cutoff as “anything finite is confident”.

class ExternalAnswer(prompt, answer, entropy, state)[source]

Bases: object

An external model’s answer plus its self-measured uncertainty (semantic entropy).

answer is itself one of the samples measured for entropy (MXR-080-0270), drawn from the same batch rather than a separate, unassessed call – so state genuinely describes the returned text’s own membership in the measured cluster.

Parameters:

  • prompt (Any)

  • answer (Any)

  • entropy (float)

  • state (Confidence)

property confident: bool

True iff state is Confidence.CONFIDENT.

Never itself the source of truth – state is (MXR-080-0270): both Confidence.UNCERTAIN and Confidence.UNCALIBRATED read False here, so an uncalibrated model can no longer read as confident just because its entropy happened to be finite.

Type:

Read-only convenience view

property calibrated: bool

True unless state is Confidence.UNCALIBRATED.

class ExternalModel(generate, *, calibration_prompts=None, equivalent=None, max_entropy=None, alpha=0.1, samples=8)[source]

Bases: object

An external generate callable wrapped so each answer carries semantic-entropy UQ.

Parameters:

  • generate (Callable[[Any], Any]) – prompt -> answer (an external agent / LLM / remote tool). Called multiple times per query to measure how much its meaning varies (the uncertainty signal).

  • calibration_prompts (Any) – example prompts whose (1-alpha) semantic-entropy quantile becomes the “too uncertain” cutoff – a calibration policy. Required unless max_entropy is given directly: with neither (or with calibration prompts too few/degenerate to yield a finite quantile), no calibration policy exists at all, and every answer’s state is Confidence.UNCALIBRATED (MXR-080-0270) – never silently treated as confident just because its entropy happened to be finite.

  • equivalent (Callable[[Any, Any], bool] | None) – (a, b) -> bool meaning-equivalence for clustering samples (default: exact match).

  • max_entropy (float | None) – an explicit uncertainty cutoff, overriding the calibrated one. Must be finite and non-negative – entropy is never negative, so a non-finite or negative cutoff is always a caller error, not a valid (if strict) policy.

  • samples (int) – how many draws to sample per answer() call. The first draw becomes the returned answer; entropy is measured over the same batch it came from (MXR-080-0270).

  • alpha (float)

property max_entropy: float | None

Semantic-entropy cutoff used to decide confidence, or None if no calibration policy is configured (MXR-080-0270).

None whenever neither calibration_prompts nor an explicit max_entropy produced a finite cutoff – callers must fail closed on None, never treat a missing cutoff as “anything finite is confident” the way an unguarded entropy <= inf silently would.

answer(prompt)[source]

Sample the external model; the first draw is the returned answer (MXR-080-0270).

Every draw – the one returned and every other sample used to measure entropy – comes from the same batch, so the confidence verdict genuinely describes the returned text’s own semantic position in the cluster, rather than a separately generated call that was never actually compared to anything. Fails closed to Confidence.UNCALIBRATED when no calibration policy (calibration_prompts or an explicit max_entropy) is configured, however low the measured entropy happens to be.

Parameters:

prompt (Any)

Return type:

ExternalAnswer

confident(prompt)[source]

Whether the external model is calibrated-confident on prompt.

Fails closed (False) when no calibration policy is configured (MXR-080-0270) – an uncalibrated model is never reported confident, regardless of measured entropy.

Parameters:

prompt (Any)

Return type:

bool

external_action(model, *, name='external', cost=8.0, description='', trust_uncertain=False)[source]

A reasoner delegate action backed by a UQ-wrapped external model (see module docstring).

By default (trust_uncertain=False) the action contributes evidence only when the external model’s answer is Confidence.CONFIDENT; an UNCERTAIN or UNCALIBRATED answer yields no fragment, so the reasoner treats it as no answer rather than a guess (MXR-080-0270: a never-calibrated model is withheld exactly like a genuinely self-contradicting one – both are “we cannot vouch for this”, not “confident”). The fragment carries the model’s entropy and verdict so the trace records how sure the external source was, and why. Cost defaults high – external calls are the escalation of last resort.

Parameters:

  • model (ExternalModel)

  • name (str)

  • cost (float)

  • description (str)

  • trust_uncertain (bool)

Return type:

Any