mixle.reason.llm module

Uncertainty quantification for LLM output on Mixle’s UQ surface.

An LLM can emit fluent text without calibrated confidence. Wrapping any generate(prompt) -> str callable, LLMUncertainty turns repeated stochastic samples into calibrated uncertainty:

  • Semantic entropy (Kuhn et al. 2023): sample the model n times, cluster the answers by meaning (not surface form), and take the entropy over meaning-clusters. High = the model disagrees with itself about what the answer is – a hallucination signal – while merely rephrasing one answer clusters to low entropy. (mixle.inference.semantic_entropy().)

  • Epistemic vs aleatoric split: draw samples under several members (paraphrased prompts, or higher temperature as a proxy ensemble); the disagreement across members is epistemic (the model is unsure), the spread within is aleatoric (the question is genuinely open). (mixle.inference.decompose_entropy().)

  • Selective answer-or-abstain (LLMUncertainty.calibrate()): calibrate a confidence threshold on labeled examples so that when the model answers, it is correct with probability >= 1 - alpha – with probability >= 1 - delta over the random calibration set itself (Geifman & El-Yaniv 2017; Angelopoulos et al. 2021 “Learn then Test”): a proper finite-sample (alpha, delta)-PAC selective-risk guarantee via an exact Clopper-Pearson bound, Bonferroni-corrected across every candidate threshold tried – not a same-sample point estimate. The guarantee is conditional on i.i.d. traffic AND the entire serving policy (generator behavior, equivalence relation, sample count) remaining exactly the calibrated one – identity checks catch rebinding, not in-place state change; pin and compare the policy token (STAT-RR22-09/RR23-09). The model abstains on questions it does not know instead of confabulating.

  • Claim-level corroboration (LLMUncertainty.assess_claims()): lexical overlap only ever establishes candidacy – that a resample is plausibly about the same thing as a claim – never support by itself; a negation/polarity check on the shared content words is what separates genuine corroboration from a resample that shares the claim’s vocabulary while disagreeing with it.

The dependency boundary is domain-neutral: it takes a plain generate callable and an equivalent relation, so it works with a local mixle.task model, an OpenAI-compatible endpoint, or a test double without a hard LLM dependency in this module.

sentence_claims(text)[source]

Split a response into atomic claims (sentence-ish units) – the default claim extractor.

Parameters:

text (str)

Return type:

list[str]

content_overlap(sample, claim, *, threshold=0.6)[source]

Simple corroboration test: does sample cover >= threshold of claim’s content words?

Counts every content word equally, so boilerplate shared across responses (“the tower is located in …”) can mask that the informative word (the city) differs. information_corroborator() fixes that by weighting words by their information content, and is the candidacy signal behind LLMUncertainty.assess_claims()’s default corroborator. Like information_corroborator(), this measures topical overlap only, never polarity: a negated sentence sharing a claim’s content words still counts as covering them. mixle.substrate.factuality layers a negation/polarity check on top of this exact function for that reason (see its Corroboration), and _default_claim_corroborator() below does the analogous thing for this module.

Parameters:
Return type:

bool

information_corroborator(samples, *, overlap=0.5)[source]

Build a corroboration test that weights each word by its information content over samples.

A word appearing in nearly every sample is boilerplate (low information, low weight); a rare word carries the actual claim (high weight). A sample corroborates a claim when it covers at least overlap of the claim’s information-weighted words – so whether the distinctive fact (a city, a number, a name) matches drives the decision, not the shared filler. Inverse-document-frequency weighting: w(word) = log((N + 1) / (df + 0.5)).

This is a CANDIDACY test, not a support test: it says nothing about polarity, so a sample that negates the claim while sharing its distinctive words still corroborates by this measure alone (content_overlap() has the identical limitation). LLMUncertainty.assess_claims()’s default corroborator (_default_claim_corroborator()) uses this as its candidacy signal, gated by a negation/polarity check – see Corroboration.

Parameters:
Return type:

Callable[[str, str], bool]

class Corroboration(*values)[source]

Bases: StrEnum

A corroborator’s verdict on one (sample, claim) pair – three-way, never a bare bool.

Mirrors mixle.substrate.factuality.Corroboration (MXR-080-0258): lexical candidacy (matching content words) is necessary but not sufficient for SUPPORTED – a contradiction can share every content word with the claim it contradicts (see the module-level comment above this section for the exact example). UNVERIFIED is a real, distinct outcome from SUPPORTED – “found no reason to doubt it” is not “confirmed” – so a caller can never mistake silence for verification.

class LLMAssessment(answer, confidence, semantic_entropy, clusters, samples)[source]

Bases: object

One prompt’s assessed answer with uncertainty.

answer is the majority meaning-cluster’s representative; confidence its cluster share in [0, 1]; semantic_entropy the nats of meaning-uncertainty; clusters the [(representative, probability), ...] distribution over meanings; samples the raw draws.

Parameters:
class Generation(text, logprob=None)[source]

Bases: object

One stochastic sample from generate, normalized: text immutably paired with its log-probability when the generator provides one (MXR-080-0296).

Before this, only LLMUncertainty.assess() split a (text, logprob) pair from a plain string – LLMUncertainty.decompose() clustered the raw tuples themselves (comparing (text, logprob) tuples for equality instead of the text, silently discarding every logprob), and LLMUncertainty.assess_claims() handed a raw tuple straight to string-only claim extractors and corroborators. LLMUncertainty.sample() now builds list[Generation] once, at the boundary, so every consumer sees the same normalized shape and a text/logprob split can no longer happen in only some call paths.

Parameters:
class ClaimAssessment(claim, support, contradicted, reliable)[source]

Bases: object

Reliability of one claim inside a response, by cross-sample corroboration.

support is the fraction of independent resamples whose corroboration verdict was Corroboration.SUPPORTED (in [0, 1]); contradicted is the fraction verdicted Corroboration.CONTRADICTED – a distinct, stronger-than-merely-unsupported signal: some resample did not just fail to back the claim, it actively disagreed (MXR-080-0295; mirrors mixle.substrate.factuality.ClaimVerdict). reliable requires both good support AND zero contradiction (contradicted == 0.0 and support >= threshold): lexical overlap with a resample is never enough by itself, since a negated resample can share most of a claim’s vocabulary while disagreeing with it. A claim the model actually knows recurs, unnegated, across samples (high support, zero contradiction); a fabricated one appears once and vanishes (low support); a claim the model contradicts itself on is flagged regardless of raw overlap.

Parameters:
class InformationAssessment(claims, reliability)[source]

Bases: object

UQ over the information content of a response: every claim scored, plus a summary.

claims is the per-claim reliability; reliability the mean support across claims (how trustworthy the response’s information is overall) – None when the response had no extractable claims at all (empty, unparseable, or evasive) rather than a vacuous 1.0 (MXR-080-0295; mirrors mixle.substrate.factuality.FactualityReceipt.grounded_fraction): there is nothing that was checked, so this is UNASSESSED, not “everything checked out.” fabricated is the claims below threshold (likely hallucinated).

Parameters:

  • claims (list[ClaimAssessment])

  • reliability (float | None)

property fabricated: list[ClaimAssessment]

Return claims assessed as unreliable.

is_reliable(threshold=1.0)[source]

True iff overall reliability meets threshold (default 1.0: every claim must be reliable).

Fails closed when there is nothing to assess: a response with no extractable claims is never reported as “reliable”, regardless of threshold – mirrors mixle.substrate.factuality.FactualityReceipt.is_grounded().

Parameters:

threshold (float)

Return type:

bool

class FactualityModel(calibrator, signal, discrimination)[source]

Bases: object

A fitted map from a per-prompt uncertainty signal to a calibrated P(answer is correct).

The signal (self-consistency, a token likelihood, …) is only a raw number; the calibrator turns it into a genuine probability of the information being correct, learned against labeled facts. discrimination (RESUBSTITUTION AUC on the fit rows, tie-correct – see _auc(); “held-out” was a false label, STAT-RR23-07) reports how much the signal actually knew about correctness – ~0.5 means the signal was unrelated to truth, no matter how confident it looked.

Parameters:
probability(prompt)[source]

Calibrated probability that the answer’s information is correct.

Parameters:

prompt (str)

Return type:

float

class LLMUncertainty(generate, *, equivalent=None, n=10)[source]

Bases: object

Calibrated uncertainty and selective prediction for any generate(prompt) -> str LLM.

Parameters:

  • generate (Callable[[str], Any]) – callable(prompt) -> str – one stochastic sample from the model.

  • equivalent (Callable[[Any, Any], bool] | None) – callable(a, b) -> bool deciding whether two answers mean the same thing (default exact match; pass a normalizer / embedding / entailment check for real text).

  • n (int) – default number of samples per prompt (must be a positive integer).

sample(prompt, n=None)[source]

Draw n stochastic responses to prompt, normalized to Generation records.

generate may return a plain string, or a (text, logprob) pair – the sequence log-probability log P(s). Normalized ONCE here, at the boundary (MXR-080-0296): every one of assess() / decompose() / assess_claims() consumes the same list[Generation] shape rather than each re-deriving (or failing to re-derive) whether generate returned tuples. generate must return the same shape on every call (plain text throughout, or (text, logprob) throughout) – a generator that mixes the two raises rather than silently letting the unrecognized shape through unflagged. n (explicit or the constructor default) must be a positive integer – see _require_positive_n().

Parameters:
Return type:

list[Generation]

assess(prompt, n=None)[source]

Sample, marginalize the string distribution over meaning classes, and report the answer.

The reported confidence is the marginal probability of the top meaning (summed over its equivalence class of strings), and semantic_entropy the entropy of that meaning marginal – not a per-string token probability.

Parameters:
Return type:

LLMAssessment

decompose(prompts, n=None)[source]

Epistemic/aleatoric split across member prompts (paraphrases of one question).

Each prompt is a member; all members’ samples are pooled to define shared meaning-clusters, then each member’s distribution over those clusters feeds decompose_entropy(). Epistemic = disagreement across paraphrasings (prompt-sensitivity / model uncertainty); aleatoric = within-member spread. Clusters on each sample’s TEXT (MXR-080-0296) – previously this clustered whatever sample() returned raw, so a (text, logprob) generator got its tuples compared for equality instead of their text, and every logprob was silently discarded.

Parameters:
Return type:

UncertaintyDecomposition

assess_claims(prompt, *, extract=None, corroborates=None, n=None, threshold=0.5)[source]

Score the reliability of each claim in the response by cross-sample corroboration.

Finer-grained than assess(): a response can be internally consistent (low semantic entropy) yet contain one fabricated fact. This decomposes the response into claims and checks each unit of information separately – a claim the model knows recurs, unnegated, across independent resamples; a hallucinated one appears once; a claim a resample actively disagrees with is flagged regardless of how much vocabulary it shares with that resample. This is UQ on the information in what is said, not just on the answer as a whole.

Parameters:

  • prompt (str) – the query.

  • extract (Callable[[str], Sequence[str]] | None) – response -> [claim, ...] (default sentence_claims()).

  • corroborates (Callable[[str, str], Corroboration | bool] | None) – (other_sample, claim) -> Corroboration – does a resample support the claim? (default: information_corroborator()’s overlap as candidacy, gated by a negation/polarity check – see _default_claim_corroborator(); pass an entailment/NLI check for real text.) A plain bool-returning callable is still accepted for compatibility: True -> SUPPORTED, False -> UNVERIFIED, never CONTRADICTED (a bare bool cannot express disagreement).

  • n (int | None) – number of samples (the first is the response scored; the rest corroborate). Must be a positive integer – see _require_positive_n().

  • threshold (float) – support below which a claim is flagged as unreliable/fabricated.

Return type:

InformationAssessment

calibrate(examples, *, correct=None, alpha=0.1, delta=0.05, n=None, policy_token=None)[source]

Calibrate a selective-risk threshold with a finite-sample (alpha, delta)-PAC guarantee.

For each labeled example, the model’s answer (majority meaning-cluster) and its confidence are computed; correct(answer, gold) (default the equivalent relation) marks it right or wrong. The threshold is chosen by _selective_risk_threshold() (MXR-080-0293): the smallest confidence tau (the most inclusive answered set) from a pre-specified 1001-point candidate grid whose exact Clopper-Pearson upper confidence bound on the selective error rate – Bonferroni-corrected across the whole grid – is <= alpha.

Statistical guarantee and assumptions: assuming the calibration (prompt, gold) examples and future queries are i.i.d. draws from the same distribution, AND the entire stochastic serving policy – the generator’s conditional output law, the equivalence relation’s behavior, and the per-prompt sample count – remains BEHAVIORALLY IDENTICAL to the calibrated one, with probability >= 1 - delta over the randomness of the calibration set, the deployed threshold’s TRUE selective risk (P(wrong | confidence >= threshold) on a fresh query) is <= alpha. The behavioral condition is load-bearing and only PARTLY enforceable: answer() refuses rebound callables and sample-count overrides (STAT-RR21-17), but IN-PLACE state mutation inside the same callable passes any identity check – an exact-wheel probe that flipped generator.mode served selective risk 1.0 under a retained 0.10 certificate (STAT-RR22-09). Pin the model identifier, prompt template, decoding parameters, and equivalence version, pass them as policy_token, and recalibrate whenever any of them changes; the token is stored on the certificate and echoed so serving infrastructure can compare it against the live deployment. This is a proper finite-sample (alpha, delta)-PAC guarantee (Geifman & El-Yaniv 2017 “Selective Classification for Deep Neural Networks”; Angelopoulos et al. 2021 “Learn then Test”) – NOT a same-sample point estimate. The previous implementation picked the smallest threshold whose SAME-SAMPLE empirical error happened to be <= alpha, with no correction for either (a) the calibration set being finite, or (b) implicitly searching over every candidate threshold on that same data – so a small calibration set could approve a threshold after one lucky correct example, with no relationship to the advertised 1 - alpha guarantee. delta is the honest second parameter that same-sample selection omitted entirely: no finite calibration set can certify a threshold with zero failure probability.

If no threshold’s bound clears alpha (too little calibration data for the requested alpha/delta), the threshold is set just above the maximum observed confidence, so answer() abstains on everything rather than deploying an uncertified threshold.

Parameters:

  • examples (Sequence[tuple[str, Any]]) – labeled (prompt, gold_answer) pairs to calibrate against; must be non-empty.

  • correct (Callable[[Any, Any], bool] | None) – (answer, gold) -> bool (default the equivalent relation). The return value must be a real Boolean or an exact 0/1 integer; anything else raises rather than being truthiness-coerced (bool("false") is True).

  • alpha (float) – target selective risk (miscoverage), in the open interval (0.0, 1.0) – the guarantee is on the TRUE risk, not this calibration set’s empirical risk.

  • delta (float) – failure probability of the guarantee itself, in the open interval (0.0, 1.0) – the guarantee holds with probability >= 1 - delta over the random calibration set.

  • n (int | None) – samples per prompt (passed to assess()).

  • policy_token (str | None)

Return type:

LLMUncertainty

certified_policy_token()[source]

The caller-pinned policy version the certificate covers, or None if none was supplied.

STAT-RR22-09: callable identity cannot bind behavior; this token is the caller’s own record of the model/prompt/decoding/equivalence version, stored at calibration for infrastructure-level comparison against the live deployment.

Return type:

str | None

answer(prompt, n=None)[source]

Answer prompt if confident enough, else None (abstain).

Requires a prior calibrate(). Returns the LLMAssessment when confidence >= threshold (so the answer meets the calibrated (alpha, delta) selective-risk guarantee), else None.

The guarantee covers exactly the CALIBRATED policy – generator, equivalence relation, and per-prompt sample count – so serving under any other is refused rather than silently keeping the certificate (STAT-RR21-17: an n=1 override on an n=2 certificate answered 20,000/20,000 fresh queries at selective risk 0.6015 against the certified 0.10; a mutated equivalence reached 0.604 and a swapped generator 1.0). WHAT THE REFUSALS CAN AND CANNOT CATCH (STAT-RR22-09): rebinding and count drift are caught by identity checks; IN-PLACE mutation of state inside the same callable is not detectable from here – a flipped generator.mode passed the guard and served risk 1.0 under the retained 0.10 certificate – so the certificate is conditional on the serving policy remaining behaviorally identical, and certified_policy_token() exposes the caller-pinned model/prompt/decoding/version token for infrastructure-level comparison. To serve a different policy, calibrate it: build the object with that generator/equivalence/n and call calibrate() again. assess() remains available at any n – it claims no certificate.

Parameters:
Return type:

LLMAssessment | None

fit_factuality(examples, *, signal=None, correct=None, method='isotonic', n=None)[source]

Learn a calibrated P(answer is correct) from a raw signal, on labeled (prompt, gold).

The model’s raw confidence (its self-consistency, or a token likelihood) is not a probability that the information is correct – it can be systematically over/under-confident, or unrelated to truth. This fits a ProbabilityCalibrator mapping the signal to the empirical correctness rate, so the output is a probability of the information being right. discrimination is the RESUBSTITUTION (fit-sample) tie-corrected AUC of signal vs correctness – the same rows that fit the calibrator, so it is optimistic and unstable at small n (STAT-RR23-07: a 20-row independent-null draw read 0.96 there while 100,000 fresh rows measured 0.4997); measure discrimination you intend to report on rows this fit never saw. correct must return a REAL Boolean or exact 0/1 integer – the correctness verdicts are the calibration TARGET, and truthiness coercion let a callback returning the string “false” turn an always-wrong generator (actual correctness 0.0) into probability(...) == 1.0; the same refusal already guards calibrate().

Parameters:

  • examples (Sequence[tuple[str, Any]]) – labeled (prompt, gold_answer) pairs.

  • signal (Callable[[str], float] | None) – prompt -> float raw score (default: the self-consistency confidence from assess()).

  • correct (Callable[[Any, Any], bool] | None) – (answer, gold) -> bool (default the equivalent relation).

  • method (str) – calibration map – "isotonic" or "platt".

  • n (int | None) – samples per prompt.

Return type:

FactualityModel