LLM Uncertainty

This tutorial wraps an arbitrary LLM-like callable with mixle.reason.LLMUncertainty. The goal is not to make generation reliable by assertion. The goal is to turn repeated samples into behavior: answer, abstain, inspect disagreement, or escalate.

The wrapper needs only one callable:

generate(prompt) -> answer

For a real system, generate might call a hosted model, a local model, or an internal agent. For a test, it can be a deterministic fixture.

1. Define Equivalence

Semantic uncertainty depends on what counts as the same answer. Exact string matching is fine for normalized labels, but most LLM outputs need a domain relation.

import re
from mixle.reason import LLMUncertainty

def normalize(text):
    return re.sub(r"[^a-z0-9]+", "", str(text).lower())

def equivalent(a, b):
    return normalize(a) == normalize(b)

uq = LLMUncertainty(generate, equivalent=equivalent, n=20)

The equivalence function is part of the model. Record it with the application because changing it changes the uncertainty numbers.

2. Assess One Prompt

assessment = uq.assess("Who discovered penicillin?")

print(assessment.answer)
print(assessment.confidence)
print(assessment.semantic_entropy)
print(assessment.clusters)

confidence is the mass of the majority meaning cluster. semantic_entropy is high when samples disagree about meaning, not merely wording.

3. Decompose Prompt Sensitivity

Use paraphrases as ensemble members when you want to separate within-prompt ambiguity from prompt sensitivity.

dec = uq.decompose(
    [
        "Who discovered penicillin?",
        "Name the scientist credited with discovering penicillin.",
        "Penicillin was discovered by whom?",
    ],
    n=10,
)

print(dec.epistemic)
print(dec.aleatoric)
print(dec.total)

Epistemic uncertainty points to reducible sensitivity: prompt wording, model choice, retrieval context, or missing evidence. Aleatoric uncertainty points to ambiguity that remains inside each prompt.

4. Calibrate Abstention

Raw sample agreement is a signal, not a guarantee. Calibrate it on labeled examples before using it to decide whether to answer.

examples = [
    ("Capital of France?", "Paris"),
    ("2 + 2?", "4"),
    ("Who discovered penicillin?", "Alexander Fleming"),
]

uq.calibrate(examples, correct=equivalent, alpha=0.1)
answer = uq.answer("Capital of Japan?")

if answer is None:
    escalate_to_human_or_frontier_model()
else:
    print(answer.answer)

After calibration, answer returns None below the learned confidence threshold. This is the operational payoff: the model can decline rather than fabricate.

5. Inspect Claim Reliability

A response can have a stable headline answer and still contain unsupported details. Claim assessment extracts claims from one response and checks whether independent samples corroborate them.

info = uq.assess_claims(
    "Summarize the contract renewal and include dates and parties.",
    threshold=0.6,
)

print(info.reliability)
for claim in info.fabricated:
    print(claim.claim, claim.support)

For serious use, pass a task-specific claim extractor and an entailment-style corroborates function. The defaults are intentionally lightweight.

Validation Checklist

Before serving an LLM uncertainty wrapper:

• define and version the equivalence relation;

• calibrate on examples from the same prompt distribution;

• monitor abstention rate and false-answer rate together;

• inspect high-entropy prompts instead of averaging them away;

• route abstentions to a human, retrieval step, frontier model, or safer fallback.

Read Uncertainty for the full API and Reasoning Systems for claim reliability, graph-producing LLMs, and cross-modal evidence.