Uncertainty

mixle has several uncertainty surfaces, all with the same design bias: uncertainty should change behavior. It should decide whether to answer, escalate, collect more data, or report which evidence source mattered.

This page covers:

• uncertainty over LLM answers;

• claim-level reliability for generated text;

• epistemic versus aleatoric decomposition;

• the uq dispatcher for fitted models, point predictors, ensembles, and LLM-like callables;

• conformal answer-or-abstain behavior;

• cross-modal latent evidence fusion;

• calibrated task cascades.

Unified uq Dispatcher

mixle.inference.uq provides one front door when the caller owns a heterogeneous predictor and wants Mixle to choose an uncertainty route from the object’s capabilities.

from mixle.inference import uq

fitted_uq = uq(model, training_rows)
interval_uq = uq(point_predictor, (x_cal, y_cal), alpha=0.1)
llm_uq = uq(generate, example_prompts, alpha=0.1)

UQResult exposes method-specific accessors:

• sample_models and credible_interval for fitted Mixle models through a Laplace parameter posterior;

• interval and epistemic_std for point predictors or ensembles through split conformal calibration;

• semantic_entropy and confident for LLM-style generation callables.

Use the specialized APIs below when you already know the exact uncertainty method. Use uq when the application boundary should accept several kinds of predictor behind one call.

LLMUncertainty

LLMUncertainty wraps any stochastic callable:

generate(prompt) -> answer

It samples multiple answers, clusters them by meaning, and returns the majority meaning plus a confidence and semantic entropy.

from mixle.reason import LLMUncertainty

def equivalent(a, b):
    return str(a).strip().lower() == str(b).strip().lower()

llm_uq = LLMUncertainty(generate, equivalent=equivalent, n=20)
assessment = llm_uq.assess("Which city is the Eiffel Tower in?")

print(assessment.answer)
print(assessment.confidence)
print(assessment.semantic_entropy)
print(assessment.clusters)

High confidence means most samples fell into the same meaning cluster. High semantic entropy means the model is disagreeing with itself about the answer, not just rephrasing it.

These quantities are decision signals, not truth guarantees. Use retrieval, tools, labels, or human review when factual correctness matters.

Equivalence Matters

The default equivalence relation is exact equality. That is fine for labels or normalized short answers. For prose, pass a domain-specific relation:

import re

def normalize_city(text):
    return re.sub(r"[^a-z]", "", text.lower())

llm_uq = LLMUncertainty(
    generate,
    equivalent=lambda a, b: normalize_city(a) == normalize_city(b),
    n=20,
)

In production this relation might use canonicalization, embeddings, entailment, or a task-specific parser.

Version the equivalence relation with the artifact. Changing it changes the clusters, confidence, entropy, and abstention threshold.

Epistemic and Aleatoric Split

Use several prompts as ensemble members, for example paraphrases of the same question. decompose separates disagreement across members from spread within each member.

dec = llm_uq.decompose(
    [
        "Who discovered penicillin?",
        "Name the scientist credited with discovering penicillin.",
        "Penicillin was discovered by whom?",
    ],
    n=10,
)

print(dec.epistemic)
print(dec.aleatoric)
print(dec.total)

Epistemic uncertainty is reducible uncertainty: model or prompt sensitivity. Aleatoric uncertainty is within-member ambiguity: the question or output space itself remains variable.

Calibrated Answer-or-Abstain

Sampling confidence is still only a signal until calibrated. calibrate uses labeled examples to choose the lowest confidence threshold whose answered set has empirical error at most alpha.

examples = [
    ("Capital of France?", "Paris"),
    ("2 + 2?", "4"),
]

llm_uq.calibrate(examples, alpha=0.1)

answer = llm_uq.answer("Capital of Japan?")
if answer is None:
    escalate_to_human_or_frontier_model()
else:
    print(answer.answer)

After calibration, answer returns None below the threshold. This is the important behavioral change: the LLM can abstain instead of hallucinating.

Calibration examples should match the served prompt distribution. A threshold learned on short factual questions should not be reused for extraction, planning, or legal/scientific summaries without new evidence.

Claim-Level Reliability

A response can have a stable main answer and still contain one fabricated detail. assess_claims takes one sampled response, extracts claims, and checks whether independent samples corroborate each claim.

info = llm_uq.assess_claims(
    "Summarize the contract renewal and include dates and parties.",
    threshold=0.6,
)

print(info.reliability)
for claim in info.fabricated:
    print(claim.claim, claim.support)

Defaults:

• claim extraction is sentence-like splitting;

• corroboration uses information-weighted content overlap across samples.

For serious text, pass your own extractor or entailment-based corroborator:

info = llm_uq.assess_claims(
    prompt,
    extract=my_claim_extractor,
    corroborates=my_entailment_check,
)

Uncertainty Helpers

The underlying decomposition functions are available from mixle.inference.uncertainty:

from mixle.inference.uncertainty import (
    Clustering,
    UncertaintyDecomposition,
    cluster_samples,
    decompose_entropy,
    decompose_uncertainty,
    decompose_variance,
    marginalize_meaning,
    posterior_ensemble,
    predictive_distribution,
    semantic_entropy,
)

Use them when you already have samples, probability vectors, or prediction ensembles and do not need the LLM wrapper.

UncertaintyDecomposition is the shared result object for epistemic, aleatoric, and total uncertainty summaries. predictive_distribution and posterior_ensemble build ensemble-style predictive objects from fitted or posterior models, decompose_variance performs the variance analogue of the entropy split, and marginalize_meaning aggregates probabilities over semantic clusters represented by Clustering.

Cross-Modal Reasoning

mixle.reason.reason fuses evidence into a shared latent belief. Each evidence source is a linear-Gaussian observation:

y = H z + noise,  noise ~ N(0, R)

Example:

import numpy as np
from mixle.reason import Evidence, Latent, reason

prior = Latent.vector(2, mean=0.0, var=10.0)
evidence = [
    Evidence(np.array([[1.0, 0.0]]), np.array([2.0]), 0.2, name="sensor-a"),
    Evidence(np.array([[0.0, 1.0]]), np.array([-1.0]), 0.5, name="sensor-b"),
]

ans = reason(prior, evidence)

print(ans.mean)
print(ans.interval(level=0.9))
print(ans.information_gain())
print(ans.attribution(normalize=True))

ReasonedAnswer exposes:

• posterior mean and covariance;

• credible intervals;

• total information gain;

• per-modality attribution;

• prediction-level epistemic/aleatoric variance split.

Mechanistic Latents

Latent.mechanistic builds a Gaussian prior over a trajectory constrained by a linear dynamical law. Evidence at one time step updates all time steps through the dynamics.

A = np.array([[1.0, 0.1], [0.0, 1.0]])
prior = Latent.mechanistic(A, steps=20, process_cov=0.01 * np.eye(2))

Use block_selector from mixle.reason.core to observe a specific time block of the stacked trajectory.

For mechanistic latents, record the dynamical law, process covariance, observation selector, and evidence times. Those assumptions define how one observation updates unobserved trajectory steps.

Task Calibration

For local task models, uncertainty becomes an answer/escalate decision through CalibratedTaskModel and Cascade:

from mixle.task import CalibratedTaskModel, Cascade

model = CalibratedTaskModel(student, alpha=0.1).calibrate(cal_x, cal_y)
cascade = Cascade(model, teacher)
y = cascade("new request")

See Task Distillation for the full serving workflow.

Track answered accuracy and escalation rate together. A cascade that answers too often can be unsafe; a cascade that escalates everything may be correct but not useful.

Related Reasoning Workflows

This page focuses on uncertainty behavior. See Reasoning Systems for the broader mixle.reason stack: finite-hypothesis reasoning, cross-modal retrieval with raw-data fallback, graph-producing LLMs, evidence acquisition, amortized modality encoders, and trainable cross-modal latent models.

Choosing the Right Tool

Need	Use
Does the LLM know the answer?	LLMUncertainty.assess and semantic entropy
I have a fitted model or predictor and want Mixle to pick a UQ route	mixle.inference.uq
Should the LLM answer or abstain?	LLMUncertainty.calibrate and answer
Which claim in this answer is suspect?	LLMUncertainty.assess_claims
Is uncertainty due to prompt/model sensitivity?	LLMUncertainty.decompose
How do multiple modalities update a latent?	reason, Evidence, Latent
Should a local task model escalate?	CalibratedTaskModel and Cascade

Release Evidence

For uncertainty workflows, preserve:

• sample count, prompt template, model identifier, and sampling settings;

• equivalence, claim extraction, and corroboration functions;

• calibration examples, alpha, learned threshold, and abstention policy;

• false-answer and unnecessary-abstention rates;

• evidence model assumptions for cross-modal or mechanistic latents; and

• cascade scorecards separating answered quality from escalation behavior.