mixle.substrate.factuality module

check_factuality() – turn an answer into a per-claim receipt grounded in the substrate (B3).

An LLM answer is a paragraph of assertions; some are supported by what the system actually knows and some are not. check_factuality() makes that checkable: it splits the answer into claims (mixle.reason.llm.sentence_claims()), retrieves evidence for each from the substrate, and marks the claim SUPPORTED only when retrieved evidence both scores above a floor and overlaps its content – attaching the citing item as provenance. The result is a FactualityReceipt: every claim tagged supported/unsupported with its evidence, plus the grounded fraction.

This is the knowledge-grounded twin of mixle.reason.llm.LLMUncertainty.assess_claims() (which corroborates against self-consistency samples): same claim-level discipline, but the corroborator is the substrate, so “is this answer true?” becomes “which of its claims can I cite, and which can’t I?” – the no-claim-without-provenance rule applied after the fact to any answer, whatever produced it.

class ClaimVerdict(claim, supported, score, citations=<factory>)[source]

Bases: object

One claim from an answer, marked supported or not, with the evidence that (dis)confirms it.

Parameters:
claim: str
supported: bool
score: float
citations: list[dict[str, Any]]
class FactualityReceipt(answer, verdicts=<factory>)[source]

Bases: object

A per-claim grounding of an answer against the substrate – the receipt behind ‘is this true?’.

Parameters:

  • answer (str)

  • verdicts (list[ClaimVerdict])

answer: str
verdicts: list[ClaimVerdict]
property grounded_fraction: float
unsupported()[source]

The claims the substrate could not corroborate – exactly what to flag or retract.

Return type:

list[ClaimVerdict]

is_grounded(threshold=1.0)[source]

True iff the grounded fraction meets threshold (default 1.0: every claim must be cited).

Parameters:

threshold (float)

Return type:

bool

as_dict()[source]
Return type:

dict[str, Any]

check_factuality(substrate, answer, *, extract=None, corroborates=None, min_score=0.2, k=4, scope=None)[source]

Ground each claim of answer against substrate, returning a FactualityReceipt.

Parameters:

  • extract (Callable[[str], list[str]] | None) – answer -> [claim, ...] (default mixle.reason.llm.sentence_claims()).

  • corroborates (Callable[[str, str], bool] | None) – (evidence_text, claim) -> bool deciding if retrieved evidence supports a claim (default content-overlap; pass an NLI/entailment check for stronger grounding).

  • min_score (float) – retrieval-score floor; evidence below it doesn’t count (guards tiny-embedder noise).

  • k (int) – evidence items retrieved per claim.

  • scope (str | None) – restrict retrieval to a team/access scope.

  • substrate (Substrate)

  • answer (str)

Return type:

FactualityReceipt