mixle.substrate.factuality module

check_factuality() – turn an answer into a per-claim receipt grounded in the substrate (B3).

An LLM answer is a paragraph of assertions; some are supported by what the system actually knows, some are actively contradicted by it, and some it simply says nothing about. check_factuality() makes that checkable: it splits the answer into claims (mixle.reason.llm.sentence_claims()), retrieves evidence for each from the substrate, and classifies every (evidence, claim) pair as SUPPORTED, CONTRADICTED, or UNVERIFIED (Corroboration) – attaching the citing item as provenance either way. The result is a FactualityReceipt: every claim tagged supported/contradicted/uncited with its evidence, plus the grounded fraction.

Lexical overlap alone is never treated as support: two statements can share almost every content word and still mean opposite things (“the drug cures cancer” / “the drug cures no cancer and does not work” overlap almost completely and disagree). Overlap only establishes retrieval CANDIDACY – that the evidence is plausibly about the same thing as the claim. The default corroborator (_default_corroborates()) additionally checks negation/polarity agreement on the content words the claim and evidence share, and calls a pair SUPPORTED only when candidacy holds and polarity agrees; a polarity mismatch on shared content is CONTRADICTED, and anything it cannot confidently place either way – including plain lack of overlap – is UNVERIFIED. This is a negation-cue heuristic, not a real entailment model: pass a genuine NLI/entailment check as corroborates for stronger grounding.

An answer with no extractable claims (empty, unparseable, or evasive) is UNKNOWN, not grounded: grounded_fraction is None and FactualityReceipt.is_grounded() fails closed (False) – there is nothing to have verified, so it is never reported as a perfect factuality result.

This is the knowledge-grounded twin of mixle.reason.llm.LLMUncertainty.assess_claims() (which corroborates against self-consistency samples): same claim-level discipline, but the corroborator is the substrate, so “is this answer true?” becomes “which of its claims can I cite, which does it contradict, and which can I do neither for?” – the no-claim-without-provenance rule applied after the fact to any answer, whatever produced it.

class Corroboration(*values)[source]

Bases: StrEnum

A corroborator’s verdict on one (evidence, claim) pair – three-way, never a bare bool.

Lexical candidacy (matching content words) is necessary but not sufficient for SUPPORTED: a contradiction can share every content word with the claim it contradicts (see the module docstring’s drug/cancer example). UNVERIFIED is a real, distinct outcome from SUPPORTED – “found no reason to doubt it” is not “confirmed” – so a caller can never mistake silence for verification.

class ClaimVerdict(claim, supported, score, citations=<factory>, contradicted=False, contradictions=<factory>)[source]

Bases: object

One claim from an answer, with what the substrate said about it and the evidence either way.

supported is True only when some qualifying evidence was judged Corroboration.SUPPORTED (entailment-aware – see _default_corroborates() – never bare lexical overlap). contradicted is True when some qualifying evidence was judged Corroboration.CONTRADICTED – a stronger and distinct signal from merely being uncited: the substrate does not just fail to back this claim, something in it disagrees. The two are not mutually exclusive: conflicting evidence for the same claim (one item backs it, another disputes it) sets both.

Parameters:
class FactualityReceipt(answer, verdicts=<factory>)[source]

Bases: object

A per-claim grounding of an answer against the substrate – the receipt behind ‘is this true?’.

Parameters:

  • answer (str)

  • verdicts (list[ClaimVerdict])

property grounded_fraction: float | None

Fraction of extracted claims supported by substrate evidence.

None when the answer had no extractable claims at all (empty, unparseable, or evasive) – there is nothing that was checked, so this is UNKNOWN rather than a vacuous 1.0.

unsupported()[source]

The claims the substrate could not corroborate – exactly what to flag or retract.

Return type:

list[ClaimVerdict]

contradicted()[source]

Claims where retrieved evidence actively disagreed – worse than merely uncited.

Return type:

list[ClaimVerdict]

is_grounded(threshold=1.0)[source]

True iff the grounded fraction meets threshold (default 1.0: every claim must be cited).

Fails closed when there is nothing to assess: an answer with no extractable claims is never reported as “grounded”, regardless of threshold – a full-grounding gate must not wave through silence just because a vacuous fraction happens to clear the bar.

Parameters:

threshold (float)

Return type:

bool

as_dict()[source]

Return a JSON-serializable factuality receipt.

Return type:

dict[str, Any]

check_factuality(substrate, answer, *, extract=None, corroborates=None, min_score=0.2, k=4, scope=None)[source]

Ground each claim of answer against substrate, returning a FactualityReceipt.

Parameters:

  • extract (Callable[[str], list[str]] | None) – answer -> [claim, ...] (default mixle.reason.llm.sentence_claims()).

  • corroborates (Callable[[str, str], Corroboration | bool] | None) – (evidence_text, claim) -> Corroboration classifying retrieved evidence against a claim as SUPPORTED / CONTRADICTED / UNVERIFIED (default: lexical candidacy gated by a negation/polarity check – see _default_corroborates(); pass a real NLI/entailment model for stronger grounding). A plain bool is still accepted for compatibility: True maps to SUPPORTED, False to UNVERIFIED.

  • min_score (float) – retrieval-score floor; evidence below it doesn’t count (guards low-signal embedder noise).

  • k (int) – evidence items retrieved per claim.

  • scope (str | None) – restrict retrieval to a team/access scope.

  • substrate (Substrate)

  • answer (str)

Return type:

FactualityReceipt