mixle.reason.graph_llm module

Knowledge-graph-producing LLM: UQ on the information, by marginalizing over graphs.

The likelihood an LLM emits is over strings; the thing we care about is the likelihood of the information. Bridging them needs P(meaning) = sum over strings with that meaning of P(string) – but “same meaning” over free text is fuzzy and its equivalence class is unbounded.

Have the model emit a knowledge graph (a set of triples) instead of prose and the difficulty dissolves: the information is the generated object, and “same meaning” becomes exact graph equality – a computable equivalence, no embeddings or entailment needed. Then the answer to any query is obtained by marginalizing over the graphs (subgraphs) that produce it:

P(outcome = c) = sum over graphs G with outcome(G) = c  of  P(G)

and the reliability of a single fact is its edge marginal P(triple in G) – exactly a knowledge-graph edge posterior (feed it to mixle.inference.ProbabilityCalibrator to calibrate against truth; the per-graph samples are an ensemble, so BALD-style epistemic splits apply too).

GraphLLM wraps any generate(prompt) -> str plus a parse(str) -> triples; it samples the model, canonicalizes each generation to a graph, and marginalizes – by Monte-Carlo counting, or by summing sequence likelihoods when log_probs are supplied (the correct, lower-variance estimator).

canonical_graph(triples)[source]

Canonical form of a graph: the frozenset of its triples (order-independent, dedup, hashable).

Parameters:

triples (Iterable[Any])

Return type:

frozenset

class GraphDistribution(graphs, probs)[source]

Bases: object

A distribution over knowledge graphs – the LLM’s belief about the information.

graphs are the distinct canonical graphs observed; probs[i] = P(graphs[i]) is the string distribution marginalized onto graphs (so it sums to 1 over distinct graphs). Every query is answered by marginalizing this distribution over the graphs that produce the queried outcome.

Parameters:
graphs: list[frozenset]
probs: ndarray
marginalize(outcome)[source]

P(outcome = c) = sum_{G : outcome(G) = c} P(G) – marginalize over subgraphs.

outcome maps a graph to a hashable value (a fact’s object, a boolean property, an aggregate). Returns [(value, probability), ...] sorted by descending probability.

Parameters:

outcome (Callable[[frozenset], Hashable])

Return type:

list[tuple[Any, float]]

entropy(outcome)[source]

Entropy (nats) of the marginal P(outcome = .) – the model’s uncertainty about that query.

Parameters:

outcome (Callable[[frozenset], Hashable])

Return type:

float

edge_marginals()[source]

P(triple in G) for every triple – the per-fact reliability (a KG edge posterior).

Return type:

dict[tuple, float]

fact_probability(triple)[source]

P(triple in G) for one fact (0 if never asserted).

Parameters:

triple (Any)

Return type:

float

calibrated_edge_marginals(calibrator)[source]

Edge marginals mapped through a fitted calibrator -> a calibrated P(fact is true).

A raw edge marginal is the model’s internal assertion rate for a fact, not a probability that the fact is true – a confidently-hallucinated fact has a high marginal yet is false. Fit the calibrator with fit_fact_calibrator() on labeled facts, then this reports, per fact, the empirical truth rate at that marginal. (Its residual limit – confident hallucinations that look exactly like known facts – is why an external check is needed; see the validation tests.)

Parameters:

calibrator (ProbabilityCalibrator)

Return type:

dict[tuple, float]

query(*prefix)[source]

Answer-completion posterior: P(object | prefix) over triples whose leading fields match.

query("eiffel", "city") marginalizes over graphs, collecting the objects of every triple starting ("eiffel", "city", ...) weighted by P(G), then renormalizes over the objects actually asserted. Returns [(object, probability), ...] best-first.

Parameters:

prefix (Any)

Return type:

list[tuple[Any, float]]

most_likely_graph()[source]

The single most probable graph and its probability.

Return type:

tuple[frozenset, float]

class GraphLLM(generate, parse, *, n=10)[source]

Bases: object

Turn a generate(prompt) -> str LLM into a distribution over knowledge graphs.

Parameters:

  • generate (Callable[[str], str]) – callable(prompt) -> str – one stochastic generation.

  • parse (Callable[[str], Iterable[Any]]) – callable(str) -> iterable[triple] – extract the asserted facts (semantic parse / structured-output decode). Generations that parse to the same triple-set are the same meaning – exact equality, no fuzzy matching.

  • n (int) – default number of samples per prompt.

sample_graphs(prompt, n=None)[source]

Sample n generations and parse each into a canonical graph.

Parameters:
Return type:

list[frozenset]

distribution(prompt, n=None, *, log_probs=None, graphs=None)[source]

Sample, parse, and marginalize strings onto graphs -> a GraphDistribution.

Marginalization uses Monte-Carlo counting by default (P(G) = fraction of samples that parse to G); pass log_probs (one log P(string) per sample) to instead sum the sequence likelihoods within each graph – the lower-variance, estimator-correct form that does not assume every string realizing a graph is equiprobable.

Parameters:
Return type:

GraphDistribution

fit_fact_calibrator(distributions, truth, *, method='isotonic')[source]

Fit edge marginal -> P(fact is true) over the facts asserted across many graph distributions.

Turn the model’s internal assertion rate (the edge marginal) into a calibrated probability of truth, learned against ground-truth labels. Collect every (triple, marginal) the model asserts, label it with truth(triple), and fit a ProbabilityCalibrator.

This does NOT rescue confident hallucination – a fact the model reliably confabulates has a high marginal indistinguishable from a genuinely-known one, so calibration lowers the overall fact-ECE but cannot pull those specific facts down. Separating them needs a signal external to the model (retrieval / a checker); the validation tests quantify both the gain and this residual.

Parameters:
Return type:

ProbabilityCalibrator