mixle.reason.graph_llm module

Knowledge-graph-producing LLM uncertainty by marginalizing over graphs.

An LLM’s raw likelihood is over strings, but many applications care about the information asserted by those strings. This module has the model emit a knowledge graph, represented as a set of triples, so equivalent information can be canonicalized by exact graph equality. Answers are then obtained by marginalizing over the graphs that produce them:

P(outcome = c) = sum over graphs G with outcome(G) = c  of  P(G)

The reliability of a single fact is its edge marginal P(triple in G). Feed those marginals to mixle.inference.ProbabilityCalibrator to calibrate against labeled truth when such labels are available.

GraphLLM wraps any generate(prompt) -> str callable plus a parse(str) -> triples callable. It samples the model, canonicalizes each generation to a graph, and marginalizes by Monte Carlo counting or by summing sequence likelihoods when log_probs are supplied.

canonical_graph(triples)[source]

Return an order-independent, deduplicated graph representation.

Parameters:

triples (Iterable[Any])

Return type:

frozenset

class GraphDistribution(graphs, probs)[source]

Bases: object

A distribution over knowledge graphs.

graphs are the distinct canonical graphs observed; probs[i] = P(graphs[i]) is the string distribution marginalized onto graphs (so it sums to 1 over distinct graphs). Every query is answered by marginalizing this distribution over the graphs that produce the queried outcome.

Parameters:
marginalize(outcome)[source]

Return P(outcome = c) = sum_{G : outcome(G) = c} P(G).

outcome maps a graph to a hashable value (a fact’s object, a boolean property, an aggregate). Returns [(value, probability), ...] sorted by descending probability.

Parameters:

outcome (Callable[[frozenset], Hashable])

Return type:

list[tuple[Any, float]]

entropy(outcome)[source]

Return entropy in nats of the marginal outcome distribution.

Parameters:

outcome (Callable[[frozenset], Hashable])

Return type:

float

edge_marginals()[source]

Return P(triple in G) for every asserted triple.

Return type:

dict[tuple, float]

fact_probability(triple)[source]

P(triple in G) for one fact (0 if never asserted).

Parameters:

triple (Any)

Return type:

float

calibrated_edge_marginals(calibrator)[source]

Map edge marginals through a fitted calibrator.

A raw edge marginal is the model’s internal assertion rate for a fact, not a probability that the fact is true – a confidently-hallucinated fact has a high marginal yet is false. Fit the calibrator with fit_fact_calibrator() on labeled facts, then this reports, per fact, the empirical truth rate at that marginal. Confident hallucinations that look exactly like known facts still require an external check.

Parameters:

calibrator (ProbabilityCalibrator)

Return type:

dict[tuple, float]

query(*prefix)[source]

Return per-object fact marginals for triples matching prefix.

A graph may assert several matching objects; each is then an event with its own marginal and may have probability one. The result is therefore not renormalized into a categorical outcome. Use functional_query() when the relation is declared single-valued and a categorical outcome, including no assertion, is required.

Parameters:

prefix (Any)

Return type:

list[tuple[Any, float]]

functional_query(*prefix, no_assertion=None)[source]

Return a categorical outcome for an explicitly functional relation.

Every graph must assert at most one matching suffix. Graphs with no match contribute to no_assertion rather than disappearing.

Parameters:
Return type:

list[tuple[Any, float]]

most_likely_graph()[source]

The single most probable graph and its probability.

Return type:

tuple[frozenset, float]

class GraphLLM(generate, parse, *, n=10)[source]

Bases: object

Turn a generate(prompt) -> str LLM into a distribution over knowledge graphs.

Parameters:

  • generate (Callable[[str], str]) – callable(prompt) -> str for one stochastic generation.

  • parse (Callable[[str], Iterable[Any]]) – callable(str) -> iterable[triple] to extract asserted facts. Generations that parse to the same triple set are treated as the same canonical graph.

  • n (int) – default number of samples per prompt.

sample_graphs(prompt, n=None)[source]

Sample n generations and parse each into a canonical graph.

Parameters:
Return type:

list[frozenset]

distribution(prompt, n=None, *, log_probs=None, graphs=None, strings=None)[source]

Sample, parse, and marginalize strings onto graphs.

Marginalization uses Monte-Carlo counting by default (P(G) = fraction of samples that parse to G); pass log_probs (one log P(string) per sample) to instead sum the sequence likelihoods within each graph. This lower-variance estimator does not assume every string realizing a graph is equiprobable.

Parameters:
Return type:

GraphDistribution

fit_fact_calibrator(distributions, truth, *, method='isotonic')[source]

Fit edge marginal -> P(fact is true) over the facts asserted across many graph distributions.

Turn the model’s internal assertion rate (the edge marginal) into a calibrated probability of truth, learned against ground-truth labels. Collect every (triple, marginal) the model asserts, label it with truth(triple), and fit a ProbabilityCalibrator.

This does not by itself identify confident hallucinations: a false fact the model reliably emits can have a high marginal. Calibration can improve the aggregate reliability curve, but separating those cases requires an external signal such as retrieval or a checker.

Parameters:
Return type:

ProbabilityCalibrator