mixle.substrate.accum module

Measure whether accumulated calibrated knowledge improves held-out answers.

The knowledge-accumulation flywheel measures whether adding calibrated knowledge to the belief store (mixle.substrate.belief) improves answers on a held-out question set, with no model retraining. Distillation improves the student model; accumulation improves what the student or teacher can retrieve.

Three guards keep the measurement grounded:

  • attribution – the improvement must disappear when the newly-assimilated items are withheld from retrieval, proving the gain came from the store growing rather than anything else (timing, caching, a lucky answer_fn). Withholding is delta-level, not belief-level (MXR-080-0246): a belief assimilate_batch UPDATED (rather than created from nothing) already existed for the before pass, so the withheld measurement rolls it back to exactly its pre-batch snapshot for that one measurement rather than hiding it outright – otherwise pre-existing evidence that had nothing to do with the batch would vanish from the counterfactual too, driving withheld quality below the true baseline and fabricating or inflating the measured attribution.

  • evaluator control (MXR-080-0247) – answer_fn is reset (via reset_answer_fn) to an identical starting state before every one of the before/after/withheld passes, so a stateful evaluator – a cache, a call counter, a seeded sampler – cannot drift across passes and masquerade as a store-content effect. trials > 1 repeats each pass that many times (reset before every replicate) and keeps every individual rate, not just the mean, so attribution becomes a paired comparison across matched replicates with visible spread instead of one unrepeated scalar.

  • credence weighting – retrieval goes through mixle.substrate.belief.retrieve_beliefs(), which ranks by relevance * credence and can be hard-thresholded with min_credence; a batch of low-credence (e.g. pure MODEL_ASSERTION) knowledge must not inflate the measured improvement – it is down-weighted, never treated as ground truth.

class QAItem(question, answer)[source]

Bases: object

One held-out question: answer_fn is judged correct on it if it produces answer from the retrieved context alone.

Parameters:
class FlywheelMeasurement(solve_rate, grounded_fraction, trial_solve_rates=<factory>, trial_grounded_fractions=<factory>)[source]

Bases: object

Held-out answer quality and grounding rate for one flywheel measurement.

trial_solve_rates / trial_grounded_fractions carry one entry per replicate trial (MXR-080-0247): a 1-tuple equal to solve_rate / grounded_fraction for the default trials=1, longer when measure_flywheel() is asked to repeat each pass. Exposed raw – rather than collapsed into one particular spread statistic – so a caller can compute whatever uncertainty measure (stderr, a confidence interval, a paired significance test) actually fits its own answer_fn, instead of this module asserting one test is correct for an arbitrary evaluator.

Parameters:
class FlywheelReport(before, after, withheld, attribution_confirmed)[source]

Bases: object

Before/after/withheld flywheel measurements with an attribution check.

Parameters:

  • before (FlywheelMeasurement)

  • after (FlywheelMeasurement)

  • withheld (FlywheelMeasurement)

  • attribution_confirmed (bool)

measure_flywheel(sub, questions, answer_fn, assimilate_batch, *, k=5, min_credence=None, reset_answer_fn=None, trials=1)[source]

Measure answer_fn against questions before and after assimilate_batch(sub) adds a batch of calibrated beliefs (returning the ids it touched), with a THIRD measurement that isolates exactly the batch’s own contribution – the attribution control. answer_fn and every other piece of the system stay fixed throughout: only the store’s content (and what it makes retrievable) changes.

Withheld isolation is delta-level, not belief-level (MXR-080-0246): assimilate_batch may UPDATE a belief that already existed – and was already visible to the before pass – rather than create one from nothing. An id it returns that was already present pre-batch is rolled back, for the duration of the withheld measurement only, to exactly its pre-batch snapshot (its pre-existing evidence, unmodified); an id that did not exist pre-batch is excluded from retrieval outright, same as before, since there is no pre-batch state for it to roll back to. Either way sub is restored to its true post-assimilation content before this function returns – the rollback is an internal, temporary device for taking one measurement, never an observable side effect of calling this function.

answer_fn must be a pure function of (question, context), OR reset_answer_fn must rewind whatever internal state it has to an identical starting point (MXR-080-0247): when given, it is called immediately before every individual measurement – every trial of every one of the three passes – so a stateful evaluator (a cache, a call counter, a seeded sampler) cannot drift across before/after/withheld and masquerade as a store-content effect. trials (requires reset_answer_fn when greater than 1 – without a reset, repeated calls just accumulate whatever state answer_fn carries between them rather than sampling independent replicates) repeats each pass that many times and keeps every individual rate alongside the mean, turning the attribution check into a paired comparison across matched replicates instead of one unrepeated scalar.

Parameters:
Return type:

FlywheelReport