mixle.inference.model_comparison module¶
Model comparison: paired score differences and non-nested tests.
Is model A actually better than model B, or did it just win by chance on this sample? These tools answer that from paired, per-observation held-out scores or log-likelihoods – pairing removes the observation-to-observation variance that swamps a comparison of two separate score totals:
paired_score_difference()– the mean held-out score difference with a confidence interval and a paired test (works for any proper score frommixle.inference.scoring: CRPS, log score, …).
vuong_test()– the Vuong (1989) likelihood-ratio test for non-nested models, with an optional AIC/BIC complexity correction.
clarke_test()– Clarke’s distribution-free paired sign test, a robust alternative to Vuong when the log-likelihood-ratio distribution is non-normal.
compare_elpd()– the standard LOO/WAIC comparison: the expected-log-predictive-density difference with the standard error of the pointwise difference (pair these with thepointwisearrays frommixle.ppl.diagnostics.psis_loo()).
For scores lower is better; for log-likelihoods / elpd higher is better. Each result names the favored model.
- paired_score_difference(scores_a, scores_b, *, lower_is_better=True, ci_level=0.95)[source]
Mean paired held-out score difference with a CI and a paired t-test.
- Parameters:
scores_a (ndarray) –
(n,)per-observation held-out scores for the two models (same observations, same order).scores_b (ndarray) –
(n,)per-observation held-out scores for the two models (same observations, same order).lower_is_better (bool) – True for losses/scores (CRPS, log loss, pinball); False for higher-is-better metrics.
ci_level (float) – confidence level for the interval on the mean difference.
- Returns:
{'mean_diff', 'se', 'ci_low', 'ci_high', 't', 'p_value', 'favored'}wheremean_diffismean(a - b)andfavoredis'A'/'B'/'tie'at the given level.- Return type:
- vuong_test(loglik_a, loglik_b, *, k_a=0, k_b=0, correction='none')[source]
Vuong’s test for non-nested model selection.
Compares two models by their pointwise log-likelihoods. Under the null that both are equally close to the truth, the statistic
sqrt(n) * mean(m) / sd(m)(withm_i = ll_a_i - ll_b_i, minus an optional complexity correction) is asymptotically standard normal. A large positive value favors A.- Parameters:
loglik_a (ndarray) –
(n,)pointwise log-likelihoods of the two (non-nested) models.loglik_b (ndarray) –
(n,)pointwise log-likelihoods of the two (non-nested) models.k_a (int) – parameter counts, used only if
correctionis set.k_b (int) – parameter counts, used only if
correctionis set.correction (str) –
"none","aic"(subtractk_a - k_b), or"bic"(subtract(k_a - k_b) log n / 2) from the log-likelihood ratio.
- Returns:
{'statistic', 'p_value', 'favored'}.- Return type:
- clarke_test(loglik_a, loglik_b, *, k_a=0, k_b=0, correction='none')[source]
Clarke’s distribution-free paired sign test for non-nested models.
Counts how often model A’s pointwise log-likelihood beats B’s; under the null this count is
Binomial(n, 0.5). More robust thanvuong_test()when the per-observation log-ratio is heavy-tailed or skewed (where the normal approximation behind Vuong fails).
- compare_elpd(pointwise_a, pointwise_b)[source]
Compare two models’ expected log pointwise predictive density (LOO/WAIC).
Takes the per-observation
elpdcontributions (thepointwisearrays returned bymixle.ppl.diagnostics.psis_loo()/waic) and returns the elpd difference with the standard error of the pointwise difference – the honest SE for model comparison (a difference within ~2 SE of zero is not decisive).