Introduce `model_compare()` for model comparison with support of new predictive measures by florence-bockting · Pull Request #380 · stan-dev/loo

florence-bockting · 2026-07-07T06:31:54Z

Fixes #220

Summary

This PR adds model_compare(). The function compares models on all predictive measures of the *_pred_measure() API (#363).

model_compare() replaces loo_compare().
loo_compare() is deprecated. It warns once per session. It still compares "loo", "waic", and "kfold" objects on ELPD.
loo_compare methods in other packages (e.g. loo_compare.brmsfit) still dispatch.
model_compare() accepts results from loo_pred_measure(), kfold_pred_measure(), test_pred_measure(), and
insample_pred_measure().
For each measure that all models share, the function computes the paired difference and its standard error.
Each measure uses its own best model as the reference.
The function flips loss measures (e.g. mse) to the utility scale. A higher difference is then always better. print() marks each flipped measure.
The new custom_measure() defines a custom measure. It sets the name, the orientation, and the standard error of the difference.

The output for "loo", "waic", and "kfold" objects does not change.

The comment below lists all new features and functions.

The review decisions are in notes/design-discussions/model_compare.md.

Examples

Print snapshots: tests/testthat/_snaps/compare.md
New article: vignettes/articles-online-only/model-comparison.Rmd

codecov-commenter · 2026-07-07T06:35:07Z

Codecov Report

❌ Patch coverage is 91.28978% with 104 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.65%. Comparing base (2a77502) to head (d03261f).
⚠️ Report is 1 commits behind head on pred_measure.

Files with missing lines	Patch %	Lines
R/model_compare-print.R	86.73%	37 Missing ⚠️
R/model_compare-pred_measure.R	92.70%	28 Missing ⚠️
R/pred_measure-builtin.R	67.85%	27 Missing ⚠️
R/pred_measure-helpers.R	92.30%	5 Missing ⚠️
R/pred_measure-compute.R	95.89%	3 Missing ⚠️
R/loo_compare.R	75.00%	2 Missing ⚠️
R/model_compare.R	99.14%	2 Missing ⚠️

Additional details and impacted files

@@               Coverage Diff                @@
##           pred_measure     #380      +/-   ##
================================================
+ Coverage         91.29%   91.65%   +0.36%     
================================================
  Files                35       38       +3     
  Lines              4168     5044     +876     
================================================
+ Hits               3805     4623     +818     
- Misses              363      421      +58

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

github-actions · 2026-07-07T06:47:28Z

This is how benchmark results would change (along with a 95% confidence interval in relative change) if d03261f is merged into pred_measure:

✔️loo_function: 1.09s -> 1.09s [-1.44%, +1.47%]
✔️loo_matrix: 1.18s -> 1.17s [-2.51%, +0.62%]
Further explanation regarding interpretation and methodology can be found in the documentation.

…nals Re-derives the file split on top of the current branch rather than merging the earlier WIP, which git resolved into eight duplicate definitions. model_compare.R (2082 lines) is split by concern into model_compare.R (the generic, default method, ordering and diagnostics), model_compare-pred_measure.R (the multi-measure path, rank resolution, standard errors) and model_compare-print.R (print.compare.loo and its table helpers). Every moved body is byte-identical to its previous version. loo_compare.R keeps only the forwarding generic and its methods. The copies of elpd_diffs, se_elpd_diff, find_model_names, middle_idx, order_stat_heuristic, diag_elpd, diag_diff and print.compare.loo it also held were dead: alphabetical collation meant model_compare.R won. diag_diff and print.compare.loo had diverged, so the dead copies were also wrong. loo_compare_checks, loo_compare_matrix, loo_compare_order and loo_order_stat_check are removed for the same reason. Warning helpers renamed to the package convention: .warn_insample_compare, .warn_kfold_K_mismatch and .warn_omitted_compare_measures become throw_*_warning(); .inform_compare_sign_conversion loses its dot prefix. Adds loo_compare.psis_loo_ss_list so subsampling objects keep dispatching under the old name, and fixes the undefined `loo3` in the loo_compare examples. NAMESPACE and man/ still need regenerating with roxygen2 8.0.0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The refactor copied an older print.compare.loo into model_compare.R, which alphabetical collation made the live one, so the simplify argument added later at a user's request stopped existing. Because print.compare.loo takes `...`, `print(comp, simplify = FALSE)` was silently swallowed rather than erroring, and the snapshot tests that would have caught it skip unless NOT_CRAN is set. Merges the two: keeps the pred_measure dispatch, the compare_ref_model message and .print_compare_diag_message() from the model_compare.R version, and restores the flexible column selection and simplify argument from the loo_compare.R one. test_compare.R with NOT_CRAN=true: 355 pass, 0 fail (was 352 pass, 3 fail on 8897e1f and on the split commit). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…_compare # Conflicts: # NEWS.md # R/loo.R # R/loo_compare.psis_loo_ss_list.R # R/pred_measure-builtin.R # R/pred_measure-compute.R # R/pred_measure.R # man/pred_measure.Rd # tests/testthat/_snaps/loo_subsampling_cases.md # tests/testthat/test_pred_measure.R # vignettes/articles-online-only/overview-measures.Rmd # vignettes/articles-online-only/pred-measure-workflow.Rmd # vignettes/loo2-large-data.Rmd

…nt folds; update vignette example

…es other than elpd, mlpd, and ic

florence-bockting · 2026-09-28T06:35:08Z

Overview of changes

Features

Comparison

Compares all pred_measure types. All models in one call must use the same evaluation source. Mixed sources cause an error.
Matches measures on bare names. rmse_loo and rmse_kfold both become rmse.
Uses the best model of each measure as the reference for that measure. The attribute compare_reference records each reference.
Orders rows by the first shared measure (elpd if it is present).
Accepts named models: model_compare(A = m1, B = m2).
Still supports subsampled LOO (psis_loo_ss).

Standard error of differences

"sum" and "mean": from paired pointwise differences.
"measure_specific": special formulas for rmse, r2, and bacc.
"custom": set with attr(fun, "measure_se_diff") or custom_measure(). If a custom measure sets nothing, the SE is NA. A message then tells the user.

Checks

Condition	Output
Models use different measure sets	warning (lists omitted measures)
Models disagree on `loss` or `se_diff_fun`	error
Models use different `y`	warning
k-fold models use different `K` or folds	warning
In-sample comparison	warning (optimistic bias)
More than 11 models	warning (order-statistic check)
`loo_moment_match()` or `reloo()` with a measure other than `elpd`, `mlpd`, `ic`	warning

Print method (print.compare.loo())

measures: NULL (first measure), "all", or a character vector.
simplify = FALSE: also shows the per-model estimate and SE.
digits = NULL: sets the digits for each measure.
Prints the reference model for each measure table.
Prints the PSIS Pareto k diagnostics once per model, above the tables.
Marks each loss measure with sign flipped in the table header. A note below the tables names these measures.
Shows the diagnostic glossary only when the output includes an ELPD measure.
Keeps each output line at 80 characters or fewer.

User-facing functions

Function	Status
`model_compare()` (methods `default`, `psis_loo_ss_list`)	new
`custom_measure()`	new
`print.compare.loo()`	new arguments `measures`, `simplify`
`loo_compare()`	deprecated
`print.compare.loo_ss()`	removed

Internal functions

R/model_compare.R

New: .model_compare_inputs(), .measure_ref_model(), .model_compare_estimates_table(), throw_kfold_K_mismatch_warning(), throw_kfold_folds_mismatch_warning()
Renamed from loo_compare_*: model_compare_checks(), model_compare_matrix(), model_compare_order(), model_order_stat_check()
Moved from R/loo_compare.R: elpd_diffs(), se_elpd_diff(), find_model_names(), middle_idx(), order_stat_heuristic(), diag_elpd(), diag_diff()

R/model_compare-pred_measure.R (new file)

Dispatch and checks: is.pred_measure(), is.loo_pred_measure(), compare_pred_measure(), .compare_source(), .compare_metadata_check()
Measure sets: .compare_measures(), .compare_pointwise_cols(), .resolve_rank_measure(), .is_elpd_measure(), .get_measure_info()
Orientation: .builtin_loss_measures(), .measure_is_loss(), .compare_sign_converted_measures()
SE of differences: .measure_pointwise_diff_method(), .check_declared_aggregation(), .measure_se_diff_fun(), .check_se_diff_value(), .resolve_custom_se_diffs(), .se_diff_input(), .validate_se_diff(), .pair_measure_stats()
Conditions: throw_insample_compare_warning(), throw_omitted_compare_measures_warning(), inform_missing_custom_se_diff()

R/model_compare-print.R (new file)

.print_compare_pred_measure(), .print_compare_measure_table(), .print_psis_diag_block(), .print_compare_diag_message(), .compare_reference_line(), .compare_source_phrase(), .parse_diag_psis(),
.compare_suffix(), .display_name(), .n_of_models(), .cat_wrapped(), .print_sign_flip_note()

R/model_compare.psis_loo_ss_list.R (renamed from loo_compare.psis_loo_ss_list.R)

model_compare_ss(), model_compare_ss_naive(), model_compare_ss_diff()

R/pred_measure-builtin.R

.se_diff_rmse(), .se_diff_r2(), .se_diff_bacc(), .se_r2_delta(), .measure_info()

R/pred_measure-compute.R

.detect_posthoc(), .warn_posthoc()

R/print.R

.measure_digits(), .se_digits(), .resolve_digits(), .format_estimates()

…_compare # Conflicts: # R/pred_measure-compute.R # R/pred_measure-helpers.R # R/pred_measure.R # man/insample_pred_measure.Rd # man/pred_measure_params.Rd

jgabry

Thanks @florence-bockting, this is great! Here's a first round of review comments, but I probably missed some things. I should probably do another round, but we can start with these first and then I'll review again

jgabry · 2026-09-28T17:35:40Z

With the addition of a file this big the package tarball goes from about 4.1 MB to 7.3 MB, which is over CRAN's 5 MB limit. We might be able to get an exception, but I'm not sure. I guess the issue is that it stores full ypred and mupred draws for three models. Could we use fewer draws, e.g. 100 maybe? Or could we use a script to generate this instead of saving it? Would that be too slow?

I reduced the size in two ways:

test_data_roaches_compare.Rds now comes from a brms fit with thin = 4, so it holds 100 draws instead of 400.

All test_data_*.Rds files are now saved with compress = "xz".

The tarball is now about 4.4 MB, built without the vignettes.

Commit: ee7b

jgabry · 2026-09-28T20:00:19Z

+  if (!length(cols)) {
+    stop("No measure is shared by all models.", call. = FALSE)
+  }
+  internal <- cols[1L]


The doc says if elpd is present then rows are ordered by elpd, but the code doesn't prefer elpd when present. I think it just takes the first shared measure of whichever model you pass first, regardless of whether elpd is present, right?

Also, if the models list measures in different orders, then the ranking measure depends on the order you pass the models in, so e.g. if you pass in the models as list(m1, m2) that can lead to a different measure used as the ranking measure compared to list(m2, m1). Is that intentional?

Should we prefer elpd explicitly and if there's no elpd then use an order that doesn't depend on which model comes first?

Thank you.

The ranking measure is now elpd if all models share it. Otherwise it is the first shared measure in alphabetical order. The other measures follow in alphabetical order.

The order no longer depends on the order of the models.

The print method and the compare_measures attribute use the same order.

Tests:

updated existing tests: without elpd now expects alphabetical order of measures; with elpd expects elpd to be first.

added new test that checks list(m1, m2) and list(m2, m1) give the same measure order.

I updated the docs for the new behavior: NEWS.md, model_compare documentation, the glossary, and the model-comparison vignette.

Commit: fd42

jgabry · 2026-09-28T20:01:57Z

+    intersect,
+    lapply(loos, function(x) colnames(x$pointwise))
+  )
+  cols[!grepl("^p_", cols)]


I think this means you can't have a custom measure with a name that starts with p_ because this will drop it. I think just dropping columns by exact name would fix it.

Good point.
I changed it to: cols <- cols[!cols %in% c("p_loo", "p_waic", "p_kfold")] and added a test that ensures a custom measure starting with p_ is not dropped.

Commit: f7a7

jgabry · 2026-09-28T20:03:11Z

+#'   print(comp, measures = "all")    # all measure diff tables
+#'
+#'   # the same works for k-fold CV
+#'   kf1 <- brms::kfold(fit1, folds = folds, save_fits = TRUE)


This errors because folds isn't defined

indeed. I added a definition of folds before calling brms::kfold

Commit: a509

jgabry · 2026-09-28T20:04:35Z

+#'     measures = c("rmse", "r2")
+#'   )
+#'   comp <- model_compare(pm1, pm2)
+#'   print(comp)                      # ranked by elpd (default)


the comment says ranked by elpd (default) but elpd isn't one of the measures, there's only rmse and r2 here

thanks, I removed the comment. (The ordering information is now already at several places in the documentation so no need for a code comment)

Commit: 102a

jgabry · 2026-09-28T20:34:15Z

+    toc: true
+    toc_depth: 3
+params:
+  EVAL: TRUE #!r identical(Sys.getenv("NOT_CRAN"), "true")


Do we definitely want this as online only? If so, we can remove the commented out part here. Or should we consider it to be included in the CRAN package too? I don't have a very strong opinion, just curious what your thoughts were about this.

jgabry · 2026-09-28T20:35:40Z

+print(model_compare(loos), simplify = FALSE)
+```
+
+## Preview: Glimpse into the model comparison results


I think we should maybe just title this section "Basic usage", even though it's not a super exciting title

jgabry · 2026-09-28T20:44:53Z

+  attr(fun, "measure_name") <- name
+  attr(fun, "measure_loss") <- loss
+  attr(fun, "measure_se_diff") <- se_diff_fun
+  .measure_entry_custom(fun)


This calls .measure_entry_custom() only for the validation checks and throws away the result, but that means lines 80-84 of pred_measure-helpers.R can produce a confusing error message for bad name arguments. custom_measure(f, name = "") or name = NULL, or name = c("a", "b"), etc., get this error message:

Error: A custom function passed to 'measure' must have attribute 'measure_name', e.g. attr(my_fun, "measure_name") <- "my_metric".

But the user never passed anything to measure and also the error message suggests they should assign the attribute themselves, but custom_message exists to avoid them having to do that.

Just noticed that the other errors thrown by .measure_entry_custom() (e.g. for loss and se_diff) also suggest assigning the attribute directly. I guess that's still supported behavior, but not what we prefer if custom_measure exists.

I think there are a couple ways to potentially handle all of these:

Use custom_measure in the wording of the messages, e.g "Custom measure 'huber' must declare loss as TRUE or FALSE; see custom_measure()"

or

Let the calling function pick the wording. There's already an origin argument for .check_se_diff_value() for this, so .measure_entry_custom could do something similar and custom_measure could pass it the argument names.

I would prefer option 1 probably, and updating the examples to use custom_measure instead of assigning attributes even if technically that's still allowed.

jgabry · 2026-09-28T20:48:58Z

 #'
+#' Custom measures are assumed to be on a utility scale (higher is better) in
+#' [model_compare()]. Declare a custom loss with
+#' `attr(my_fun, "measure_loss") <- TRUE` so that [model_compare()] converts and


Should we avoid suggesting to assign attributes and just encourage using the custom_measure function? Same for se diff below

jgabry · 2026-09-28T20:50:10Z

 #' attr(my_abs_err, "measure_name") <- "my_abs_err"
+#' # the estimate is the mean of the pointwise values, so declare "mean"
+#' attr(my_abs_err, "measure_se_diff") <- "mean"


same point as above about whether we should be showing custom_measure instead?

…abetical order

…measures

…remove unnecessary line break

Florence Bockting added 6 commits July 7, 2026 09:28

refactor: update loo_compare to work with loo_pred_measure

89d5e9a

refactor: update pred_measure API for integration with loo_compare()

7ce2d50

tests: new tests for loo_compare with loo_pred_measure

7394ba3

docs: update documentation and glossary

494d24f

docs: update documentation

9611656

chore: update developer-notes

a3019e3

florence-bockting and others added 22 commits July 8, 2026 10:49

Merge branch 'pred_measure' into integrate-loo_compare

3e1bf93

refactor: clean-up refactoring and adjust for renamed arg

06d6281

fix: update missing argument and corresponding docs

b46ff58

chore: update NEWS.md

f1fcaf0

Merge branch 'pred_measure' into integrate-loo_compare

e17e46d

refactor: update computation of SE for loo_compare with pred_measure_loo

abc74b5

refactor: create model_compare and workout design

faa27c5

update model_compare() functionality

3e078aa

Merge branch 'pred_measure' into integrate-loo_compare

8897e1f

review: update the model_compare function

c0c1fc0

update: deprecate loo_compare but maintain backwards comptability

c594818

review: code refactoring review

aa30e54

docs: update function documentation to roxygen2 8.0.0

b1f0b32

docs: update glossary to match loo_compare and model_compare

63c16a8

fix: throw deprecation warning for loo_compare only once per session.

d874044

fix: remove old 'estimates_only' label

9e930e3

refactor: avoid duplicate computation

4fe4b0d

docs: add loo_compare and model_compare to seealso statement

88890b6

refactor: add warning for disagreeing measure name in attribute and list

51b3c9c

review: minor style corrections and doc adjustment

81ae8eb

VisruthSK added this to the v3.0.0 milestone Sep 17, 2026

florence-bockting changed the title ~~Update loo_compare() to support new pred_measure API~~ Introduce model_compare() for model comparison with support of new predictive measures Sep 21, 2026

Florence Bockting added 8 commits September 21, 2026 15:55

docs: update developer-notes wrt custom_measure

6971e13

fix: model_compare throws warning when kfold_pred_measure use differe…

8815998

…nt folds; update vignette example

fix: pass moment_match loss into pred_measure

eec2140

fix: use separate digits for measures in print method

dcf5fde

fix: update snapshots

ba29be1

refactor: remove the 'rank_by' argument for model_compare

c5af96b

fix: check whether pothoc measures have been used and warn for measur…

9ffff9c

…es other than elpd, mlpd, and ic

fix: remove diagnostic flags for measures other than elpd

dd2ac81

florence-bockting mentioned this pull request Sep 28, 2026

Add case-study vignette for new model_compare function #401

Open

Florence Bockting added 4 commits September 28, 2026 09:12

docs: rebuild loo_pred_measure.Rd for the post-hoc correction note

e372858

feat: mark and explain flipped measure signs in print

36795c2

docs: move model_compare design notes to design-discussions

999cfa7

docs: shorten NEWS entries for model_compare

de54993

Florence Bockting added 2 commits September 28, 2026 11:24

docs: update model_compare vignette

cd1085d

Merge remote-tracking branch 'origin/pred_measure' into integrate-loo…

954bdb1

…_compare # Conflicts: # R/pred_measure-compute.R # R/pred_measure-helpers.R # R/pred_measure.R # man/insample_pred_measure.Rd # man/pred_measure_params.Rd

florence-bockting requested a review from jgabry September 28, 2026 09:00

florence-bockting marked this pull request as ready for review September 28, 2026 09:00

jgabry requested changes Sep 28, 2026

View reviewed changes

florence-bockting and others added 8 commits September 29, 2026 09:33

Merge branch 'pred_measure' into integrate-loo_compare

390eeb1

Merge branch 'pred_measure' into integrate-loo_compare

fbe6e88

tests: make the test fixtures smaller for CRAN

ee7b99b

model_compare: rank by elpd, else by the first shared measure in alph…

fd427d9

…abetical order

model_compare: drop complexity columns by exact name, keep custom p_ …

f7a7b4d

…measures

fix: define missing fold in example of model_compare

a509447

docs: remove measure-ordering code comment from example

102a76b

chore: update deprecation warning to not print internal function and …

d03261f

…remove unnecessary line break

Uh oh!

Conversation

florence-bockting commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Examples

Uh oh!

codecov-commenter commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions Bot commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

florence-bockting commented Sep 28, 2026

Overview of changes

Features

User-facing functions

Internal functions

Uh oh!

jgabry left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

florence-bockting Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

florence-bockting Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

florence-bockting Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

florence-bockting Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

florence-bockting Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

florence-bockting commented Jul 7, 2026 •

edited

Loading

codecov-commenter commented Jul 7, 2026 •

edited

Loading

github-actions Bot commented Jul 7, 2026 •

edited

Loading

florence-bockting Sep 29, 2026 •

edited

Loading

florence-bockting Sep 29, 2026 •

edited

Loading

florence-bockting Sep 29, 2026 •

edited

Loading

florence-bockting Sep 29, 2026 •

edited

Loading

florence-bockting Sep 29, 2026 •

edited

Loading