stan-dev · florence-bockting · Sep 29, 2026
diff --git a/notes/loo-v3-cleanup-todos.md b/notes/loo-v3-cleanup-todos.md
@@ -0,0 +1,215 @@
+# loo v3.0.0 clean-up: open TODOs
+
+> Last update: 2026-09-29
+> 
+> Author: Florence Bockting
+
+The PRs and issues that relate to this work are in
+[loo-v3-related-prs-issues.md](loo-v3-related-prs-issues.md).
+The names of the documents are in
+[loo-v3-documentation.md](loo-v3-documentation.md). Use only those names.
+
+## Tutorials (new in loo v3)
+
+### `vignettes/articles-online-only/overview-measures.Rmd`
+
+*Model performance: Overview of predictive measures*. PR #363 "pred_measure".
+
+- [ ] Change the YAML `title:` and `\VignetteIndexEntry{}` to the name above.
+  Now: "Overview of predictive measures". Waits for: nothing.
+- [ ] Search `(TODO-Vehtari et al., 2026)` in the introduction. Cite
+  *Predictive measures: A formula reference*. Waits for: arXiv ID.
+- [ ] Search `our supplement` in the introduction. Cite
+  *Predictive measures: A formula reference*. Waits for: arXiv ID.
+- [ ] Search `[TODO-article-suppl]` in the "Aggregation" bullet below the table.
+  Link the formula reference. Waits for: arXiv ID.
+- [ ] Search `[TODO: arXiv ID]` in the reference list. Waits for: arXiv ID.
+- [ ] Search `[TODO-article]` in the callout "What this article does not cover".
+  Link *Model performance: Predictive schemes with `pred_measure()`*.
+  Waits for: nothing.
+- [ ] Search `[TODO: schemes and pred_measure() article]` in the reference list.
+  Link the same article. Waits for: nothing.
+- [ ] Search `[model-comparison TODO]` in the callout. Link
+  *Model comparison: Case Study with `model_compare()`*. Waits for: issue
+  #401.
+- [ ] Search `](model-comparison.html)`. Use the link text
+  *Model comparison: Explanation of `model_compare()`*. That article exists
+  only on `integrate-loo_compare`, so the link is broken on `pred_measure`.
+  Waits for: #380 merged.
+
+Use the `articles/articles-online-only/` URL form for all links.
+
+### `vignettes/articles-online-only/pred-measure-workflow.Rmd`
+
+*Model performance: Predictive schemes with `pred_measure()`*. PR #363.
+
+- [ ] Change the YAML `title:` and `\VignetteIndexEntry{}` to the name above.
+  Now: "Predictive schemes with pred_measure()". Waits for: nothing.
+- [ ] Search `[TODO: arXiv ID]` in the reference list. Waits for: arXiv ID.
+- [ ] Search `](model-comparison.html)`. Use the link text
+  *Model comparison: Explanation of `model_compare()`*. Waits for: #380
+  merged.
+- [ ] Search `LOAD-BRMS-GITHUB.txt`. Delete the `child=` chunk that loads brms
+  from GitHub. Waits for: brms on CRAN.
+
+### `vignettes/articles-online-only/model-comparison.Rmd`
+
+*Model comparison: Explanation of `model_compare()`*. PR #380.
+
+- [ ] Change the YAML `title:` and `\VignetteIndexEntry{}` to the name above.
+  Now: "Model comparison: Using the `model_compare()` function.".
+  Waits for: nothing.
+- [ ] Search `TODO-ARXIV` and `TODO-Vehtari`. Waits for: arXiv ID.
+- [ ] Search `the supplement`. Replace it with the name of the
+  formula reference. Waits for: arXiv ID.
+- [ ] Search `Computing predictive performance measures` in "See also". Use
+  *Model performance: Predictive schemes with `pred_measure()`*.
+  Waits for: nothing.
+- [ ] Search `TODO-case study` and `[TODO-LINK]`. Link
+  *Model comparison: Case Study with `model_compare()`*. Waits for: issue
+  #401.
+
+## Tutorials (existing)
+
+`loo_compare()` warns once per session (`.deprecate_once()`). A vignette build
+therefore shows the warning.
+
+### `vignettes/loo2-large-data.Rmd`
+
+- [ ] Search `loo_compare(loo_ss_1, loo_ss_2)`. `integrate-loo_compare`
+  replaced the other 3 calls. Replace it with `model_compare()`. Check the
+  output against #393 first. Waits for: #363 merged, subsampling decision.
+
+### `vignettes/loo2-elpd.Rmd`
+
+- [ ] Search `elpd(log_pd` (two calls) and "using `elpd()`" (two sentences).
+  Replace them with `measure_elpd()` or `test_pred_measure()`. Waits for: #363 merged.
+
+### All `vignettes/*.Rmd`
+
+- [ ] Grep for `crps(`, `scrps(`, `loo_crps(`, `loo_scrps(` and
+  `loo_predictive_metric(`. Do the two files above first. Waits for: #363
+  merged.
+
+## Other
+
+### `R/loo-glossary.R`
+
+*The `loo` glossary*. PR #380.
+
+- [ ] Add the general terms: measure, metric, score, utility, loss.
+  Waits for: nothing.
+
+## Project management
+
+### `vignettes/migration-guide.Rmd`
+
+*Migration guide*. PR #379, extended on #363, #380 and #378.
+
+- [ ] Merge the four versions. `loo-v3.0.0` has 182 lines, `pred_measure` 307,
+  `integrate-loo_compare` 337 and `parallelization` 216. #363 and #378 both
+  extend the file, so the second merge into `loo-v3.0.0` conflicts.
+  Waits for: #363 merged, #378 merged.
+- [ ] Decide if the sections "Maintainer checklist" and "`_pkgdown.yml`
+  reference (this branch)" belong in a user vignette. Waits for: nothing.
+
+### `vignettes/articles-online-only/roadmap.Rmd`
+
+*Development roadmap*. PR #396.
+
+- [ ] Delete `vignettes/articles-online-only/roadmap.html` from the PR. The
+  PR's own TODO asks for this. Waits for: nothing.
+
+## R code
+
+### `R/pred_measure.R`
+
+PR #363.
+
+- [ ] Search `overview of scores and metrics`. Use
+  *Model performance: Overview of predictive measures*. Waits for: nothing.
+- [ ] Search `pred-measure workflow article` (6 places). Use
+  *Model performance: Predictive schemes with `pred_measure()`*.
+  Waits for: nothing.
+
+### `R/pred_measure-compute.R`
+
+PR #363.
+
+- [ ] Search `See developer notes on computation`. Users cannot see `notes/`.
+  Cite the formula reference, or delete the `@note`. Waits for: nothing.
+- [ ] Search `not yet implemented`. Decide: keep `group_ids` as a reserved
+  argument, or remove it. If you remove it, also delete the test in
+  `tests/testthat/test_pred_measure.R` (search `group_ids errors`).
+  Waits for: nothing.
+- [ ] PR #380. Search `include this correction`. This is the warning for moment
+  matching and `reloo()`: only `elpd`, `mlpd` and `ic` are correct. Decide:
+  ship v3.0.0 with the warning, or wait for #399. Waits for: #399.
+
+### `R/crps.R`
+
+PR #363.
+
+- [ ] Resolve the conflict with #398. Both PRs edit this file. If #398 merges
+  first, merge `master` into `loo-v3.0.0`, then `loo-v3.0.0` into
+  `pred_measure`. Waits for: nothing.
+
+## Tests
+
+### `tests/testthat/data-for-tests/test_data_generation.R`
+
+PR #363, #380. In 2026-09 `data-for-tests/` is 5.9 MB. CRAN allows a 5 MB
+tarball.
+
+- [ ] Search `saveRDS(`. Add `compress = "xz"` to each call. Then re-save the
+  `.Rds` fixtures. The earlier re-save was never committed. It gave
+  5.85 → 4.13 MB. Waits for: #363 merged (C2).
+- [ ] Search `roaches_compare = 110` in `N_KEEP`. Change it to 53. Then
+  re-create `test_data_roaches_compare.Rds`. Waits for: #363 merged (C2).
+
+### `tests/testthat/test_compare.R`
+
+PR #380.
+
+- [ ] Search the assertion that expects `diag_diff` to be `""`. With 53
+  observations it becomes `"N < 100"`. Rewrite it. Do this with the `N_KEEP`
+  change above.
+
+## Package files
+
+### `NEWS.md`
+
+PR #363.
+
+- [ ] Search `[supported_measures_list()]`. Use backticks, as for every other
+  name. Waits for: nothing.
+- [ ] Search `vignette("migration-guide")`. Add `package = "loo"`, as in the
+  roxygen. Waits for: nothing.
+
+### `.Rbuildignore`
+
+- [ ] Add `^notes$`. `loo-v3.0.0` does not exclude `notes/` yet.
+  Waits for: nothing.
+
+### `.github/workflows/pkgdown.yaml`
+
+PR #363.
+
+- [ ] Search `paul-buerkner/brms`. Replace it with `any::brms`.
+  Waits for: brms on CRAN (C7).
+
+## Branches and PRs (no document)
+
+- [ ] Merge `pred_measure` into `integrate-loo_compare`. Waits for: nothing.
+- [ ] Finish the description of #380. The draft is outside the repo. Waits for: nothing.
+- [ ] Update the description of #363. "Known limitations" says
+  "`loo_compare` integration still outstanding". #380 does that. Mark #220 as
+  fixed, not partial. Waits for: #380 merged.
+- [ ] Add "Fixes #281", "Fixes #223", "Fixes #213", "Fixes #201",
+  "Fixes #135" and "Fixes #220" to the description of #379. GitHub does not
+  close them from #363, because #363 does not merge into `master`.
+  Waits for: #363 merged.
+- [ ] Measure the tarball of `master` with `R CMD build`. It gives the size
+  budget. Waits for: nothing.
+- [ ] Run `R CMD build` with the vignettes on this branch. Only this gives the
+  real size. Waits for: the two fixture items above.
diff --git a/notes/loo-v3-documentation.md b/notes/loo-v3-documentation.md
@@ -0,0 +1,47 @@
+# loo v3.0.0: documentation overview
+
+> Last update: 2026-09-29.
+>
+> Author: Florence Bockting
+
+Related notes: [loo-v3-cleanup-todos.md](loo-v3-cleanup-todos.md),
+[loo-v3-related-prs-issues.md](loo-v3-related-prs-issues.md).
+
+## 1. Tutorials (existing) — users
+
+| Document | Name | Status | Branch / PR |
+| :-- | :-- | :-- | :-- |
+| `vignettes/loo2-example.Rmd` | Using the loo package | changed: `model_compare()` | `integrate-loo_compare` (#380) |
+| `vignettes/loo2-with-rstan.Rmd` | Writing Stan programs for use with the loo package | changed: `model_compare()` | `integrate-loo_compare` (#380) |
+| `vignettes/loo2-elpd.Rmd` | Holdout validation and K-fold cross-validation | changed: prose; `elpd()` calls left | `integrate-loo_compare` (#380) |
+| `vignettes/loo2-large-data.Rmd` | Using leave-one-out cross-validation for large data | changed: 3 of 4 `loo_compare()` calls replaced | `integrate-loo_compare` (#380) |
+| `vignettes/loo2-weights.Rmd` | Bayesian stacking and pseudo-BMA weights | changed: parallelism | `parallelization` (#378) |
+| `vignettes/loo2-non-factorized.Rmd` | Leave-one-out cross-validation for non-factorized models | unchanged | `master` |
+| `vignettes/loo2-lfo.Rmd` | Approximate leave-future-out cross-validation | unchanged | `master` |
+| `vignettes/loo2-moment-matching.Rmd` | Avoiding model refits with moment matching | unchanged | `master` |
+| `vignettes/loo2-mixis.Rmd` | Mixture IS leave-one-out cross-validation | unchanged | `master` |
+| external: `https://users.aalto.fi/~ave/CV-FAQ.html` | Cross-validation FAQ | unchanged | `master` (`_pkgdown.yml`) |
+
+## 2. Tutorials (new in loo v3) — users
+
+| Document | Name | Status | Branch / PR |
+| :-- | :-- | :-- | :-- |
+| `vignettes/articles-online-only/overview-measures.Rmd` | Model performance: Overview of predictive measures | exists; placeholders left | `pred_measure` (#363) |
+| `vignettes/articles-online-only/pred-measure-workflow.Rmd` | Model performance: Predictive schemes with `pred_measure()` | exists; placeholder left | `pred_measure` (#363) |
+| `vignettes/articles-online-only/model-comparison.Rmd` | Model comparison: Explanation of `model_compare()` | exists; placeholders left | `integrate-loo_compare` (#380) |
+| not yet created | Model comparison: Case Study with `model_compare()` | planned | issue #401 |
+
+## 3. Other — users
+
+| Document | Name | Status | Branch / PR |
+| :-- | :-- | :-- | :-- |
+| `R/loo-glossary.R` | The `loo` glossary | extended; general terms planned | `integrate-loo_compare` (#380) |
+| ArXiV preprint | Predictive measures: A formula reference | in progress; no arXiv ID yet | in Overleaf |
+
+## 4. Project management — users and developers
+
+| Document | Name | Status | Branch / PR |
+| :-- | :-- | :-- | :-- |
+| `vignettes/migration-guide.Rmd` | Migration guide | exists; four versions on four branches | `loo-v3.0.0` (#379), extended on #363, #380, #378 |
+| `vignettes/articles-online-only/roadmap.Rmd` | Development roadmap | draft | `roadmap` (#396) |
+
diff --git a/notes/loo-v3-related-prs-issues.md b/notes/loo-v3-related-prs-issues.md
@@ -0,0 +1,84 @@
+# loo v3.0.0: related PRs and issues
+
+> Last update: 2026-09-29
+>
+> Author: Florence Bockting
+
+This note lists the PRs and issues that relate to the v3.0.0 refactoring.
+The open work is in [loo-v3-cleanup-todos.md](loo-v3-cleanup-todos.md).
+
+## Merge chain
+
+| PR | Head branch | Base branch | Brings |
+| :-- | :-- | :-- | :-- |
+| #380 | `integrate-loo_compare` | `pred_measure` | `model_compare()`, `model-comparison.Rmd`, `notes/design-discussions/`, compare fixture, deprecation of `loo_compare()` |
+| #363 (draft) | `pred_measure` | `loo-v3.0.0` | `*_pred_measure()`, `measure_*()`, `overview-measures.Rmd`, `pred-measure-workflow.Rmd`, deprecations of `elpd()`, `crps()` and others, test fixtures |
+| #379 (draft) | `loo-v3.0.0` | `master` | the release |
+| this | `loo-v3-cleanup` | `loo-v3.0.0` | the items below |
+
+**Targeted for v3.0.x:**
+
+| PR | Head branch | Base branch | Brings |
+| :-- | :-- | :-- | :-- |
+| #378 (draft) | `parallelization` | `loo-v3.0.0` | mirai/mori parallelism, fixes #308. Not part of v3.0.0, but it merges into the same base. |
+
+## Issues that the refactoring fixes
+
+| Issue | Title | Fixed by |
+| :-- | :-- | :-- |
+| #281 | New functions for better support for different scores and metrics | #363 |
+| #223 | (loo_)(s)crps could ask for only one argument with predictions | #363 |
+| #213 | loo_predictive_metric and loo_crps could accept psis objects | #363 |
+| #201 | Add R2 | #363 |
+| #135 | user defined loss/utility functions | #363 |
+| #220 | loo_compare for crps and loo_crps | #363 (part), #380 (rest) |
+
+All six are open. GitHub closes an issue from a "Fixes #n" line only when the
+PR merges into `master`. #363 merges into `loo-v3.0.0`.
+
+## Open PRs into `master` that touch the same code
+
+| PR | Topic | Relation to this refactoring |
+| :-- | :-- | :-- |
+| #398 | `crps.numeric()` passes `permutations` wrongly (fixes #397) | Edits `R/crps.R`. #363 deprecates `crps()` in the same file. Merge conflict. |
+| #393 (draft) | `loo_compare()` returns a data.frame for subsampling (fixes #392) | #380 deprecates `loo_compare()`. |
+| #399 (draft) | Correct `pred_measure` values after post-hoc methods | |
+| #340 | Export `srs_diff_est()` (fixes #333) | Subsampling comparison. `model_compare()` has a subsampling method. |
+| #178 (draft) | LOO difference plot | #394 (milestone v3.0.0) extends it to the new measures. |
+| #291 (draft) | Format the project with Air | Rewrites every file. Merge it before #363 or after #379, never between them. |
+| #396 | Roadmap on the website | Describes the v3.0.0 plan. Update it when this list changes. |
+
+## Other issues, by target release
+
+All of these issues have the GitHub milestone v3.0.0. This list gives the
+release that each one targets.
+
+**v3.0.0**
+
++ #401 (case study for model comparison)
++ #388 (clean up issues): general clean-up and release preparation
++ #353 (warnings and messages): standardize and improve warning/messaging behavior implemented in loo
+
+**v3.0.0, developer discussion**
+
++ #384 (LLM code optimizations): How do we want to integrate this in the development process
+
+**v3.0.0 and later**
+
++ #343 (subset psis objects)
++ #249 (more `posterior` functions): avoid duplicated code and decide which functions should go into posterior and which should stay in loo
++ #394 (extend the difference plot)
+
+**v3.0.x**
+
++ #308 / #378 (parallelism): improve parallelization in loo
+
+**v3.x.x**
+
++ (loo diagnostics): improved posthoc diagnostics using loo
++ #192 (n_eff to ESS): improve diagnostics
+
+**Target not decided**
+
++ #385 (mutation testing)
+