From b15a95ffce062b688684c19fee9978aad9d735d5 Mon Sep 17 00:00:00 2001 From: Florence Bockting Date: Tue, 29 Sep 2026 10:50:36 +0300 Subject: [PATCH] docs: include developer notes for organizing release --- notes/loo-v3-cleanup-todos.md | 215 +++++++++++++++++++++++++++++ notes/loo-v3-documentation.md | 47 +++++++ notes/loo-v3-related-prs-issues.md | 84 +++++++++++ 3 files changed, 346 insertions(+) create mode 100644 notes/loo-v3-cleanup-todos.md create mode 100644 notes/loo-v3-documentation.md create mode 100644 notes/loo-v3-related-prs-issues.md diff --git a/notes/loo-v3-cleanup-todos.md b/notes/loo-v3-cleanup-todos.md new file mode 100644 index 00000000..3fc252b4 --- /dev/null +++ b/notes/loo-v3-cleanup-todos.md @@ -0,0 +1,215 @@ +# loo v3.0.0 clean-up: open TODOs + +> Last update: 2026-09-29 +> +> Author: Florence Bockting + +The PRs and issues that relate to this work are in +[loo-v3-related-prs-issues.md](loo-v3-related-prs-issues.md). +The names of the documents are in +[loo-v3-documentation.md](loo-v3-documentation.md). Use only those names. + +## Tutorials (new in loo v3) + +### `vignettes/articles-online-only/overview-measures.Rmd` + +*Model performance: Overview of predictive measures*. PR #363 "pred_measure". + +- [ ] Change the YAML `title:` and `\VignetteIndexEntry{}` to the name above. + Now: "Overview of predictive measures". Waits for: nothing. +- [ ] Search `(TODO-Vehtari et al., 2026)` in the introduction. Cite + *Predictive measures: A formula reference*. Waits for: arXiv ID. +- [ ] Search `our supplement` in the introduction. Cite + *Predictive measures: A formula reference*. Waits for: arXiv ID. +- [ ] Search `[TODO-article-suppl]` in the "Aggregation" bullet below the table. + Link the formula reference. Waits for: arXiv ID. +- [ ] Search `[TODO: arXiv ID]` in the reference list. Waits for: arXiv ID. +- [ ] Search `[TODO-article]` in the callout "What this article does not cover". + Link *Model performance: Predictive schemes with `pred_measure()`*. + Waits for: nothing. +- [ ] Search `[TODO: schemes and pred_measure() article]` in the reference list. + Link the same article. Waits for: nothing. +- [ ] Search `[model-comparison TODO]` in the callout. Link + *Model comparison: Case Study with `model_compare()`*. Waits for: issue + #401. +- [ ] Search `](model-comparison.html)`. Use the link text + *Model comparison: Explanation of `model_compare()`*. That article exists + only on `integrate-loo_compare`, so the link is broken on `pred_measure`. + Waits for: #380 merged. + +Use the `articles/articles-online-only/` URL form for all links. + +### `vignettes/articles-online-only/pred-measure-workflow.Rmd` + +*Model performance: Predictive schemes with `pred_measure()`*. PR #363. + +- [ ] Change the YAML `title:` and `\VignetteIndexEntry{}` to the name above. + Now: "Predictive schemes with pred_measure()". Waits for: nothing. +- [ ] Search `[TODO: arXiv ID]` in the reference list. Waits for: arXiv ID. +- [ ] Search `](model-comparison.html)`. Use the link text + *Model comparison: Explanation of `model_compare()`*. Waits for: #380 + merged. +- [ ] Search `LOAD-BRMS-GITHUB.txt`. Delete the `child=` chunk that loads brms + from GitHub. Waits for: brms on CRAN. + +### `vignettes/articles-online-only/model-comparison.Rmd` + +*Model comparison: Explanation of `model_compare()`*. PR #380. + +- [ ] Change the YAML `title:` and `\VignetteIndexEntry{}` to the name above. + Now: "Model comparison: Using the `model_compare()` function.". + Waits for: nothing. +- [ ] Search `TODO-ARXIV` and `TODO-Vehtari`. Waits for: arXiv ID. +- [ ] Search `the supplement`. Replace it with the name of the + formula reference. Waits for: arXiv ID. +- [ ] Search `Computing predictive performance measures` in "See also". Use + *Model performance: Predictive schemes with `pred_measure()`*. + Waits for: nothing. +- [ ] Search `TODO-case study` and `[TODO-LINK]`. Link + *Model comparison: Case Study with `model_compare()`*. Waits for: issue + #401. + +## Tutorials (existing) + +`loo_compare()` warns once per session (`.deprecate_once()`). A vignette build +therefore shows the warning. + +### `vignettes/loo2-large-data.Rmd` + +- [ ] Search `loo_compare(loo_ss_1, loo_ss_2)`. `integrate-loo_compare` + replaced the other 3 calls. Replace it with `model_compare()`. Check the + output against #393 first. Waits for: #363 merged, subsampling decision. + +### `vignettes/loo2-elpd.Rmd` + +- [ ] Search `elpd(log_pd` (two calls) and "using `elpd()`" (two sentences). + Replace them with `measure_elpd()` or `test_pred_measure()`. Waits for: #363 merged. + +### All `vignettes/*.Rmd` + +- [ ] Grep for `crps(`, `scrps(`, `loo_crps(`, `loo_scrps(` and + `loo_predictive_metric(`. Do the two files above first. Waits for: #363 + merged. + +## Other + +### `R/loo-glossary.R` + +*The `loo` glossary*. PR #380. + +- [ ] Add the general terms: measure, metric, score, utility, loss. + Waits for: nothing. + +## Project management + +### `vignettes/migration-guide.Rmd` + +*Migration guide*. PR #379, extended on #363, #380 and #378. + +- [ ] Merge the four versions. `loo-v3.0.0` has 182 lines, `pred_measure` 307, + `integrate-loo_compare` 337 and `parallelization` 216. #363 and #378 both + extend the file, so the second merge into `loo-v3.0.0` conflicts. + Waits for: #363 merged, #378 merged. +- [ ] Decide if the sections "Maintainer checklist" and "`_pkgdown.yml` + reference (this branch)" belong in a user vignette. Waits for: nothing. + +### `vignettes/articles-online-only/roadmap.Rmd` + +*Development roadmap*. PR #396. + +- [ ] Delete `vignettes/articles-online-only/roadmap.html` from the PR. The + PR's own TODO asks for this. Waits for: nothing. + +## R code + +### `R/pred_measure.R` + +PR #363. + +- [ ] Search `overview of scores and metrics`. Use + *Model performance: Overview of predictive measures*. Waits for: nothing. +- [ ] Search `pred-measure workflow article` (6 places). Use + *Model performance: Predictive schemes with `pred_measure()`*. + Waits for: nothing. + +### `R/pred_measure-compute.R` + +PR #363. + +- [ ] Search `See developer notes on computation`. Users cannot see `notes/`. + Cite the formula reference, or delete the `@note`. Waits for: nothing. +- [ ] Search `not yet implemented`. Decide: keep `group_ids` as a reserved + argument, or remove it. If you remove it, also delete the test in + `tests/testthat/test_pred_measure.R` (search `group_ids errors`). + Waits for: nothing. +- [ ] PR #380. Search `include this correction`. This is the warning for moment + matching and `reloo()`: only `elpd`, `mlpd` and `ic` are correct. Decide: + ship v3.0.0 with the warning, or wait for #399. Waits for: #399. + +### `R/crps.R` + +PR #363. + +- [ ] Resolve the conflict with #398. Both PRs edit this file. If #398 merges + first, merge `master` into `loo-v3.0.0`, then `loo-v3.0.0` into + `pred_measure`. Waits for: nothing. + +## Tests + +### `tests/testthat/data-for-tests/test_data_generation.R` + +PR #363, #380. In 2026-09 `data-for-tests/` is 5.9 MB. CRAN allows a 5 MB +tarball. + +- [ ] Search `saveRDS(`. Add `compress = "xz"` to each call. Then re-save the + `.Rds` fixtures. The earlier re-save was never committed. It gave + 5.85 → 4.13 MB. Waits for: #363 merged (C2). +- [ ] Search `roaches_compare = 110` in `N_KEEP`. Change it to 53. Then + re-create `test_data_roaches_compare.Rds`. Waits for: #363 merged (C2). + +### `tests/testthat/test_compare.R` + +PR #380. + +- [ ] Search the assertion that expects `diag_diff` to be `""`. With 53 + observations it becomes `"N < 100"`. Rewrite it. Do this with the `N_KEEP` + change above. + +## Package files + +### `NEWS.md` + +PR #363. + +- [ ] Search `[supported_measures_list()]`. Use backticks, as for every other + name. Waits for: nothing. +- [ ] Search `vignette("migration-guide")`. Add `package = "loo"`, as in the + roxygen. Waits for: nothing. + +### `.Rbuildignore` + +- [ ] Add `^notes$`. `loo-v3.0.0` does not exclude `notes/` yet. + Waits for: nothing. + +### `.github/workflows/pkgdown.yaml` + +PR #363. + +- [ ] Search `paul-buerkner/brms`. Replace it with `any::brms`. + Waits for: brms on CRAN (C7). + +## Branches and PRs (no document) + +- [ ] Merge `pred_measure` into `integrate-loo_compare`. Waits for: nothing. +- [ ] Finish the description of #380. The draft is outside the repo. Waits for: nothing. +- [ ] Update the description of #363. "Known limitations" says + "`loo_compare` integration still outstanding". #380 does that. Mark #220 as + fixed, not partial. Waits for: #380 merged. +- [ ] Add "Fixes #281", "Fixes #223", "Fixes #213", "Fixes #201", + "Fixes #135" and "Fixes #220" to the description of #379. GitHub does not + close them from #363, because #363 does not merge into `master`. + Waits for: #363 merged. +- [ ] Measure the tarball of `master` with `R CMD build`. It gives the size + budget. Waits for: nothing. +- [ ] Run `R CMD build` with the vignettes on this branch. Only this gives the + real size. Waits for: the two fixture items above. \ No newline at end of file diff --git a/notes/loo-v3-documentation.md b/notes/loo-v3-documentation.md new file mode 100644 index 00000000..74c4927f --- /dev/null +++ b/notes/loo-v3-documentation.md @@ -0,0 +1,47 @@ +# loo v3.0.0: documentation overview + +> Last update: 2026-09-29. +> +> Author: Florence Bockting + +Related notes: [loo-v3-cleanup-todos.md](loo-v3-cleanup-todos.md), +[loo-v3-related-prs-issues.md](loo-v3-related-prs-issues.md). + +## 1. Tutorials (existing) — users + +| Document | Name | Status | Branch / PR | +| :-- | :-- | :-- | :-- | +| `vignettes/loo2-example.Rmd` | Using the loo package | changed: `model_compare()` | `integrate-loo_compare` (#380) | +| `vignettes/loo2-with-rstan.Rmd` | Writing Stan programs for use with the loo package | changed: `model_compare()` | `integrate-loo_compare` (#380) | +| `vignettes/loo2-elpd.Rmd` | Holdout validation and K-fold cross-validation | changed: prose; `elpd()` calls left | `integrate-loo_compare` (#380) | +| `vignettes/loo2-large-data.Rmd` | Using leave-one-out cross-validation for large data | changed: 3 of 4 `loo_compare()` calls replaced | `integrate-loo_compare` (#380) | +| `vignettes/loo2-weights.Rmd` | Bayesian stacking and pseudo-BMA weights | changed: parallelism | `parallelization` (#378) | +| `vignettes/loo2-non-factorized.Rmd` | Leave-one-out cross-validation for non-factorized models | unchanged | `master` | +| `vignettes/loo2-lfo.Rmd` | Approximate leave-future-out cross-validation | unchanged | `master` | +| `vignettes/loo2-moment-matching.Rmd` | Avoiding model refits with moment matching | unchanged | `master` | +| `vignettes/loo2-mixis.Rmd` | Mixture IS leave-one-out cross-validation | unchanged | `master` | +| external: `https://users.aalto.fi/~ave/CV-FAQ.html` | Cross-validation FAQ | unchanged | `master` (`_pkgdown.yml`) | + +## 2. Tutorials (new in loo v3) — users + +| Document | Name | Status | Branch / PR | +| :-- | :-- | :-- | :-- | +| `vignettes/articles-online-only/overview-measures.Rmd` | Model performance: Overview of predictive measures | exists; placeholders left | `pred_measure` (#363) | +| `vignettes/articles-online-only/pred-measure-workflow.Rmd` | Model performance: Predictive schemes with `pred_measure()` | exists; placeholder left | `pred_measure` (#363) | +| `vignettes/articles-online-only/model-comparison.Rmd` | Model comparison: Explanation of `model_compare()` | exists; placeholders left | `integrate-loo_compare` (#380) | +| not yet created | Model comparison: Case Study with `model_compare()` | planned | issue #401 | + +## 3. Other — users + +| Document | Name | Status | Branch / PR | +| :-- | :-- | :-- | :-- | +| `R/loo-glossary.R` | The `loo` glossary | extended; general terms planned | `integrate-loo_compare` (#380) | +| ArXiV preprint | Predictive measures: A formula reference | in progress; no arXiv ID yet | in Overleaf | + +## 4. Project management — users and developers + +| Document | Name | Status | Branch / PR | +| :-- | :-- | :-- | :-- | +| `vignettes/migration-guide.Rmd` | Migration guide | exists; four versions on four branches | `loo-v3.0.0` (#379), extended on #363, #380, #378 | +| `vignettes/articles-online-only/roadmap.Rmd` | Development roadmap | draft | `roadmap` (#396) | + diff --git a/notes/loo-v3-related-prs-issues.md b/notes/loo-v3-related-prs-issues.md new file mode 100644 index 00000000..74d96f17 --- /dev/null +++ b/notes/loo-v3-related-prs-issues.md @@ -0,0 +1,84 @@ +# loo v3.0.0: related PRs and issues + +> Last update: 2026-09-29 +> +> Author: Florence Bockting + +This note lists the PRs and issues that relate to the v3.0.0 refactoring. +The open work is in [loo-v3-cleanup-todos.md](loo-v3-cleanup-todos.md). + +## Merge chain + +| PR | Head branch | Base branch | Brings | +| :-- | :-- | :-- | :-- | +| #380 | `integrate-loo_compare` | `pred_measure` | `model_compare()`, `model-comparison.Rmd`, `notes/design-discussions/`, compare fixture, deprecation of `loo_compare()` | +| #363 (draft) | `pred_measure` | `loo-v3.0.0` | `*_pred_measure()`, `measure_*()`, `overview-measures.Rmd`, `pred-measure-workflow.Rmd`, deprecations of `elpd()`, `crps()` and others, test fixtures | +| #379 (draft) | `loo-v3.0.0` | `master` | the release | +| this | `loo-v3-cleanup` | `loo-v3.0.0` | the items below | + +**Targeted for v3.0.x:** + +| PR | Head branch | Base branch | Brings | +| :-- | :-- | :-- | :-- | +| #378 (draft) | `parallelization` | `loo-v3.0.0` | mirai/mori parallelism, fixes #308. Not part of v3.0.0, but it merges into the same base. | + +## Issues that the refactoring fixes + +| Issue | Title | Fixed by | +| :-- | :-- | :-- | +| #281 | New functions for better support for different scores and metrics | #363 | +| #223 | (loo_)(s)crps could ask for only one argument with predictions | #363 | +| #213 | loo_predictive_metric and loo_crps could accept psis objects | #363 | +| #201 | Add R2 | #363 | +| #135 | user defined loss/utility functions | #363 | +| #220 | loo_compare for crps and loo_crps | #363 (part), #380 (rest) | + +All six are open. GitHub closes an issue from a "Fixes #n" line only when the +PR merges into `master`. #363 merges into `loo-v3.0.0`. + +## Open PRs into `master` that touch the same code + +| PR | Topic | Relation to this refactoring | +| :-- | :-- | :-- | +| #398 | `crps.numeric()` passes `permutations` wrongly (fixes #397) | Edits `R/crps.R`. #363 deprecates `crps()` in the same file. Merge conflict. | +| #393 (draft) | `loo_compare()` returns a data.frame for subsampling (fixes #392) | #380 deprecates `loo_compare()`. | +| #399 (draft) | Correct `pred_measure` values after post-hoc methods | | +| #340 | Export `srs_diff_est()` (fixes #333) | Subsampling comparison. `model_compare()` has a subsampling method. | +| #178 (draft) | LOO difference plot | #394 (milestone v3.0.0) extends it to the new measures. | +| #291 (draft) | Format the project with Air | Rewrites every file. Merge it before #363 or after #379, never between them. | +| #396 | Roadmap on the website | Describes the v3.0.0 plan. Update it when this list changes. | + +## Other issues, by target release + +All of these issues have the GitHub milestone v3.0.0. This list gives the +release that each one targets. + +**v3.0.0** + ++ #401 (case study for model comparison) ++ #388 (clean up issues): general clean-up and release preparation ++ #353 (warnings and messages): standardize and improve warning/messaging behavior implemented in loo + +**v3.0.0, developer discussion** + ++ #384 (LLM code optimizations): How do we want to integrate this in the development process + +**v3.0.0 and later** + ++ #343 (subset psis objects) ++ #249 (more `posterior` functions): avoid duplicated code and decide which functions should go into posterior and which should stay in loo ++ #394 (extend the difference plot) + +**v3.0.x** + ++ #308 / #378 (parallelism): improve parallelization in loo + +**v3.x.x** + ++ (loo diagnostics): improved posthoc diagnostics using loo ++ #192 (n_eff to ESS): improve diagnostics + +**Target not decided** + ++ #385 (mutation testing) +