Add debugging page and benchmarking and profiling guidance

_quarto.yml

@@ -90,6 +90,7 @@ website:
             - usage/varnamedtuple/index.qmd
             - usage/vectorisation/index.qmd
             - usage/external-samplers/index.qmd
+            - usage/debugging/index.qmd
             - usage/troubleshooting/index.qmd
 
         - section: "Tutorials"
@@ -202,6 +203,7 @@ seasonal-time-series: tutorials/bayesian-time-series-analysis
 
 usage-automatic-differentiation: usage/automatic-differentiation
 usage-custom-distribution: usage/custom-distribution
+usage-debugging: usage/debugging
 usage-dynamichmc: usage/dynamichmc
 usage-external-samplers: usage/external-samplers
 usage-external-likelihoods: usage/external-likelihoods

contributing/start-contributing/index.qmd

@@ -15,7 +15,7 @@ If you're not sure where to begin, feel free to ask on any issue, or come say he
 - **Bug reports and edge cases:** If you run into something that doesn't work, please [open an issue](https://github.com/TuringLang/Turing.jl/issues)! If you aren't sure which repository to post on, don't worry: just post it on the main Turing.jl repo and we can triage it.
 
   We're especially keen on finding cases where automatic differentiation fails.
-  If you find a model that doesn't work with AD, please report it, and we can either help to report it upstream, and/or add it to [the ADTests website](https://turinglang.org/ADTests/).
+  If you find a model that doesn't work with AD, please report it, and we can help to report it upstream.
 
 - **Tutorials and example models:** If you have an interesting model that you think others could learn from, we'd love to hear about it!
   You don't have to write a full docs page immediately; just open an issue and we'd be very happy to chat about it.

core-functionality/index.qmd

@@ -587,7 +587,7 @@ The default AD backend is [ForwardDiff](https://github.com/JuliaDiff/ForwardDiff
 Three other backends are also supported: [Mooncake](https://github.com/chalk-lab/Mooncake.jl), [ReverseDiff](https://github.com/JuliaDiff/ReverseDiff.jl), and [Enzyme](https://github.com/EnzymeAD/Enzyme.jl).
 These require the user to explicitly load them, for example `import Mooncake` next to `using Turing`.
 
-For more information on Turing's automatic differentiation backend, please see the [Automatic Differentiation]({{<meta usage-automatic-differentiation>}}) article as well as the [ADTests website](https://turinglang.org/ADTests/), where a number of AD backends (not just those above) are tested against Turing.jl.
+For more information on Turing's automatic differentiation backend, please see the [Automatic Differentiation]({{<meta usage-automatic-differentiation>}}) article.
 
 #### Progress Logging
 

faq/index.qmd

@@ -82,7 +82,7 @@ model = setthreadsafe(f(y), true)
 
 - **Observe statements**: Generally safe to use in threaded loops
 - **Assume statements** (sampling statements): Often crash unpredictably or produce incorrect results
-- **AD backend compatibility**: Many AD backends don't support threading. Check the [multithreaded column in ADTests](https://turinglang.org/ADTests/) for compatibility
+- **AD backend compatibility**: Many AD backends don't support threading. See the [Threadsafe Evaluation]({{< meta usage-threadsafe-evaluation >}}) page for which backends work reliably
 
 ## How do I check the type stability of my Turing model?
 
@@ -95,6 +95,7 @@ Type stability is crucial for performance. Check out:
 
 For debugging both statistical and syntactical issues:
 
+- [Debugging Models]({{< meta usage-debugging >}}) - tools for checking model validity, type stability, and gradients
 - [Troubleshooting Guide]({{< meta usage-troubleshooting >}}) - common errors and their solutions
 - For more advanced debugging, DynamicPPL provides [the `DynamicPPL.DebugUtils` module](https://turinglang.org/DynamicPPL.jl/stable/api/#Debugging-Utilities) for inspecting model internals
 
@@ -139,7 +140,6 @@ The choice of AD backend can significantly impact performance. See:
 
 - [Automatic Differentiation Guide]({{< meta usage-automatic-differentiation >}}) - comprehensive comparison of ForwardDiff, Mooncake, ReverseDiff, and other backends
 - [Performance Tips]({{< meta usage-performance-tips >}}#choose-your-ad-backend) - quick guide on choosing backends
-- [AD Backend Benchmarks](https://turinglang.org/ADTests/) - performance comparisons across various models
 
 ## I changed one line of my model and now it's so much slower; why?
 

usage/automatic-differentiation/index.qmd

@@ -65,14 +65,6 @@ AbstractPPL's interface provides a `prepare` step that performs a one-time setup
 Turing will automatically perform this preparation for you when calling functions such as `sample` or `vi`, so you do not need to worry about this step.
 :::
 
-### ADTests
-
-Before describing how to choose the best AD backend for your model, we should mention that we also publish a table of benchmarks for various models and AD backends in [the ADTests website](https://turinglang.org/ADTests/).
-These models aim to capture a variety of different features of Turing.jl and Julia in general, so that you can see which AD backends may be compatible with your model.
-Benchmarks are also included, although it should be noted that many of the models in ADTests are small and thus the timings may not be representative of larger, real-life models.
-
-If you have suggestions for other models to include, please do let us know by [creating an issue on GitHub](https://github.com/TuringLang/ADTests/issues/new)!
-
 ### The Best AD Backend for Your Model
 
 Given the number of possible backends, how do you choose the best one for your model?

usage/debugging/index.qmd

@@ -0,0 +1,89 @@
+---
+title: Debugging Models
+engine: julia
+---
+
+```{julia}
+#| echo: false
+#| output: false
+using Pkg;
+Pkg.instantiate();
+```
+
+This page describes the tools available for checking that a Turing model is defined correctly.
+It complements the [Troubleshooting]({{<meta usage-troubleshooting>}}) page, which explains specific error messages: the tools here help you find problems before (or without) hitting an error.
+
+```{julia}
+using Turing
+```
+
+## Checking model validity with `check_model`
+
+`DynamicPPL.check_model` evaluates a model once and checks it for common problems.
+It returns `true` if the check passes, and `false` (with warnings explaining the problem) otherwise.
+
+Specifically, it checks for:
+
+- the same variable, or overlapping variables (such as `x` and `x[1]`), appearing on the left-hand side of more than one tilde statement;
+- `NaN` values on the left-hand side of observe statements;
+- with the keyword argument `fail_if_discrete=true`, the presence of discrete random variables, which gradient-based samplers such as HMC and NUTS cannot handle.
+
+It is equally important to know what it does not catch.
+The model is evaluated a single time, with values drawn from the prior, so problems that only occur for some parameter values (for example, in a branch that this particular evaluation does not take) can be missed.
+It also checks only the structure of the model, not its statistical soundness: it cannot tell you that a prior is unreasonable or that parameters are not identifiable.
+Type instability and incorrect gradients are separate concerns, covered by the checks in the following sections.
+
+For example, this model observes a `NaN` value, which usually indicates a bug in the data preparation:
+
+```{julia}
+using DynamicPPL: check_model
+
+@model function gaussian(y)
+    x ~ Normal()
+    y ~ Normal(x)
+end
+
+check_model(gaussian(NaN))
+```
+
+whereas the same model with valid data passes:
+
+```{julia}
+check_model(gaussian(1.5))
+```
+
+Turing runs this check automatically at the start of `sample` (and the mode estimation functions).
+For gradient-based samplers, this includes the discrete-variable check.
+If the check reports a false positive for your model, you can disable it by passing `check_model=false` to `sample`.
+
+The keyword argument `error_on_failure=true` makes `check_model` throw an error instead of returning `false`, which is useful in scripts and tests.
+
+## Checking type stability
+
+Type instability is one of the most common reasons for a model being slower than expected.
+`DynamicPPL.DebugUtils.model_warntype(model)` works like `@code_warntype`, but for an entire model: types that the compiler cannot infer are marked in red in the output.
+The [Performance Tips]({{<meta usage-performance-tips>}}#ensure-that-types-in-your-model-can-be-inferred) page has a worked example, along with advice on how to fix the instabilities it finds.
+There is also `DynamicPPL.DebugUtils.model_typed`, which returns the typed representation instead of printing it.
+
+## Checking gradients
+
+If a model samples poorly with a gradient-based sampler such as NUTS, it is worth checking whether the AD backend computes correct gradients for it.
+`DynamicPPL.TestUtils.AD.run_ad` evaluates the gradient of a model with a given backend and, by default, tests it against a reference backend:
+
+```{julia}
+using ADTypes: AutoReverseDiff
+using DynamicPPL.TestUtils.AD: run_ad
+using ReverseDiff
+
+run_ad(gaussian(1.5), AutoReverseDiff())
+```
+
+If the gradient is incorrect, this errors.
+For debugging `NaN` or infinite gradients, see the [Troubleshooting]({{<meta usage-troubleshooting>}}) page.
+
+## Further help
+
+This page covers the most commonly useful tools.
+DynamicPPL provides a few more, such as `DynamicPPL.has_static_constraints`; the full list is in [the DynamicPPL API documentation](https://turinglang.org/DynamicPPL.jl/stable/api/#Debugging-Utilities).
+
+If these tools do not surface the problem, feel free to [open an issue](https://github.com/TuringLang/Turing.jl/issues), ideally with the smallest model that reproduces it.

usage/performance-tips/index.qmd

@@ -15,6 +15,52 @@ Pkg.instantiate();
 This section briefly summarises a few common techniques to ensure good performance when using Turing.
 We refer to [the Julia documentation](https://docs.julialang.org/en/v1/manual/performance-tips/index.html) for general techniques to ensure good performance of Julia programs.
 
+## Benchmark and profile your model
+
+Before changing your model for performance reasons, it is worth measuring where the time actually goes.
+When sampling with a gradient-based method such as NUTS, almost all of the time is spent evaluating the log density of the model and its gradient.
+You can time both with `DynamicPPL.TestUtils.AD.run_ad`:
+
+```{julia}
+using Turing
+using ADTypes: AutoReverseDiff
+using DynamicPPL.TestUtils.AD: run_ad
+using ReverseDiff
+
+@model function bmodel(x)
+    m ~ Normal()
+    s ~ truncated(Normal(); lower=0)
+    return x ~ MvNormal(fill(m, length(x)), s^2 * I)
+end
+
+result = run_ad(bmodel(randn(100)), AutoReverseDiff(); benchmark=true)
+result.primal_time, result.grad_time
+```
+
+Both times are median seconds per evaluation.
+If the gradient takes many times longer than the log density itself, the AD backend is the thing to change: see [Choose your AD backend](#choose-your-ad-backend) below, and the [Automatic Differentiation]({{<meta usage-automatic-differentiation>}}) page for how to compare several backends on your model.
+If the log density itself is slow, the model is the thing to change, and the remaining sections on this page apply.
+
+To find out where the time goes within a single evaluation, you can use [Julia's built-in profiler](https://docs.julialang.org/en/v1/manual/profile/) on the model's log density:
+
+```{julia}
+#| eval: false
+using DynamicPPL, LogDensityProblems, Profile
+
+model = bmodel(randn(100))
+ldf = LogDensityFunction(model, getlogjoint_internal, LinkAll(); adtype=AutoReverseDiff())
+x = rand(ldf)
+
+Profile.@profile for _ in 1:10_000
+    LogDensityProblems.logdensity_and_gradient(ldf, x)
+end
+Profile.print(; mincount=10)
+```
+
+This profiles the model evaluation and the gradient computation together, which is what a sampler actually runs.
+We do not run the profiler on this page because its output is long and machine-specific: `Profile.print` shows a tree of function calls, each annotated with the number of samples in which it appeared, and the hot spots are the lines with the largest counts (`mincount=10` hides rarely-seen calls).
+Graphical viewers such as `@profview` in VS Code, [ProfileView.jl](https://github.com/timholy/ProfileView.jl), or [PProf.jl](https://github.com/JuliaPerf/PProf.jl) make the output much easier to read than `Profile.print`.
+
 ## Use multivariate distributions
 
 It is generally preferable to use multivariate distributions if possible.

usage/threadsafe-evaluation/index.qmd

@@ -242,8 +242,6 @@ Note that for Enzyme, you should use a relatively recent version (at least v0.13
 In contrast, ReverseDiff sometimes gives right results, but quite often gives incorrect gradients.
 Mooncake [currently does not support multithreading at all](https://github.com/chalk-lab/Mooncake.jl/issues/570).
 
-For more details you can take a look at [the `threaded_...` models on ADTests](https://turinglang.org/ADTests/).
-
 ## Under the hood
 
 :::{.callout-note}