Feat parameter recovery plot#61
Conversation
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #61 +/- ##
==========================================
+ Coverage 95.87% 97.12% +1.24%
==========================================
Files 7 10 +3
Lines 631 904 +273
==========================================
+ Hits 605 878 +273
Misses 26 26 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
|
||
| def plot_parameter_recovery( | ||
| sbc, | ||
| ci_prob=0.94, |
There was a problem hiding this comment.
Do we want to keep this in line with ArviZ? If that's the case, we can use the hardcoded value (the default value in ArviZ is 0.89) or take the value from arviz's rcparams.
There was a problem hiding this comment.
More generally, I think we want to make this plot more arvizian. We are already using plot_ecdf_pit from ArviZ. By this I mean using plot_collection, visuals, etc, see https://python.arviz.org/projects/plots/en/latest/tutorials/plots_intro.html
| return fig | ||
|
|
||
|
|
||
| def plot_ecdf(sbc, if_show=True): |
There was a problem hiding this comment.
ArviZ plots works with 3 backends, matplotlib, plotly and bokeh, this assumes it only use matplotlib (plt.show()). ArviZ's plots return a plotcollection object, using "pc" or "plot_collection" is a better choice that "fig"
There was a problem hiding this comment.
ok, yeah I'll try to make it more arvizy
Co-authored-by: Osvaldo A Martin <aloctavodia@gmail.com>
3 participants
resolves #40
This implements the parameter recovery plot. This requires keep_fits=True, as we need it to compute the mean of each simulation, the credible interval, etc.
The backend adapters is moved to their own folders. The plotting function is added to plots.py, a wrapper for arviz_plots.plot_ecdf_pit is also added. This means we no longer need to manually import arviz_plots to get plot_ecdf_pit
The credible interval is quantile-based, 94% CI ranges from 3rd percentile, to 97th percentile.
The title indicates a expected coverage using the binomial distribution of the prior value landing in the CI, the empirical fraction of simulations whose CI contains the true/prior value, this expected coverage uses the binomial standard error, which accounts for the sampling noise from having a finite number sbc simulations, but doesn't consider the noise in the estimation of the CI from finite MCMC draws.
Implements tests for plots.py.