Repository navigation
Expand file tree
/
Copy pathindex.html
More file actions
1570 lines (1500 loc) · 90.6 KB
/
Copy pathindex.html
File metadata and controls
1570 lines (1500 loc) · 90.6 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<!DOCTYPE html>
<html lang="en"><head>
<script src="presentation_files/libs/clipboard/clipboard.min.js"></script>
<script src="presentation_files/libs/quarto-html/tabby.min.js"></script>
<script src="presentation_files/libs/quarto-html/popper.min.js"></script>
<script src="presentation_files/libs/quarto-html/tippy.umd.min.js"></script>
<link href="presentation_files/libs/quarto-html/tippy.css" rel="stylesheet">
<link href="presentation_files/libs/quarto-html/light-border.css" rel="stylesheet">
<link href="presentation_files/libs/quarto-html/quarto-syntax-highlighting-7f8f88aac4f3542376d5c11b86a4c14d.css" rel="stylesheet" id="quarto-text-highlighting-styles"><meta charset="utf-8">
<meta name="generator" content="quarto-1.9.37">
<meta name="author" content="Jonas Arruda University of Bonn">
<meta name="dcterms.date" content="2026-08-05">
<title>Simulation-Based Inference with Generative Neural Networks</title>
<meta name="apple-mobile-web-app-capable" content="yes">
<meta name="apple-mobile-web-app-status-bar-style" content="black-translucent">
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no, minimal-ui">
<link rel="stylesheet" href="presentation_files/libs/revealjs/dist/reset.css">
<link rel="stylesheet" href="presentation_files/libs/revealjs/dist/reveal.css">
<style>
/* Default styles provided by pandoc.
** See https://pandoc.org/MANUAL.html#variables-for-html for config info.
*/
code{white-space: pre-wrap;}
span.smallcaps{font-variant: small-caps;}
div.columns{display: flex; gap: min(4vw, 1.5em);}
div.column{flex: auto; overflow-x: auto;}
div.hanging-indent{margin-left: 1.5em; text-indent: -1.5em;}
ul.task-list{list-style: none;}
ul.task-list li input[type="checkbox"] {
width: 0.8em;
margin: 0 0.8em 0.2em -1em; /* quarto-specific, see https://github.com/quarto-dev/quarto-cli/issues/4556 */
vertical-align: middle;
}
/* CSS for syntax highlighting */
html { -webkit-text-size-adjust: 100%; }
pre > code.sourceCode { white-space: pre; position: relative; }
pre > code.sourceCode > span { display: inline-block; line-height: 1.25; }
pre > code.sourceCode > span:empty { height: 1.2em; }
.sourceCode { overflow: visible; }
code.sourceCode > span { color: inherit; text-decoration: inherit; }
div.sourceCode { margin: 1em 0; }
pre.sourceCode { margin: 0; }
@media screen {
div.sourceCode { overflow: auto; }
}
@media print {
pre > code.sourceCode { white-space: pre-wrap; }
pre > code.sourceCode > span { text-indent: -5em; padding-left: 5em; }
}
pre.numberSource code
{ counter-reset: source-line 0; }
pre.numberSource code > span
{ position: relative; left: -4em; counter-increment: source-line; }
pre.numberSource code > span > a:first-child::before
{ content: counter(source-line);
position: relative; left: -1em; text-align: right; vertical-align: baseline;
border: none; display: inline-block;
-webkit-touch-callout: none; -webkit-user-select: none;
-khtml-user-select: none; -moz-user-select: none;
-ms-user-select: none; user-select: none;
padding: 0 4px; width: 4em;
color: #aaaaaa;
}
pre.numberSource { margin-left: 3em; border-left: 1px solid #aaaaaa; padding-left: 4px; }
div.sourceCode
{ color: #003b4f; background-color: #f1f3f5; }
@media screen {
pre > code.sourceCode > span > a:first-child::before { text-decoration: underline; }
}
code span { color: #003b4f; } /* Normal */
code span.al { color: #ad0000; } /* Alert */
code span.an { color: #5e5e5e; } /* Annotation */
code span.at { color: #657422; } /* Attribute */
code span.bn { color: #ad0000; } /* BaseN */
code span.bu { } /* BuiltIn */
code span.cf { color: #003b4f; font-weight: bold; } /* ControlFlow */
code span.ch { color: #20794d; } /* Char */
code span.cn { color: #8f5902; } /* Constant */
code span.co { color: #5e5e5e; } /* Comment */
code span.cv { color: #5e5e5e; font-style: italic; } /* CommentVar */
code span.do { color: #5e5e5e; font-style: italic; } /* Documentation */
code span.dt { color: #ad0000; } /* DataType */
code span.dv { color: #ad0000; } /* DecVal */
code span.er { color: #ad0000; } /* Error */
code span.ex { } /* Extension */
code span.fl { color: #ad0000; } /* Float */
code span.fu { color: #4758ab; } /* Function */
code span.im { color: #00769e; } /* Import */
code span.in { color: #5e5e5e; } /* Information */
code span.kw { color: #003b4f; font-weight: bold; } /* Keyword */
code span.op { color: #5e5e5e; } /* Operator */
code span.ot { color: #003b4f; } /* Other */
code span.pp { color: #ad0000; } /* Preprocessor */
code span.sc { color: #5e5e5e; } /* SpecialChar */
code span.ss { color: #20794d; } /* SpecialString */
code span.st { color: #20794d; } /* String */
code span.va { color: #111111; } /* Variable */
code span.vs { color: #20794d; } /* VerbatimString */
code span.wa { color: #5e5e5e; font-style: italic; } /* Warning */
/* CSS for citations */
div.csl-bib-body { }
div.csl-entry {
clear: both;
}
.hanging-indent div.csl-entry {
margin-left:2em;
text-indent:-2em;
}
div.csl-left-margin {
min-width:2em;
float:left;
}
div.csl-right-inline {
margin-left:2em;
padding-left:1em;
}
div.csl-indent {
margin-left: 2em;
} </style>
<link rel="stylesheet" href="presentation_files/libs/revealjs/dist/theme/quarto-25190c6ea0be213ef7a37cd96698819e.css">
<script>window.backupDefine = window.define; window.define = undefined;</script><script src="https://cdn.jsdelivr.net/npm/katex@latest/dist/katex.min.js"></script>
<script>document.addEventListener("DOMContentLoaded", function () {
var mathElements = document.getElementsByClassName("math");
var macros = [];
for (var i = 0; i < mathElements.length; i++) {
var texText = mathElements[i].firstChild;
if (mathElements[i].tagName == "SPAN") {
katex.render(texText.data, mathElements[i], {
displayMode: mathElements[i].classList.contains('display'),
throwOnError: false,
macros: macros,
fleqn: false
});
}}});
</script>
<script>window.define = window.backupDefine; window.backupDefine = undefined;</script><link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/katex@latest/dist/katex.min.css">
<link href="presentation_files/libs/revealjs/plugin/quarto-line-highlight/line-highlight.css" rel="stylesheet">
<link href="presentation_files/libs/revealjs/plugin/reveal-menu/menu.css" rel="stylesheet">
<link href="presentation_files/libs/revealjs/plugin/reveal-menu/quarto-menu.css" rel="stylesheet">
<link href="presentation_files/libs/revealjs/plugin/quarto-support/footer.css" rel="stylesheet">
<style type="text/css">
.reveal div.sourceCode {
margin: 0;
overflow: auto;
}
.reveal div.hanging-indent {
margin-left: 1em;
text-indent: -1em;
}
.reveal .slide:not(.center) {
height: 100%;
}
.reveal .slide.scrollable {
overflow-y: auto;
}
.reveal .footnotes {
height: 100%;
overflow-y: auto;
}
.reveal .slide .absolute {
position: absolute;
display: block;
}
.reveal .footnotes ol {
counter-reset: ol;
list-style-type: none;
margin-left: 0;
}
.reveal .footnotes ol li:before {
counter-increment: ol;
content: counter(ol) ". ";
}
.reveal .footnotes ol li > p:first-child {
display: inline-block;
}
.reveal .slide ul,
.reveal .slide ol {
margin-bottom: 0.5em;
}
.reveal .slide ul li,
.reveal .slide ol li {
margin-top: 0.4em;
margin-bottom: 0.2em;
}
.reveal .slide ul[role="tablist"] li {
margin-bottom: 0;
}
.reveal .slide ul li > *:first-child,
.reveal .slide ol li > *:first-child {
margin-block-start: 0;
}
.reveal .slide ul li > *:last-child,
.reveal .slide ol li > *:last-child {
margin-block-end: 0;
}
.reveal .slide .columns:nth-child(3) {
margin-block-start: 0.8em;
}
.reveal blockquote {
box-shadow: none;
}
.reveal .tippy-content>* {
margin-top: 0.2em;
margin-bottom: 0.7em;
}
.reveal .tippy-content>*:last-child {
margin-bottom: 0.2em;
}
.reveal .slide > img.stretch.quarto-figure-center,
.reveal .slide > img.r-stretch.quarto-figure-center {
display: block;
margin-left: auto;
margin-right: auto;
}
.reveal .slide > img.stretch.quarto-figure-left,
.reveal .slide > img.r-stretch.quarto-figure-left {
display: block;
margin-left: 0;
margin-right: auto;
}
.reveal .slide > img.stretch.quarto-figure-right,
.reveal .slide > img.r-stretch.quarto-figure-right {
display: block;
margin-left: auto;
margin-right: 0;
}
</style>
</head>
<body class="quarto-light">
<div class="reveal">
<div class="slides">
<section id="title-slide" class="quarto-title-block center">
<h1 class="title">Simulation-Based Inference with Generative Neural Networks</h1>
<p class="subtitle">ProbAI School 2026</p>
<div class="quarto-title-authors">
<div class="quarto-title-author">
<div class="quarto-title-author-name">
Jonas Arruda<br>University of Bonn
</div>
</div>
</div>
<p class="date">2026-08-05</p>
</section>
<section id="warm-up" class="slide level2" data-background-color="#fcba00">
<h2>Warm-Up</h2>
<div class="poll-embed">
<iframe src="https://wall.sli.do/event/w2t8HrKer4May9M26mADCT?section=9adab690-fd83-4411-863b-debe7f0798f2" height="100%" width="100%" frameborder="0" style="min-height: 560px;" allow="clipboard-write" title="Slido">
</iframe>
</div>
</section>
<section id="overview" class="slide level2">
<h2>Overview</h2>
<div class="fragment">
<ol type="1">
<li><p><strong>What is SBI?</strong> Introduction, Bayesian model types, normalizing flows.</p>
<p>→ <em>Exercise 1: from MCMC to amortized inference.</em></p></li>
</ol>
</div>
<div class="fragment">
<ol start="2" type="1">
<li><p><strong>Did it work?</strong> Calibration checking, model misspecification.</p>
<p>→ <em>Exercise 2: diagnostics on an epidemic time-series model.</em></p></li>
</ol>
</div>
<div class="fragment">
<ol start="3" type="1">
<li><p><strong>How to leverage modern generative models?</strong> Diffusion models, flow matching, consistency models.</p>
<p>→ <em>Exercise 3: diffusion models with post-hoc guidance.</em></p></li>
</ol>
</div>
</section>
<section>
<section id="part-1-what-is-simulation-based-inference" class="title-slide slide level1 special center" data-background-color="#004e9f">
<h1>Part 1: What is Simulation-Based Inference?</h1>
</section>
<section id="learning-densities" class="slide level2">
<h2>Learning Densities</h2>
<video src="media/videos/manim_diffusion/1080p60/ReverseODE.mp4" data-autoplay="" preload="auto" loop="" muted="" playsinline="" style="max-height:560px;display:block;margin:0 auto;">
</video>
<p>Everything today is about transporting a Gaussian onto our target, here a checkerboard.</p>
</section>
<section id="everything-starts-with-a-simulator" class="slide level2">
<h2>Everything Starts with a Simulator</h2>
<p>A <strong>prior</strong> and a scientific <strong>simulator</strong> together define the <em>forward process</em> from unknown parameters <span class="math inline">\boldsymbol{\theta}</span> to observables <span class="math inline">\mathbf{y}</span>:</p>
<p><span class="math display">
\boldsymbol{\theta} \sim p(\boldsymbol{\theta}), \qquad
\mathbf{y} = \operatorname{Sim}(\boldsymbol{\theta}, \mathbf{u}),\quad \mathbf{u}\sim\text{RNG}(\cdot)
</span></p>
<img data-src="media/images/stefan_forward-process.png" alt="SBI forward process: prior to observables" class="r-stretch"><p>Parameters and data are drawn from the joint <span class="math inline">\boldsymbol{\theta}, \mathbf{y} \sim p(\boldsymbol{\theta}, \mathbf{y})=p(\boldsymbol{\theta})\, p(\mathbf{y} \mid \boldsymbol{\theta})</span>: our <strong>Bayesian model</strong>.</p>
<p><em>Figure: S. Radev, BayesFlow.</em></p>
</section>
<section id="a-simple-toy-simulator" class="slide level2">
<h2>A Simple Toy Simulator</h2>
<p>3-segment planar robot arm <span class="citation" data-cites="kruse2021benchmarking">(<a href="#/additional-resources" role="doc-biblioref" onclick="">Kruse et al., 2021</a>)</span></p>
<div class="columns">
<div class="column" style="width:45%;">
<ul>
<li><strong>Parameters</strong> (not observed): <span class="math display">\boldsymbol{\theta}=(h,\alpha_1,\alpha_2,\alpha_3)</span></li>
<li><strong>Observation:</strong> end position <span class="math inline">\mathbf{y} = M(\boldsymbol{\theta})</span>.</li>
<li><strong>Task:</strong> recover <span class="math inline">p(\boldsymbol{\theta}\mid \mathbf{y})</span>.</li>
<li><strong>Challenge:</strong> different angles give the same observation <span class="math inline">\mathbf{y}</span>, so the posterior is <strong>multimodal</strong>.</li>
</ul>
</div><div class="column" style="width:55%;">
<div class="toy-sim">
<p><img id="toy-sim-img" src="media/images/toy_simulator/set-01.svg" alt="Three draws from the toy simulator: arm configurations and the end position they produce"> <button id="toy-sim-btn" type="button" class="sim-btn">↻ New simulations</button></p>
</div>
<script>
(function () {
var N = 12, i = 0;
var src = function (k) { return "media/images/toy_simulator/set-" + (k < 9 ? "0" : "") + (k + 1) + ".svg"; };
for (var k = 0; k < N; k++) { var pre = new Image(); pre.src = src(k); }
var img = document.getElementById("toy-sim-img");
document.getElementById("toy-sim-btn").addEventListener("click", function () {
i = (i + 1) % N;
img.src = src(i);
});
})();
</script>
</div></div>
</section>
<section id="forward-and-inverse" class="slide level2">
<h2>Forward and Inverse</h2>
<ul>
<li>Running the simulator forward can be slow, but it is usually possible.</li>
<li>What we actually want is the <strong>inverse process</strong>, the posterior <span class="math inline">p(\boldsymbol{\theta} \mid \mathbf{y}_{\text{obs}})</span>:</li>
</ul>
<img data-src="media/images/stefan_inverse-process.png" alt="SBI forward and inverse process" class="r-stretch"><div class="columns">
<div class="column" style="width:49%;">
<div class="fragment">
<p><strong>Likelihood-based (explicit)</strong></p>
<ul>
<li><span class="math inline">p(\boldsymbol{\theta})</span>: sample and evaluate.</li>
<li><span class="math inline">p(\mathbf{y}\mid\boldsymbol{\theta})</span>: sample and evaluate.</li>
</ul>
<p>→ MCMC, VI (Day 1 & 2).</p>
</div>
</div><div class="column" style="width:49%;">
<div class="fragment">
<p><strong>Simulation-based (implicit)</strong></p>
<ul>
<li><span class="math inline">p(\boldsymbol{\theta})</span>: sample, evaluation <em>optional</em>.</li>
<li><span class="math inline">p(\mathbf{y}\mid\boldsymbol{\theta})</span>: <strong>sample only</strong>, since the likelihood may be intractable.</li>
</ul>
</div>
</div></div>
</section>
<section id="amortized-posterior-estimation" class="slide level2">
<h2>Amortized Posterior Estimation</h2>
<p>Train a conditional density estimator <span class="math inline">q_{\boldsymbol{\phi}}(\boldsymbol{\theta}\mid\mathbf{y})</span> using the <strong>forward KL-divergence</strong>:</p>
<p><span class="math display">
\operatorname{KL}\!\big(p(\boldsymbol{\theta}\mid\mathbf{y})\,\Vert\, q_{\boldsymbol{\phi}}(\boldsymbol{\theta}\mid\mathbf{y})\big)
</span></p>
<div class="fragment">
<ul>
<li>We can take the expectation over the marginal distribution of the data <span class="math inline">\mathbf{y}</span>: <span class="math display">
\begin{align*}
\mathbb{E}_{p(\mathbf{y})}\!\left[
\operatorname{KL}(p(\boldsymbol{\theta} \mid \mathbf{y}) \Vert q_{\phi}(\boldsymbol{\theta} \mid \mathbf{y}))
\right] \notag
&=\mathbb{E}_{p(\mathbf{y})}\mathbb{E}_{p(\boldsymbol{\theta} \mid \mathbf{y})}\!\left[
\log p(\boldsymbol{\theta} \mid \mathbf{y}) - \log q_{\phi}(\boldsymbol{\theta} \mid \mathbf{y})
\right] \notag \\
&=\mathbb{E}_{p(\boldsymbol{\theta},\mathbf{y})}\!\left[
\log p(\boldsymbol{\theta} \mid \mathbf{y})\right]
-\mathbb{E}_{(\boldsymbol{\theta},\mathbf{y})}\!\left[
\log q_{\phi}(\boldsymbol{\theta} \mid \mathbf{y})
\right].
\end{align*}
</span></li>
</ul>
</div>
<div class="fragment">
<ul>
<li>Minimizing allows us to drop the true posterior as it does not depend on <span class="math inline">\boldsymbol{\phi}</span>: <span class="math display">
\boldsymbol{\phi}^* = \arg\min_{\boldsymbol{\phi}}\;
\mathbb{E}_{(\boldsymbol{\theta},\mathbf{y})\sim p(\boldsymbol{\theta},\mathbf{y})}\big[-\log q_{\boldsymbol{\phi}}(\boldsymbol{\theta}\mid\mathbf{y})\big]
</span></li>
</ul>
</div>
<div class="fragment">
<ul>
<li>We train here on pairs of simulation and parameters <span class="math inline">(\boldsymbol{\theta},\mathbf{y})</span>.</li>
<li><strong>Amortization</strong>: at inference, <span class="math inline">\boldsymbol{\theta} \sim q_{\boldsymbol{\phi}^*}(\cdot \mid \mathbf{y}_{\text{obs}})</span> for as many new <span class="math inline">\mathbf{y}_{\text{obs}}</span> as we want without further training.</li>
</ul>
</div>
</section>
<section id="normalizing-flows-as-density-estimators" class="slide level2">
<h2>Normalizing Flows as Density Estimators</h2>
<ul>
<li>How do we represent a flexible <span class="math inline">q_{\boldsymbol{\phi}}(\boldsymbol{\theta}\mid\mathbf{y})</span> that we can both <strong>sample</strong> and <strong>evaluate</strong>?</li>
</ul>
<p>→ With a neural network, a <strong>neural posterior estimator</strong>, but an invertible one!</p>
<div class="fragment">
<ul>
<li>A <strong>normalizing flow</strong> transports a simple base <span class="math inline">p(\mathbf{z})=\mathcal{N}(\mathbf{0},\mathbf{I})</span> through a learned bijection <span class="math inline">f_{\boldsymbol{\phi}}(\cdot;\mathbf{y})</span> <span class="citation" data-cites="rezende2015variational papamakarios2021normalizing">(<a href="#/additional-resources" role="doc-biblioref" onclick="">Papamakarios et al., 2021</a>; <a href="#/additional-resources" role="doc-biblioref" onclick="">Rezende and Mohamed, 2015</a>)</span>: <span class="math display">
\boldsymbol{\theta} = f_{\boldsymbol{\phi}}^{-1}(\mathbf{z};\mathbf{y}),\qquad \mathbf{z}\sim\mathcal{N}(\mathbf{0},\mathbf{I})
</span></li>
</ul>
</div>
<div class="fragment">
<ul>
<li><p><strong>Change of variables</strong> gives a tractable density: <span class="math display">
q_{\boldsymbol{\phi}}(\boldsymbol{\theta}\mid\mathbf{y})
= p\big(f_{\boldsymbol{\phi}}(\boldsymbol{\theta};\mathbf{y})\big)\,
\big|\det J_{f_{\boldsymbol{\phi}}}(\boldsymbol{\theta};\mathbf{y})\big|
</span></p></li>
<li><p>Forward KL means minimize the log-likelihood of the flow.</p></li>
<li><p>The neural network must be <strong>invertible</strong>, and we must be able to compute <span class="math inline">\det J</span>.</p></li>
</ul>
</div>
</section>
<section id="normalizing-flows-on-a-checkerboard" class="slide level2">
<h2>Normalizing Flows on a Checkerboard</h2>
<video src="media/videos/manim_diffusion/1080p60/NFLayers.mp4" data-autoplay="" preload="auto" loop="" muted="" playsinline="" style="max-height:560px;display:block;margin:0 auto;">
</video>
<p>We stack many layers <span class="math inline">f=f^1 \circ \cdots \circ f^L</span> to get a more expressive network.</p>
</section>
<section id="making-flows-cheap-with-coupling-layers" class="slide level2">
<h2>Making Flows Cheap with Coupling Layers</h2>
<div class="fragment">
<p><strong>Coupling flow</strong> <span class="citation" data-cites="dinh2017density">(<a href="#/additional-resources" role="doc-biblioref" onclick="">Dinh et al., 2017</a>)</span>: split <span class="math inline">\boldsymbol{\theta}=(\boldsymbol{\theta}^A,\boldsymbol{\theta}^B)</span>, transform one block conditioned on the other: <span class="math display">
\mathbf{z}^A = f(\boldsymbol{\theta}, \mathbf{y}) = \boldsymbol{\theta}^A \odot \exp\!\big(t_1(\boldsymbol{\theta}^B,\mathbf{y})\big) + t_2(\boldsymbol{\theta}^B,\mathbf{y}),
\qquad \mathbf{z}^B = \boldsymbol{\theta}^B
</span></p>
</div>
<div class="fragment">
<p>Then the inverse follows as: <span class="math display">
\boldsymbol{\theta}^A
=
\left(\mathbf{z}^A-t_2(\mathbf{z}^B,\mathbf{y})\right)
\odot
\exp\!\left(-t_1(\mathbf{z}^B,\mathbf{y})\right),
\quad \boldsymbol{\theta}^B=\mathbf{z}^B.
</span></p>
</div>
<div class="fragment">
<ul>
<li>Computation of log-determinant costs <span class="math inline">\mathcal{O}(d^3)</span> in general, but only <span class="math inline">\mathcal{O}(d)</span> if the Jacobian is <strong>triangular</strong>.</li>
<li>The log-determinant is <span class="math inline">\sum s(\cdot)</span>, and the inverse costs the same as the forward.</li>
<li><span class="math inline">t_1, t_2</span> are <strong>arbitrary neural nets</strong> (they need not be invertible).</li>
<li>Stack many layers <span class="math inline">f=f^1 \circ \cdots \circ f^L</span>, alternating which block is transformed.</li>
</ul>
</div>
<div class="fragment">
<ul>
<li><strong>Spline flows</strong> <span class="citation" data-cites="durkan2019neural">(<a href="#/additional-resources" role="doc-biblioref" onclick="">Durkan et al., 2019</a>)</span> use a more flexible transformation than affine layers.</li>
<li>Stacked coupling blocks are <strong>universal density approximators</strong> for sufficiently regular densities <span class="citation" data-cites="draxler2024universality">(<a href="#/additional-resources" role="doc-biblioref" onclick="">Draxler et al., 2024</a>)</span>.</li>
</ul>
</div>
</section>
<section id="summary-inference-networks" class="slide level2">
<h2>Summary & Inference Networks</h2>
<ul>
<li>Raw data is rarely a fixed-size vector.</li>
<li>A summary network compresses <span class="math inline">\mathbf{y}</span> first, and both networks can be trained <strong>jointly</strong>:</li>
</ul>
<p><span class="math display">
\boldsymbol{\theta}\sim q_{\boldsymbol{\phi}}\big(\boldsymbol{\theta}\mid \underbrace{s(\mathbf{y})}_{\text{summary}}\big)
</span></p>
<ul>
<li><strong>Summary network</strong> <span class="math inline">s</span>: compresses raw data <span class="math inline">\mathbf{y}</span> (set, series, hierarchy) into a fixed-size vector, using the inductive bias that matches the data.</li>
<li><strong>Inference network</strong> <span class="math inline">q_{\boldsymbol{\phi}}</span>: the conditional network over parameters, taking <span class="math inline">s(\mathbf{y})</span> as its condition.</li>
</ul>
<div class="fragment">
<ul>
<li>Ideal case: <span class="math inline">s</span> is a <strong>sufficient statistic</strong> for <span class="math inline">\boldsymbol{\theta}</span>. In practice it is a <em>learned compression</em>, optimized for posterior inference <span class="citation" data-cites="radev2020bayesflow">(<a href="#/additional-resources" role="doc-biblioref" onclick="">Radev et al., 2020</a>)</span>.</li>
</ul>
</div>
</section>
<section id="what-structure-does-the-data-have" class="slide level2">
<h2>What Structure Does the Data Have?</h2>
<p>Different data structures call for different summary networks:</p>
<table class="caption-top">
<colgroup>
<col style="width: 33%">
<col style="width: 33%">
<col style="width: 33%">
</colgroup>
<thead>
<tr class="header">
<th>Data type</th>
<th>Symmetry</th>
<th>Summary network</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td>Exchangeable set</td>
<td>permutation invariance</td>
<td>DeepSet / Set Transformer</td>
</tr>
<tr class="even">
<td>Time series</td>
<td>temporal order</td>
<td>GRU / CNN / Transformer</td>
</tr>
<tr class="odd">
<td>Hierarchical</td>
<td>nested groups</td>
<td>set-of-sets (composed)</td>
</tr>
</tbody>
</table>
</section>
<section id="data-type-1-exchangeable-data" class="slide level2">
<h2>Data Type 1: Exchangeable Data</h2>
<ul>
<li><span class="math inline">N</span> i.i.d. observations; the posterior does <strong>not</strong> depend on ordering. What about <strong>variable <span class="math inline">N</span></strong>?</li>
</ul>
<p><span class="math display">
s(\mathbf{y}_{\pi(1)},\dots,\mathbf{y}_{\pi(N)}) = s(\mathbf{y}_1,\dots,\mathbf{y}_N)\quad\text{for all permutations }\pi
</span></p>
<div class="fragment">
<ul>
<li><strong>DeepSets</strong> <span class="citation" data-cites="zaheer2017deep">(<a href="#/additional-resources" role="doc-biblioref" onclick="">Zaheer et al., 2017</a>)</span> enforce this by a <em>sum-decomposition</em>: <span class="math display">
s(\mathbf{y}_1,\dots,\mathbf{y}_N) = \phi_2\Big(\textstyle\bigoplus_{i=1}^N \phi_1(\mathbf{y}_i)\Big)
</span> embed each element with <span class="math inline">\phi_1</span>, pool with a symmetric aggregator <span class="math inline">\bigoplus</span> (sum, mean, max), decode with <span class="math inline">\phi_2</span>.</li>
</ul>
</div>
<div class="fragment">
<ul>
<li><strong>Set Transformer</strong> <span class="citation" data-cites="lee2019set">(<a href="#/additional-resources" role="doc-biblioref" onclick="">Lee et al., 2019</a>)</span>: attention lets each element’s representation depend on the whole set, which is more flexible but more expensive.</li>
</ul>
</div>
</section>
<section id="data-type-2-time-series" class="slide level2">
<h2>Data Type 2: Time Series</h2>
<p>Ordering now carries information, so the summary network needs a <strong>temporal</strong> inductive bias.</p>
<div class="columns">
<div class="column" style="width:50%;">
<p><strong>Convolutional</strong> summaries share local filters across time: <span class="math display">
s_{t,k} = \sigma\Big(b_k + \sum_{l=-L}^{L}\sum_j w_{l,j,k}\,y_{t+l,j}\Big)
</span> Short-range patterns, position-invariant.</p>
</div><div class="column" style="width:50%;">
<p><strong>Recurrent (GRU/LSTM)</strong> summaries carry a hidden state: <span class="math display">
\mathbf{h}_n = f_{\boldsymbol{\phi}}(\mathbf{y}_n, \mathbf{h}_{n-1})
</span> Long-range dependence via gating.</p>
</div></div>
<div class="fragment">
<p><strong>Transformers</strong> with time or positional embeddings also handle irregularly sampled series.</p>
</div>
</section>
<section id="data-type-3-hierarchical" class="slide level2">
<h2>Data Type 3: Hierarchical</h2>
<p>Data comes in <strong>groups</strong> (subjects, experiments, sites): local variation, shared global structure <span class="citation" data-cites="gelman2013bayesian">(<a href="#/additional-resources" role="doc-biblioref" onclick="">Gelman et al., 2013</a>)</span>.</p>
<div class="columns">
<div class="column" style="width:55%;">
<p><span class="math display">
\boldsymbol{\eta}\sim p(\boldsymbol{\eta}),\quad
\boldsymbol{\theta}^{(r)}\sim p(\boldsymbol{\theta}\mid\boldsymbol{\eta}),\quad
\mathbf{y}^{(r)}\sim p(\mathbf{y}\mid\boldsymbol{\theta}^{(r)})
</span></p>
<p>Two coupled inference targets: <span class="math display">
p(\boldsymbol{\eta}\mid\{\mathbf{y}^{(r)}\}_{r=1}^R),\qquad
p(\boldsymbol{\theta}\mid\mathbf{y}^{(r)},\boldsymbol{\eta})
</span></p>
<p><strong>Summary:</strong> a <em>set of sets</em>. Encode each group, then aggregate across groups with a permutation-invariant network.</p>
</div><div class="column" style="width:45%;">
<p><img data-src="media/images/bayesian-models-2-color%20small.png" alt="Hierarchical Bayesian model"></p>
</div></div>
<div class="fragment">
<p><strong>Bottleneck:</strong> naive amortization needs many simulator calls <em>per group</em>, so the budget grows with <span class="math inline">R</span>. Part 3 comes back to this.</p>
</div>
</section>
<section id="bayesflow-amortized-bayesian-workflows" class="slide level2">
<h2>BayesFlow: Amortized Bayesian Workflows</h2>
<img data-src="media/images/bf_landing_light.png" alt="BayesFlow landing" class="r-stretch"><p>An open-source library for the full amortized Bayesian workflow: <em><a href="https://bayesflow.org" target="_blank">bayesflow.org</a></em></p>
</section>
<section id="the-amortized-workflow" class="slide level2">
<h2>The Amortized Workflow</h2>
<div class="r-stack">
<p><img data-src="media/images/workflow-1.png" alt="Workflow: simulator"></p>
<p><img data-src="media/images/workflow-2.png" class="fragment" data-fragment-index="1" alt="Workflow: adapter"></p>
<p><img data-src="media/images/workflow-3.png" class="fragment" data-fragment-index="2" alt="Workflow: networks"></p>
<p><img data-src="media/images/workflow-4.png" class="fragment" data-fragment-index="3" alt="Workflow: full pipeline with diagnostics"></p>
</div>
<p><strong>Simulator</strong> → <strong>adapter</strong> → <strong>neural approximator</strong> (summary + inference net) → <strong>diagnostics</strong>.</p>
<p><em>Figure: S. Radev, BayesFlow.</em></p>
</section>
<section id="code-a-basic-workflow" class="slide level2">
<h2>Code: A Basic Workflow</h2>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb1"><pre class="sourceCode numberSource python number-lines code-with-copy"><code class="sourceCode python"><span id="cb1-1"><a href=""></a><span class="im">import</span> bayesflow <span class="im">as</span> bf</span>
<span id="cb1-2"><a href=""></a></span>
<span id="cb1-3"><a href=""></a><span class="co"># 1. Simulator: draw (theta, y) pairs. The only thing you must provide.</span></span>
<span id="cb1-4"><a href=""></a><span class="kw">def</span> prior():</span>
<span id="cb1-5"><a href=""></a> ...</span>
<span id="cb1-6"><a href=""></a> <span class="cf">return</span> {<span class="st">"beta"</span>: beta, <span class="st">"sigma"</span>: sigma}</span>
<span id="cb1-7"><a href=""></a></span>
<span id="cb1-8"><a href=""></a><span class="kw">def</span> likelihood_simulator(beta, sigma):</span>
<span id="cb1-9"><a href=""></a> ...</span>
<span id="cb1-10"><a href=""></a> <span class="cf">return</span> {<span class="st">"y"</span>: y}</span>
<span id="cb1-11"><a href=""></a></span>
<span id="cb1-12"><a href=""></a>simulator <span class="op">=</span> bf.make_simulator([prior, likelihood_simulator])</span>
<span id="cb1-13"><a href=""></a></span>
<span id="cb1-14"><a href=""></a><span class="co"># 2. Adapter: name/reshape/constrain variables for the networks</span></span>
<span id="cb1-15"><a href=""></a>adapter <span class="op">=</span> (</span>
<span id="cb1-16"><a href=""></a> bf.Adapter()</span>
<span id="cb1-17"><a href=""></a> .concatenate([<span class="st">"beta"</span>, <span class="st">"sigma"</span>], into<span class="op">=</span><span class="st">"inference_variables"</span>)</span>
<span id="cb1-18"><a href=""></a> .concatenate([<span class="st">"y"</span>], into<span class="op">=</span><span class="st">"inference_conditions"</span>)</span>
<span id="cb1-19"><a href=""></a>)</span>
<span id="cb1-20"><a href=""></a></span>
<span id="cb1-21"><a href=""></a><span class="co"># 3. Networks: inference (a normalizing flow) + optional summary net</span></span>
<span id="cb1-22"><a href=""></a>workflow <span class="op">=</span> bf.BasicWorkflow(</span>
<span id="cb1-23"><a href=""></a> simulator<span class="op">=</span>simulator,</span>
<span id="cb1-24"><a href=""></a> adapter<span class="op">=</span>adapter,</span>
<span id="cb1-25"><a href=""></a> inference_network<span class="op">=</span>bf.networks.CouplingFlow(), <span class="co"># normalizing flow</span></span>
<span id="cb1-26"><a href=""></a> summary_network<span class="op">=</span><span class="va">None</span>, <span class="co"># vector data → none needed</span></span>
<span id="cb1-27"><a href=""></a>)</span></code></pre></div><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></div>
<div class="fragment">
<p><strong>Choosing a summary network:</strong></p>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb2"><pre class="sourceCode numberSource python number-lines code-with-copy"><code class="sourceCode python"><span id="cb2-1"><a href=""></a>summary_network<span class="op">=</span>bf.networks.DeepSet() <span class="co"># exchangeable set</span></span>
<span id="cb2-2"><a href=""></a>summary_network<span class="op">=</span>bf.networks.TimeSeriesTransformer() <span class="co"># time series</span></span></code></pre></div><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></div>
</div>
</section>
<section id="code-train-then-query-any-dataset" class="slide level2">
<h2>Code: Train, then Query Any Dataset</h2>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb3"><pre class="sourceCode numberSource python number-lines code-with-copy"><code class="sourceCode python"><span id="cb3-1"><a href=""></a><span class="co"># 4. Train on simulations (online: simulate fresh batches each step)</span></span>
<span id="cb3-2"><a href=""></a>history <span class="op">=</span> workflow.fit_online(epochs<span class="op">=</span><span class="dv">50</span>, batch_size<span class="op">=</span><span class="dv">128</span>)</span>
<span id="cb3-3"><a href=""></a></span>
<span id="cb3-4"><a href=""></a><span class="co"># 5. Amortized inference: one forward pass per dataset</span></span>
<span id="cb3-5"><a href=""></a>post <span class="op">=</span> workflow.sample(conditions<span class="op">=</span>{<span class="st">"y"</span>: y_obs}, num_samples<span class="op">=</span><span class="dv">1000</span>)</span>
<span id="cb3-6"><a href=""></a><span class="co"># -> post["beta"]: (2000,1) posterior draws</span></span></code></pre></div><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></div>
</section>
<section id="exercise-1" class="slide level2" data-background-color="#fcba00">
<h2>Exercise 1</h2>
<p><strong>From MCMC to Amortized Bayesian Inference</strong></p>
<ul>
<li>Revisit a Bayesian <strong>linear regression</strong> in Pyro.</li>
<li>Hand the <em>same</em> Pyro model to BayesFlow as a <strong>simulator</strong> and train an amortized posterior.</li>
<li>Check that the BayesFlow posterior matches MCMC on that dataset.</li>
<li>Then run inference on 200 further datasets, without retraining.</li>
</ul>
<div class="qr-center">
<p><img data-src="media/images/exercise_01.png" alt="QR code for exercise 1"></p>
<p><strong><a href="https://colab.research.google.com/github/arrjon/BayesFlowTutorial/blob/main/01_Linear_Regression_Exercise.ipynb" target="_blank" data-preview-link="false">▶ Open in Colab 01_Linear_Regression at https://shorturl.at/dx6PM</a></strong></p>
</div>
</section></section>
<section>
<section id="part-2-does-it-work-calibration-misspecification" class="title-slide slide level1 special center" data-background-color="#004e9f">
<h1>Part 2: Does It Work? Calibration & Misspecification</h1>
</section>
<section id="two-distinct-questions" class="slide level2">
<h2>Two Distinct Questions</h2>
<p>We produce an approximation <span class="math inline">q(\boldsymbol{\theta}\mid\mathbf{y})</span> of the true posterior. Before we trust it, we must ask:</p>
<div class="fragment">
<ol type="1">
<li><p><strong>Is the inference faithful?</strong> Does <span class="math inline">q</span> recover the correct posterior <em>under the assumed model</em>?</p>
<p>→ <strong>calibration checks</strong> (SBC, TARP, C2ST).</p></li>
</ol>
</div>
<div class="fragment">
<ol start="2" type="1">
<li><p><strong>Is the model adequate?</strong> Does the model explain the <em>observed</em> data at all?</p>
<p>→ <strong>posterior/prior predictive checks, misspecification detection</strong>.</p></li>
</ol>
</div>
<div class="fragment">
<ul>
<li>A posterior predictive check alone <strong>cannot separate</strong> the two <span class="citation" data-cites="schmitt2024amortized">(<a href="#/additional-resources" role="doc-biblioref" onclick="">Schmitt et al., 2024</a>)</span>.</li>
<li>Amortization makes (1) cheap: re-running inference on thousands of simulated datasets is easy.</li>
</ul>
</div>
</section>
<section id="simulation-based-calibration-sbc" class="slide level2">
<h2>Simulation-Based Calibration (SBC)</h2>
<p>SBC exploits a <strong>self-consistency</strong> property of the Bayesian joint <span class="citation" data-cites="talts2018validating cook2006validation">(<a href="#/additional-resources" role="doc-biblioref" onclick="">Cook et al., 2006</a>; <a href="#/additional-resources" role="doc-biblioref" onclick="">Talts et al., 2018</a>)</span>. Define <span class="math display">
p_{\text{SBC}}(\mathbf{y},\boldsymbol{\theta},\tilde{\boldsymbol{\theta}})
= p(\boldsymbol{\theta})\,p(\mathbf{y}\mid\boldsymbol{\theta})\,q(\tilde{\boldsymbol{\theta}}\mid\mathbf{y})
= p(\mathbf{y})\,p(\boldsymbol{\theta}\mid\mathbf{y})\,q(\tilde{\boldsymbol{\theta}}\mid\mathbf{y})
</span></p>
<div class="columns">
<div class="column" style="width:40%;">
<p>If <span class="math inline">q = p(\boldsymbol{\theta}\mid\mathbf{y})</span>, then <span class="math inline">\boldsymbol{\theta}</span> and <span class="math inline">\tilde{\boldsymbol{\theta}}</span> are <strong>identically distributed</strong> given <span class="math inline">\mathbf{y}</span>.</p>
<div class="fragment">
<p>Testing procedure:</p>
<ul>
<li>many draws <span class="math inline">(\boldsymbol{\theta}^{(r)},\mathbf{y}^{(r)})\sim p(\boldsymbol{\theta},\mathbf{y})</span>,</li>
<li>sample <span class="math inline">\tilde{\boldsymbol{\theta}}\sim q(\cdot\mid\mathbf{y}^{(r)})</span>,</li>
<li>compute the <strong>rank</strong> of the true <span class="math inline">\boldsymbol{\theta}^{(r)}</span> among posterior draws.</li>
</ul>
</div>
</div><div class="column" style="width:60%;">
<div class="fragment">
<p><strong>Calibrated ⇒ ranks are uniform.</strong> <img data-src="media/images/modrak_sbc.png" alt="SBC rank plot"> <em>Image: <a href="https://hyunjimoon.github.io/SBC/articles/rank_visualizations.html" target="_blank">Martin Modrák</a>.</em></p>
</div>
</div></div>
</section>
<section id="empirical-cumulative-distribution-functions-as-a-diagnostic-tool" class="slide level2">
<h2>Empirical Cumulative Distribution Functions as a Diagnostic Tool</h2>
<img data-src="media/images/modrak_ecdf.png" alt="SBC ECDF difference plot" class="r-stretch"><p>Plot the <strong>ECDF of ranks minus uniform</strong>, with simultaneous confidence bands. <em>Image: <a href="https://hyunjimoon.github.io/SBC/articles/rank_visualizations.html" target="_blank">Martin Modrák</a>.</em></p>
</section>
<section id="sbc-in-practice-in-bayesflow" class="slide level2">
<h2>SBC in Practice in BayesFlow</h2>
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb4"><pre class="sourceCode numberSource python number-lines code-with-copy"><code class="sourceCode python"><span id="cb4-1"><a href=""></a><span class="co"># Simulate a fresh validation set the network never trained on</span></span>
<span id="cb4-2"><a href=""></a>val <span class="op">=</span> simulator.sample(<span class="dv">1000</span>)</span>
<span id="cb4-3"><a href=""></a>post <span class="op">=</span> workflow.sample(conditions<span class="op">=</span>val, num_samples<span class="op">=</span><span class="dv">500</span>)</span>
<span id="cb4-4"><a href=""></a></span>
<span id="cb4-5"><a href=""></a><span class="co"># Rank-ECDF calibration, per marginal</span></span>
<span id="cb4-6"><a href=""></a>bf.diagnostics.plots.calibration_ecdf(</span>
<span id="cb4-7"><a href=""></a> estimates<span class="op">=</span>post, targets<span class="op">=</span>val</span>
<span id="cb4-8"><a href=""></a>)</span>
<span id="cb4-9"><a href=""></a><span class="co"># Recovery: posterior mean vs. ground truth</span></span>
<span id="cb4-10"><a href=""></a>bf.diagnostics.plots.recovery(</span>
<span id="cb4-11"><a href=""></a> estimates<span class="op">=</span>post, targets<span class="op">=</span>val</span>
<span id="cb4-12"><a href=""></a>)</span></code></pre></div><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></div>
</section>
<section id="marginal-sbc-is-necessary-not-sufficient" class="slide level2">
<h2>Marginal SBC Is Necessary, Not Sufficient</h2>
<ul>
<li>Marginal rank checks can <strong>pass</strong> while the joint posterior is still wrong <span class="citation" data-cites="modrak2025simulation-based">(<a href="#/additional-resources" role="doc-biblioref" onclick="">Modrák et al., 2025</a>)</span>.</li>
</ul>
<div class="fragment">
<ul>
<li>An estimator that ignores informative parts of the data may still show uniform marginal ranks, because errors can cancel across marginals.</li>
</ul>
</div>
<div class="fragment">
<p>Fix: use <strong>data-dependent</strong> test quantities <span class="math inline">T(\boldsymbol{\theta},\mathbf{y})</span> so discrepancies accumulate instead of cancelling.</p>
<ul>
<li><strong>TARP</strong> <span class="citation" data-cites="lemos2023samplingbased">(<a href="#/additional-resources" role="doc-biblioref" onclick="">Lemos et al., 2023</a>)</span>: the distance to a random reference point, which depends on the data. Matching <em>all</em> references ⇔ <span class="math inline">q=p</span>.</li>
<li><strong>C2ST</strong> <span class="citation" data-cites="lopez-paz2017revisiting yao2023discriminative">(<a href="#/additional-resources" role="doc-biblioref" onclick="">Lopez-Paz and Oquab, 2017</a>; <a href="#/additional-resources" role="doc-biblioref" onclick="">Yao and Domke, 2023</a>)</span>: train a classifier to tell true joint <span class="math inline">p(\boldsymbol{\theta},\mathbf{y})</span> from approximate joint <span class="math inline">q(\boldsymbol{\theta}\mid\mathbf{y})p(\mathbf{y})</span>. Accuracy <span class="math inline">\approx 0.5</span> ⇒ indistinguishable.</li>
</ul>
</div>
<div class="fragment">
<div class="code-copy-outer-scaffold"><div class="sourceCode" id="cb5"><pre class="sourceCode numberSource python number-lines code-with-copy"><code class="sourceCode python"><span id="cb5-1"><a href=""></a><span class="co"># TARP</span></span>
<span id="cb5-2"><a href=""></a>bf.diagnostics.metrics.accuracy_random_points(</span>
<span id="cb5-3"><a href=""></a> estimates<span class="op">=</span>post, targets<span class="op">=</span>val</span>
<span id="cb5-4"><a href=""></a>)</span>
<span id="cb5-5"><a href=""></a><span class="co"># C2ST</span></span>
<span id="cb5-6"><a href=""></a>bf.diagnostics.metrics.classifier_two_sample_test(</span>
<span id="cb5-7"><a href=""></a> estimates<span class="op">=</span>post[<span class="dv">0</span>], <span class="co"># for one dataset</span></span>
<span id="cb5-8"><a href=""></a> targets<span class="op">=</span>reference</span>
<span id="cb5-9"><a href=""></a>)</span>
<span id="cb5-10"><a href=""></a><span class="co"># or concatenate posterior samples and data to test the joint</span></span></code></pre></div><button title="Copy to Clipboard" class="code-copy-button"><i class="bi"></i></button></div>
</div>
</section>
<section id="model-misspecification" class="slide level2">
<h2>Model Misspecification</h2>
<ul>
<li>Calibration assumes the model generated the data. Real simulators are imperfect:</li>
</ul>
<p><span class="math display">
\mathbf{y}_{\text{obs}} \nsim p(\mathbf{y}\mid\boldsymbol{\theta})\ \text{for any}\ \boldsymbol{\theta}
</span></p>
<div class="fragment">
<ul>
<li>The network only ever saw the <strong>prior predictive</strong>.</li>
<li>If <span class="math inline">\mathbf{y}_{\text{obs}}</span> falls outside that region it extrapolates and can return a <strong>confident but wrong</strong> posterior <span class="citation" data-cites="frazier2024the schmitt2023detecting">(<a href="#/additional-resources" role="doc-biblioref" onclick="">Frazier et al., 2024</a>; <a href="#/additional-resources" role="doc-biblioref" onclick="">Schmitt et al., 2023</a>)</span>.</li>
</ul>
</div>
<div class="fragment">
<ul>
<li>Nothing raises an error as the estimator returns a posterior either way.</li>
<li>The mismatch has to be <strong>detected</strong> explicitly.</li>
</ul>
</div>
</section>
<section id="detecting-misspecification" class="slide level2">
<h2>Detecting Misspecification</h2>
<div class="fragment">
<ul>
<li><strong>Posterior predictive check.</strong> Simulate replicated data <span class="math inline">\mathbf{y}^\ast</span> from the fitted posterior; compare a test statistic <span class="math inline">T</span> to <span class="math inline">T(\mathbf{y}_{\text{obs}})</span>: <span class="math display">
p_{\text{ppc}} = \Pr\{T(\mathbf{y}^\ast)\ge T(\mathbf{y}_{\text{obs}})\mid\mathbf{y}_{\text{obs}}\}
</span></li>
<li>Extreme tail probability ⇒ the model cannot reproduce the data.</li>
</ul>
</div>
<div class="fragment">
<ul>
<li><strong>Summary-space distance.</strong> Check the learned summary <span class="math inline">s(\mathbf{y}_{\text{obs}})</span> of the data <span class="citation" data-cites="schmitt2023detecting">(<a href="#/additional-resources" role="doc-biblioref" onclick="">Schmitt et al., 2023</a>)</span>.</li>
<li>An observation whose summary <span class="math inline">s(\mathbf{y}_{\text{obs}})</span> is an outlier in that space is a <strong>sign</strong> for misspecification.</li>
</ul>
</div>
<div class="fragment">
<ul>
<li><strong>Always do both</strong>: calibration on simulations, and predictive adequacy on the real observation.</li>
</ul>
</div>
</section>
<section id="exercise-2" class="slide level2" data-background-color="#fcba00">
<h2>Exercise 2</h2>
<p><strong>Epidemic Time Series & Diagnostics</strong></p>
<ul>
<li>Amortized posterior for a mechanistic <strong>SIR</strong> model (ODE + noisy reporting).</li>
<li>A recurrent (GRU) summary network for the case-count time series.</li>
<li>Run and <strong>interpret the diagnostic</strong> suite: SBC ECDF, recovery, contraction.</li>
<li>Then confront the network with <strong>real COVID-19 data</strong> and judge model adequacy.</li>
</ul>
<div class="qr-center">
<p><img data-src="media/images/exercise_02.png" alt="QR code for exercise 2"></p>
<p><strong><a href="https://colab.research.google.com/github/arrjon/BayesFlowTutorial/blob/main/02_SIR_Diagnostics_Exercise.ipynb" target="_blank" data-preview-link="false">▶ Open in Colab 02_SIR_Diagnostics at https://shorturl.at/7VdI6</a></strong></p>
</div>
</section></section>
<section>
<section id="part-3-diffusion-models-flow-matching-and-consistency-models" class="title-slide slide level1 special center" data-background-color="#004e9f">
<h1>Part 3: Diffusion Models, Flow Matching, and Consistency Models</h1>
</section>
<section id="why-go-beyond-normalizing-flows" class="slide level2">
<h2>Why Go Beyond Normalizing Flows?</h2>
<table class="caption-top">
<thead>
<tr class="header">
<th>Family</th>
<th>Architecture</th>
<th>Sampling</th>
<th>Density</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td>Normalizing Flows</td>
<td><strong>constrained</strong> (invertible)</td>
<td>1 step</td>
<td>fast</td>
</tr>
<tr class="even">
<td>Diffusion Models</td>
<td>free-form</td>
<td>multi-step</td>
<td>slow</td>
</tr>
<tr class="odd">
<td>Flow Matching</td>
<td>free-form</td>
<td>multi-step</td>
<td>slow</td>
</tr>
<tr class="even">
<td>Consistency Models</td>
<td>free-form</td>
<td>few-step</td>
<td>N/A</td>
</tr>
</tbody>
</table>
<div class="fragment">
<ul>
<li>Dropping invertibility frees the architecture.</li>
<li>What we gain is expressivity and the ability to <strong>steer sampling after training</strong>.</li>
<li>We pay with lower sampling speed and, for consistency models, no density evalutation.</li>
</ul>
</div>
</section>
<section id="benchmarking-diffusion-models" class="slide level2">
<h2>Benchmarking Diffusion Models</h2>
<img data-src="media/images/c2st_benchmark_boxplot_best.png" alt="Diffusion model benchmark" class="r-stretch"><p>C2ST across ten benchmark tasks, so lower is better and <span class="math inline">0.5</span> means indistinguishable from the reference posterior. The dashed line is the normalizing flow baseline of Lueckmann et al. <span class="citation" data-cites="arruda2025diffusion">(<a href="#/additional-resources" role="doc-biblioref" onclick="">Arruda et al., 2025</a>)</span>.</p>
</section>
<section id="diffusion-models" class="slide level2">
<h2>Diffusion Models</h2>
<img data-src="media/images/diffusion_model_review.png" alt="Diffusion model overview: forward/backward process, training and inference" class="r-stretch"><p>Forward noising (<span class="math inline">\mathbf{z}_t=\alpha_t\mathbf{z}_0+\sigma_t\epsilon</span>) → train the network on <span class="math inline">\omega_t L_t(\hat{\mathbf{z}},\mathbf{z})</span> → backward via an <strong>ODE/SDE solver</strong> <span class="citation" data-cites="arruda2025diffusion">(<a href="#/additional-resources" role="doc-biblioref" onclick="">Arruda et al., 2025</a>)</span>.</p>
</section>
<section id="diffusion-the-forward-process" class="slide level2">
<h2>Diffusion: The Forward Process</h2>
<p>Start at the target <span class="math inline">\mathbf{z}_0 = \boldsymbol{\theta}</span>. Gradually add noise via an SDE <span class="citation" data-cites="song2021score-based">(<a href="#/additional-resources" role="doc-biblioref" onclick="">Song et al., 2021</a>)</span>: <span class="math display">
\mathrm{d}\mathbf{z}_t = f(t)\,\mathbf{z}_t\,\mathrm{d}t + g(t)\,\mathrm{d}\mathbf{W}_t
</span></p>
<div class="fragment">
<p>The forward transition is <strong>Gaussian in closed form</strong>, so nothing has to be simulated: <span class="math display">
p(\mathbf{z}_t\mid\mathbf{z}_0)=\mathcal{N}(\alpha_t\mathbf{z}_0,\sigma_t^2\mathbf{I})
\quad\Longleftrightarrow\quad
\mathbf{z}_t = \alpha_t\mathbf{z}_0 + \sigma_t\boldsymbol{\epsilon},\ \ \boldsymbol{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I})
</span> We can jump to any noise level <span class="math inline">t</span> directly.</p>
</div>
</section>
<section id="forward-process-on-a-checkerboard" class="slide level2">
<h2>Forward Process on a Checkerboard</h2>
<video src="media/videos/manim_diffusion/1080p60/ForwardNoising.mp4" data-autoplay="" preload="auto" loop="" muted="" playsinline="" style="max-height:560px;display:block;margin:0 auto;">
</video>
<p>A checkerboard target <span class="math inline">\mathbf{z}_0</span> melts into isotropic Gaussian noise as <span class="math inline">t:0\!\to\!1</span>.</p>
</section>
<section id="diffusion-the-reverse-process" class="slide level2">
<h2>Diffusion: The Reverse Process</h2>
<ul>
<li>The forward SDE has a <strong>time-reversal</strong> <span class="citation" data-cites="anderson1982reverse-time">(<a href="#/additional-resources" role="doc-biblioref" onclick="">Anderson, 1982</a>)</span>: <span class="math display">
\mathrm{d}\mathbf{z}_t = \big[f(t)\mathbf{z}_t - g(t)^2\, \nabla_{\mathbf{z}_t}\!\log p_t(\mathbf{z}_t)\big]\,\mathrm{d}t + g(t)\,\mathrm{d}\bar{\mathbf{W}}_t
</span></li>
</ul>
<div class="fragment">
<ul>
<li>The only unknown is the <strong>score</strong> <span class="math inline">\nabla_{\mathbf{z}_t}\log p_t(\mathbf{z}_t)</span>.</li>
<li>If known, we can integrate backwards from noise to a sample.</li>
</ul>
</div>
</section>
<section id="what-the-score-looks-like" class="slide level2">
<h2>What the Score Looks Like</h2>
<video src="media/videos/manim_diffusion/1080p60/ScoreField.mp4" data-autoplay="" preload="auto" muted="" playsinline="" style="max-height:560px;display:block;margin:0 auto;">
</video>
<p>The <em>trained</em> model’s score <span class="math inline">s(\mathbf{z},t)\approx\nabla_{\mathbf{z}}\log p_t(\mathbf{z})</span> points toward the data.</p>
</section>
<section id="reverse-process-stochastic-sde" class="slide level2">
<h2>Reverse Process: Stochastic SDE</h2>
<video src="media/videos/manim_diffusion/1080p60/ReverseSDE.mp4" data-autoplay="" preload="auto" loop="" muted="" playsinline="" style="max-height:560px;display:block;margin:0 auto;">
</video>
<p>The <strong>reverse SDE</strong> injects noise at every step.</p>
</section>
<section id="reverse-process-stochastic-sde-close-up" class="slide level2">
<h2>Reverse Process: Stochastic SDE Close-Up</h2>
<video src="media/videos/manim_diffusion/1080p60/ReverseSDEWindow.mp4" data-autoplay="" preload="auto" loop="" muted="" playsinline="" style="max-height:560px;display:block;margin:0 auto;">
</video>
<p>The same trajectory, zoomed into the last time steps.</p>
</section>
<section id="denoising-score-matching" class="slide level2">
<h2>Denoising Score Matching</h2>
<ul>
<li><p>Regress a network <span class="math inline">\hat{s}</span> onto the score of the (Gaussian) noising kernel: <span class="math display">
\hat{s} = \arg\min_{s}\;
\mathbb{E}_{\mathbf{z}_0,\mathbf{y},\boldsymbol{\epsilon},t}\Big[\omega_t\,\big\Vert s(\mathbf{z}_t,\mathbf{y},t) - \nabla_{\mathbf{z}_t}\log p(\mathbf{z}_t\mid\mathbf{z}_0)\big\Vert_2^2\Big]
</span></p></li>
<li><p>This has the <strong>same minimizer</strong> as regression on the marginal <span class="math inline">\nabla_{\mathbf{z}_t}\log p(\mathbf{z}_t)</span> <span class="citation" data-cites="vincent2011connection song2021score-based">(<a href="#/additional-resources" role="doc-biblioref" onclick="">Song et al., 2021</a>; <a href="#/additional-resources" role="doc-biblioref" onclick="">Vincent, 2011</a>)</span>.</p></li>
<li><p>The neural network can be <strong>any architecture</strong> which predicts a score, e.g., a MLP <span class="citation" data-cites="sharrock2024sequential">(<a href="#/additional-resources" role="doc-biblioref" onclick="">Sharrock et al., 2024</a>)</span> or a transformer <span class="citation" data-cites="gloeckler2024all-in-one">(<a href="#/additional-resources" role="doc-biblioref" onclick="">Gloeckler et al., 2024</a>)</span>.</p></li>
</ul>
<div class="fragment">
<ul>
<li><p>Because <span class="math inline">\mathbf{z}_t = \alpha_t\mathbf{z}_0 + \sigma_t\boldsymbol{\epsilon}</span> is Gaussian, the conditional target is <strong>known in closed form</strong>: <span class="math display">
\nabla_{\mathbf{z}_t}\log p(\mathbf{z}_t\mid\mathbf{z}_0) = -\boldsymbol{\epsilon}/\sigma_t
</span></p></li>
<li><p>Training almost reduces to <strong>noise prediction</strong> <span class="math inline">\boldsymbol{\epsilon}</span>.</p></li>
</ul>
</div>
<div class="fragment">
<ul>
<li>Predicting <span class="math inline">\mathbf{z}_0</span>, <span class="math inline">\boldsymbol{\epsilon}</span>, the velocity, or the score are equivalent parameterizations (<strong>design choices</strong>) of the same object.</li>
</ul>
</div>
</section>
<section id="from-sde-to-ode-flow-matching" class="slide level2">
<h2>From SDE to ODE: Flow Matching</h2>
<ul>
<li>The reverse SDE has a deterministic twin, the <strong>probability-flow ODE</strong>, with the same marginals: <span class="math display">
\mathrm{d}\mathbf{z}_t = \Big[f(t)\mathbf{z}_t - \tfrac{1}{2}g(t)^2\, s(\mathbf{z}_t,\mathbf{y},t)\Big]\mathrm{d}t = v(\mathbf{z}_t,\mathbf{y},t)\,\mathrm{d}t
</span></li>
</ul>
<div class="fragment">
<ul>
<li><p><strong>Flow matching</strong> <span class="citation" data-cites="lipman2023flow wildberger2023flow">(<a href="#/additional-resources" role="doc-biblioref" onclick="">Lipman et al., 2023</a>; <a href="#/additional-resources" role="doc-biblioref" onclick="">Wildberger et al., 2023</a>)</span> skips the score and regresses the <strong>velocity field</strong> <span class="math inline">v</span> directly, along a chosen interpolation between noise and data: <span class="math display">
\hat{v} = \arg\min_v\; \mathbb{E}_{t,\mathbf{z}_0,\boldsymbol{\epsilon}}\big[\Vert v(\mathbf{z}_t,\mathbf{y},t) - (\dot\alpha_t\mathbf{z}_0 + \dot\sigma_t\boldsymbol{\epsilon})\Vert_2^2\big]
</span></p></li>
<li><p>Straighter paths, so usually fewer integration steps.</p></li>
<li><p><strong>Density evaluation</strong> is available by integrating the Jacobian trace along the ODE, but it is expensive.</p></li>
</ul>
</div>
</section>
<section id="reverse-process-deterministic-ode" class="slide level2">
<h2>Reverse Process: Deterministic ODE</h2>
<video src="media/videos/manim_diffusion/1080p60/ReverseODE.mp4" data-autoplay="" preload="auto" loop="" muted="" playsinline="" style="max-height:560px;display:block;margin:0 auto;">
</video>
<p>The <strong>probability-flow ODE</strong> has the same marginals as the SDE, but a deterministic path.</p>
</section>
<section id="consistency-models" class="slide level2">
<h2>Consistency Models</h2>
<ul>
<li>Both the SDE and the ODE need many solver steps, which makes sampling slow.</li>
<li><strong>Consistency models</strong> <span class="citation" data-cites="song2023consistency schmitt2025consistency">(<a href="#/additional-resources" role="doc-biblioref" onclick="">Schmitt et al., 2025</a>; <a href="#/additional-resources" role="doc-biblioref" onclick="">Song et al., 2023</a>)</span> learn a map that is <strong>invariant along a trajectory</strong>.</li>
<li>It can jump straight to the endpoint: <span class="math display">
c(\mathbf{z}_t,\mathbf{y},t) = c(\mathbf{z}_{t'},\mathbf{y},t'),\qquad c(\mathbf{z}_0,\mathbf{y},0)=\mathbf{z}_0
</span></li>
</ul>
<div class="fragment">
<ul>
<li>Parameterize <span class="math inline">c(\mathbf{z}_t,\mathbf{y},t) = c_{\text{skip}}(t)\,\mathbf{z}_t + c_{\text{out}}(t)\,F(\mathbf{z}_t,\mathbf{y},t)</span> with <span class="math inline">c_{\text{skip}}(0)=1,\ c_{\text{out}}(0)=0</span>, and train outputs to <strong>agree at nearby times</strong>: <span class="math display">
\min_c\ \mathbb{E}\big[\omega_t\, d\big(c(\mathbf{z}_t,\mathbf{y},t),\ \bar c(\mathbf{z}_{t-\Delta t},\mathbf{y},t-\Delta t)\big)\big]
</span> → <strong>one- or few-step</strong> sampling: jump to the endpoint, re-noise to a lower level, and refine.</li>
</ul>
</div>
</section>
<section id="consistency-sampling-few-discrete-steps" class="slide level2">
<h2>Consistency Sampling: Few Discrete Steps</h2>
<video src="media/videos/manim_diffusion/1080p60/ConsistencyMultistep.mp4" data-autoplay="" preload="auto" loop="" muted="" playsinline="" style="max-height:560px;display:block;margin:0 auto;">
</video>
<p>Each step <strong>jumps straight to data</strong>, then <strong>re-noises</strong> to a lower level.</p>
</section>
<section id="the-diffusion-model-family" class="slide level2">
<h2>The Diffusion Model Family</h2>
<img data-src="media/images/fm_cm_visual.png" alt="Diffusion, flow matching, and consistency models" class="r-stretch"><p>Diffusion model (stochastic) · flow matching (deterministic ODE) · consistency model (direct jump).</p>
</section>
<section id="post-hoc-guidance" class="slide level2">
<h2>Post-Hoc Guidance</h2>
<p>The score is <strong>additive</strong>, so sampling can be steered after training by adding a gradient term.</p>
<img data-src="media/images/adaptive_inference.png" alt="Post-hoc guidance" class="r-stretch"><p><span class="math display">
\nabla_{\boldsymbol{\theta}_t}\log p(\boldsymbol{\theta}_t\mid\text{extra}) \approx \hat{s}(\boldsymbol{\theta}_t,\mathbf{y},t) + \nabla_{\boldsymbol{\theta}_t}\log g(\boldsymbol{\theta}_t)
</span></p>
<p>→ Impose constraints, change the prior, or compose models without retraining <span class="citation" data-cites="bansal2023universal yang2026priorguide">(<a href="#/additional-resources" role="doc-biblioref" onclick="">Bansal et al., 2023</a>; <a href="#/additional-resources" role="doc-biblioref" onclick="">Yang et al., 2026</a>)</span>.</p>
</section>
<section id="careful-guidance-changes-the-target" class="slide level2">