Placebo tests and extended plotting functionality by n-delaney · Pull Request #88 · ebenmichael/augsynth

n-delaney · 2023-08-25T01:27:05Z

The proposed changes:

Extend the functionality of the augsynth package by including the options for permuted placebo tests (raw and RMPSE-adjusted)
Move inference out of summary.augsynth() and into the augsynth() function by default
Extend plotting options, including placebo permutation plots and gap plots showing the divergence between the treated units and synthetic control over time

All of the changes to the augsynth package itself take place in augsynth.R and a new file, permutation.R.

The tobacco_replication directory contains a vignette demonstrating the new functionality and replicating the results of Abadie et al. (2010).

Rmd file and data for replication of Abadie et al. (2010)

-updates functions in augsynth.R -adds permutation.R (also contains some plotting functions) -updates package documentation

Updates the github directory in tobacco replication vignette

Replaces convention of inf = F with inf_type = 'none'.

- Separates out inference from estimation. Sets augsynth(inf = "none") by default, which should be backwards compatible. - Adds plot_type as an argument for plot.augsynth (allows user to specify the "name" of the desired outputted plot). - Adds the option to generate a donor table (synthetic control weights and RMSPEs for any inference type) - Ensures that all existing tests execute successfully.

Updates to abadie et al 2010 replication notebook

Adds warnings to multi-outcome and multisynth models if a user attempts to pass in `inf_type` at the time of model fitting.

Minor edits to tobacco replication notebook

…synth class methods for accessing data characteristics.

- converts the tobacco replication notebook to a vignette - removes cache folders and plot folder from the Abadie et al. (2010) tobacco replication notebook

…ation

…ome of the MDES calculation stuff from the permutation package to clarify inference side.

- addresses number formatting in print.summary.augsynth

…ation

…verage.

…tribution

ebenmichael

Overall I think there needs to be a structural change to keep inference calculations in summary.augsynth. The new add_inference function seems to be necessary only to add inference output to the augsynth function, but that should be restricted to the summary.augsynth function. I've called out parts of the code where this can be changed, but mostly it means that all of the new plotting work should happen in the plot.summary.augsynth function, and then the augsynth object never needs any inference added to it.

ebenmichael · 2024-06-13T15:01:39Z

+
+
+
+test_that( "MDES_table corresponds to default treatment table", {


Can this test be broken up into subtests, since it looks like there are a few distinct things.

Another good test would be that the placebo gaps agree with manually changing the treatment variable in the data and re-running augsynth.

… into tobacco example more.

…utation into pr_merge

- adds a test for update_augsynth() ensuring that non-binding update criteria (ex. RMPSE multiple of infinity) return the same donor weights as the original model. - removes superfluous datasets froom the tobacco_replication.Rmd file - updates a test in test-donor_control.R to resolve an error from a variable that is called but not defined (or used as part of the test, as far as I can tell)

…to unite checking of that setting.

ebenmichael

Thanks for all of these changes. I left reviews throughout. Once those are done (and the merge conflict is fixed) we can fold this in.

ebenmichael · 2025-01-29T19:27:48Z

+
+    # These should be the same?  Or close to the same?
+    # (They were not under ridge, probably due to the cross-validation procedure.)
+    gaps7 <- augsynth:::get_placebo_gaps(syn7)


Can you test this with ridge with a fixed lambda parameter? If they should be same then the test should pass.

@n-delaney @lmiratrix the question marks here make me nervous

lmiratrix · 2026-01-23T19:29:12Z

Trying to put in my comments

…utation into pr_merge

ebenmichael

An overarching request: Please make sure that the top-level calls to the summary and plot functions don't change their default behavior and that the exisiting vignette still works as it should (it doesn't give the same results as-is). If the vignette needs to be updated (e.g. to incorporate the new plot_type argument, please do that).

ebenmichael · 2026-03-02T17:51:13Z

+plot.augsynth <- function(augsynth,
+                          cv = FALSE,
+                          plot_type = 'estimate',
+                          inf_type = NULL, ...) {
+
+    plot_augsynth_results( augsynth=augsynth, cv=cv, plot_type=plot_type, inf_type=inf_type, ... )


My understanding is that the cv option no longer does anything, and instead that is subsumed into plot_type correct? Can you update that throughout the code so that it isn't passing in cv if that isn't doing anything?

I went back through an ensured that the existing plot() calls will generate the same plots as previously. The vignette runs and delivers the expected plots without any changes the the code blocks.

That being said, the vignette doesn't currently showcase the plot_type argument (or specify the four plot types — "estimate", "outcomes", "cv", and "permutation") that you can get out of it. I will leave it up to you if and how you want to showcase these features.

One thing to note—in order to make plot.augsynth() and plot.summary.augsynth() backwards compatible, they will return a cross-validation plot if the user sets cv=TRUE regardless of plot_type setting.

ebenmichael · 2026-03-02T17:54:55Z

+#' @param ... Optional arguments
+plot_augsynth_results <- function( augsynth,
+                                   plot_type = 'estimate',
+                                   inf_type = NULL, ...) {


The default behavior of the package is to do conformal inference. Can you make whatever changes are needed so that this continues to be the case? Right now if you follow the vignette it doesn't produce the prediction intervals.

Also, the plots used to have the inf argument that governed whether uncertainty intervals were included. That should stay in to not break any existing code. It also simplifies the plot_type argument because then there doesn't need to be the 'estimate only' option.

I have reincorporated inf, which now controls whether confidence intervals appear using plot_type = "estimate". I have also stripped down the plot types to four distinct arguments (i.e. I removed a differentiation between plot_type = "estimate" (with CI) and plot_type = "estimate only" (without CI), since this is now covered by inf.

ebenmichael · 2026-03-02T18:01:26Z

+                                aes(x = !!as.name(augsynth$time_var), y = !!as.name(measure),
+                                    color = trt_status, linetype = !!as.name(augsynth$unit_var))) +
+        ggplot2::geom_line() +
+        ggplot2::geom_vline(lty = 2, xintercept = treat_year) +


For some reason the vertical line doesn't appear when you get here by running plot rather than permutation_plot.

e.g. the following creates two slightly different plots

syn <- augsynth(lngdpcapita ~ treated, fips, year_qtr, kansas, progfunc = "None", scm = T) # no vertical line at treatment time plot(syn, plot_type = "placebo") # yes vertical line at treatment time permutation_plot(syn)

Fixed! This was an issue with plotting from a summary or augsynth object

ebenmichael · 2026-03-02T18:07:21Z

+    } else if (plot_type == 'outcomes') {
+        p <- augsynth_outcomes_plot(augsynth, measure = 'synth')
+    } else if (plot_type == 'outcomes raw average') {
+        p <- augsynth_outcomes_plot(augsynth, measure = c('synth', 'average'))


These should have the option to add the uncertainty intervals (i.e. with inf = TRUE following the original logic of the code)

Done! Please see comment above on a related note

ebenmichael · 2026-03-02T18:09:57Z

+            p <- p + ggplot2::geom_ribbon(ggplot2::aes(ymin = Estimate - 2 * Std.Error,
+                                                       ymax = Estimate + 2 * Std.Error),
+                                          alpha = 0.2)


Something is happening here where if you just run plot(syn) for a augsynth object syn, the default behavior doesn't include uncertainty quantification but this line of geom_ribbon is running with a warning about all missing inputs. This might go away if the default behavior is to plot conformal inference intervals, but still it comes up using plot(syn, inf_type = NULL)

I noticed that the geom_ribbon() warning was appearing on both the main and pr_merge branches. I think it is coming from "NA" values for confidence intervals in pre-treatment periods. I have resolved this in the pr_merge branch.

ebenmichael · 2026-03-02T18:25:43Z

+
+#' Calculate MDES
+#'
+#' Use our method for Calculating SEs, p-values, and MDEs by looking


Ok can you just strip this out here then? Alternatively, you can out in a citation etc. Something more than "our" method, since I'd like for it to be clear what different functions are doing.

ebenmichael · 2026-03-02T18:30:35Z

+    units$weights[ units$ever_Tx == 1 ] = 0
+
+    # confirm that we have placebos for every entity over every observed time period
+    stopifnot("Placebos do not cover every entity over every observed time period." = (ncol(augsynth$data$X) + ncol(augsynth$data$y)) * nrow(augsynth$data$y) == nrow(lest))


@lmiratrix @n-delaney does this ever happen or were you adding in a hypothetical check?

ebenmichael · 2026-03-02T18:32:24Z

+
+
+    # Impact is difference between observed and imputed control-side outcome
+    lest$impact = lest$Yobs - lest$Yhat


Please remove it if it isn't needed

ebenmichael · 2026-03-02T18:34:03Z

+
+    # These should be the same?  Or close to the same?
+    # (They were not under ridge, probably due to the cross-validation procedure.)
+    gaps7 <- augsynth:::get_placebo_gaps(syn7)


@n-delaney @lmiratrix the question marks here make me nervous

- Simplify plot.augsynth() / plot.summary.augsynth() behavior around inference and plot types (should ensure backwards compatibility) - Remove plot_type options: * "estimate only" → use inf = FALSE to suppress CIs when using `plot_type` = "estimate" (ATT) * "outcomes raw average" → outcomes plot now always includes treated, synthetic, and donor average - Make cv = TRUE override other plotting options (backwards compatibility) - Preserve inf as the switch controlling CI display (even when inf_type is supplied) - Update summary plotting: * summary.augsynth plots inherit stored inference and do not recompute at plot time * placebo plots require permutation-based inference for summary objects * plot.augsynth() defaults to permutation inference for placebo plots unless inf_type = "permutation_rstat" is explicitly requested - Update permutation_plot() to inherit inference type from object - Fix CV plot warnings (geom_point annotations) - Update (revert) vignette to reflect new plotting behavior - Update plotting tests to match new API Notes: - plot(desyn) shows slightly different confidence intervals relative to vignette from main branch; possibly due to inference changes, not plotting? - Some tests (test_multisynth_covariates.R, test_outcome_models.R) currently fail due to numerical tolerance; temporarily disabled with skip("temporarily disabled")

Merge branch 'pr_merge' of https://github.com/n-delaney/augsynth_permutation into pr_merge # Conflicts: # tobacco_replication/tobacco_exploration.R

…utation into pr_merge

n-delaney added 10 commits August 23, 2023 17:18

add tobacco_replication files

bb79c20

Rmd file and data for replication of Abadie et al. (2010)

Update package files

ff13d1c

-updates functions in augsynth.R -adds permutation.R (also contains some plotting functions) -updates package documentation

update text in Rmd file

1a3ebd0

Updates the github directory in tobacco replication vignette

default inference

a1b858d

Replaces convention of inf = F with inf_type = 'none'.

update tobacco replication vignette

d5b21d7

add testing, address join warning

b6d0959

replication edits

ec8d52d

Updates to abadie et al 2010 replication notebook

update warnings

4ade107

Adds warnings to multi-outcome and multisynth models if a user attempts to pass in `inf_type` at the time of model fitting.

Merge branch 'master' into master

9d4af54

n-delaney closed this Feb 14, 2024

n-delaney reopened this Feb 14, 2024

n-delaney and others added 14 commits February 14, 2024 12:10

notebook updates

15c368f

Minor edits to tobacco replication notebook

Edit pass on full package with focus on summary method and making aug…

451d0b7

…synth class methods for accessing data characteristics.

tobacco vignette

b010e16

- converts the tobacco replication notebook to a vignette - removes cache folders and plot folder from the Abadie et al. (2010) tobacco replication notebook

Update .gitignore

4032043

clean up docs and whatnot

a6d447a

Merge branch 'master' of https://github.com/n-delaney/augsynth_permut…

9c534ca

…ation

Cleaned up documentation and added a few utility functions. Removed s…

e495521

…ome of the MDES calculation stuff from the permutation package to clarify inference side.

Merge branch 'master' into pr_merge

2da0e41

patch

2e57f4b

- addresses number formatting in print.summary.augsynth

cleaning up printout. Adding a small DGP function for testing.

d400901

Merge branch 'master' of https://github.com/n-delaney/augsynth_permut…

fac62f4

…ation

Cleaning up documentation and adding some testing code to increase co…

6505c82

…verage.

made plotting options work with both summary and augsynth objects

190d1f5

Clean up tobacco replication, added a helper function for placebo dis…

f7be966

…tribution

ebenmichael requested changes Jun 13, 2024

View reviewed changes

n-delaney added 3 commits August 1, 2024 11:05

plotting from summary

a17d8d8

update tests

efd7305

update plotting tests

eaca494

lmiratrix and others added 11 commits October 16, 2024 08:55

clean up a few small bugs on arguments with SCM=FALSE. Started to dig…

37b9cb2

… into tobacco example more.

Merge branch 'pr_merge' of https://github.com/n-delaney/augsynth_perm…

44a0de5

…utation into pr_merge

Merged in results, tried to repair some test cases (one test is failing)

d2f0c38

Final small cleanup to package based on eli comments.

ecadfe2

Cleaning up tobacco replication script and blog post on inference

76f6b72

update tests

d8e4a45

reintegrate augsynth_class

aed0742

update docs

6e35b6f

cleaning up the vignettes and initial blog post

f62134d

Added test of T0 = n warning, and made progfunc to lowercase earlier …

9d2157a

…to unite checking of that setting.

ebenmichael requested changes Jan 29, 2025

View reviewed changes

n-delaney and others added 4 commits January 21, 2026 16:39

update per pull request

c2e8941

responses to pull request

8ee3f98

remove tobacco replication

5eb991c

tweaks

91f4455

lmiratrix added 3 commits January 23, 2026 14:32

Merge branch 'master' into pr_merge

a419d39

redid documentation, mild auto updates

47e7e5b

Merge branch 'pr_merge' of https://github.com/n-delaney/augsynth_perm…

5187a2e

…utation into pr_merge

ebenmichael reviewed Mar 2, 2026

View reviewed changes

n-delaney requested a review from ebenmichael April 15, 2026 04:44

n-delaney and others added 8 commits April 18, 2026 01:26

further unit testing

fc48706

Merge branch 'pr_merge' of https://github.com/n-delaney/augsynth_permutation into pr_merge # Conflicts: # tobacco_replication/tobacco_exploration.R

Merge branch 'pr_merge' of https://github.com/n-delaney/augsynth_perm…

a17e34a

…utation into pr_merge

Improving plot tests and function

49bda8c

cleanup and hiding of MDES stuff

222f691

removed robust SE code

7002254

add test context

7bab6a1

outcomes plot styling

d21b8f5




		test_that( "MDES_table corresponds to default treatment table", {



		# Impact is difference between observed and imputed control-side outcome
		lest$impact = lest$Yobs - lest$Yhat

Conversation

n-delaney commented Aug 25, 2023

Uh oh!

ebenmichael left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ebenmichael left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lmiratrix commented Jan 23, 2026

Uh oh!

ebenmichael left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

n-delaney Apr 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

n-delaney Apr 20, 2026 •

edited

Loading