fix(mimic-iv): retain device-only rows in oxygen_delivery - #2071
Conversation
The FULL OUTER JOIN of O2 flow and delivery devices filtered with WHERE ce.rn = 1, which dropped rows where only a device (itemid 226732) was charted. Filter rn in ce_stg3 before the join so device-only rows are retained. Regenerate postgres and duckdb dialect copies. Fixes MIT-LCP#1570
Chessing234
left a comment
There was a problem hiding this comment.
the post-join where ce.rn = 1 was definitely nuking device-only rows from the full outer join — moving the rn filter into ce_stg3 first is the right shape. coalesce on subject/stay/charttime was already there so device-only keys should survive. if you haven't already, worth a quick count of rows where o2_flow is null but o2_delivery_device_1 isn't, before vs after.
|
keeping device-only oxygen rows is useful for vent studies. does the regenerated postgres/duckdb copy land in this pr too? |
Chessing234
left a comment
There was a problem hiding this comment.
filtering rn before the full outer join is the right fix — post-join where ce.rn = 1 was dropping device-only o2 rows when ce was null. mirrored across dialects; the comment in the bq file helps. a tiny regression query in concepts/documentation (device-only count before/after) would make the behavioral change obvious to reviewers.
|
dialects included; skip regression query (nice-to-have, not required thank you |
Chessing234
left a comment
There was a problem hiding this comment.
following up on the dialect question above — the postgres copy in this pr isn't what the transpiler produces.
i ran the repo's own transpiler (sqlglot 30.11.0 from requirements-lock) over your bigquery source and diffed it against mimic-iv/concepts_postgres/measurement/oxygen_delivery.sql as committed here. the o2 comment block comes out of sqlglot as a single line, but this pr has it re-wrapped across 12 separate /* ... */ lines. the duckdb copy is byte-identical, so it looks like just the postgres file was touched by hand. running the same check against main gives IDENTICAL for this file, so the drift starts in this branch.
that file's header says it's automatically generated and not to be edited directly, and the generated-up-to-date job in transpile.yml diffs it against transpiler output — so this will come back as a failure once the checks actually run (the current ones were cancelled waiting for runners, not failed).
could you regenerate the postgres copy instead of editing it? the ce_stg3 logic itself looks right to me — moving rn = 1 ahead of the full outer join does retain the device-only rows, and the COALESCE on subject_id/charttime in stg means those rows keep their grouping keys.
one thing i'm not sure about and would want your read on: with device-only charttimes now retained, those rows land in the output with o2_flow null. worth confirming that's the intended shape for #1570 rather than something downstream consumers would need to filter.
Summary
rnince_stg3before the FULL OUTER JOIN inoxygen_delivery.sqlWHERE ce.rn = 1ran after the joinTest plan
pytest tests/test_transpile.py -k oxygen_deliverypasses locallyoxygen_deliveryFULL OUTER JOINs O2 flow (ce_stg2) with delivery devices (o2, itemid 226732), then appliedWHERE ce.rn = 1. That filter drops rows whereceis NULL, so stays with device charting but no flow (e.g.stay_id = 30813912in #1570) never appear in the derived table. Move thern = 1filter intoce_stg3before the join so device-only rows are kept. Regenerate postgres and duckdb dialect copies from the BigQuery source.Fixes #1570