Skip to content

fix(mimic-iv): retain device-only rows in oxygen_delivery - #2071

Open
acnimma wants to merge 1 commit into
MIT-LCP:mainfrom
acnimma:fix/mimic-iv-oxygen-delivery-device-only
Open

fix(mimic-iv): retain device-only rows in oxygen_delivery#2071
acnimma wants to merge 1 commit into
MIT-LCP:mainfrom
acnimma:fix/mimic-iv-oxygen-delivery-device-only

Conversation

@acnimma

@acnimma acnimma commented Jul 19, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Filter O2 flow rn in ce_stg3 before the FULL OUTER JOIN in oxygen_delivery.sql
  • Retain device-only charttimes (itemid 226732) that previously disappeared when WHERE ce.rn = 1 ran after the join
  • Regenerate postgres and duckdb dialect copies from the BigQuery source

Test plan

  • pytest tests/test_transpile.py -k oxygen_delivery passes locally
  • CI concept build on MIMIC-IV demo

oxygen_delivery FULL OUTER JOINs O2 flow (ce_stg2) with delivery devices (o2, itemid 226732), then applied WHERE ce.rn = 1. That filter drops rows where ce is NULL, so stays with device charting but no flow (e.g. stay_id = 30813912 in #1570) never appear in the derived table. Move the rn = 1 filter into ce_stg3 before the join so device-only rows are kept. Regenerate postgres and duckdb dialect copies from the BigQuery source.

Fixes #1570

The FULL OUTER JOIN of O2 flow and delivery devices filtered with
WHERE ce.rn = 1, which dropped rows where only a device (itemid 226732)
was charted. Filter rn in ce_stg3 before the join so device-only rows
are retained. Regenerate postgres and duckdb dialect copies.

Fixes MIT-LCP#1570

@Chessing234 Chessing234 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the post-join where ce.rn = 1 was definitely nuking device-only rows from the full outer join — moving the rn filter into ce_stg3 first is the right shape. coalesce on subject/stay/charttime was already there so device-only keys should survive. if you haven't already, worth a quick count of rows where o2_flow is null but o2_delivery_device_1 isn't, before vs after.

@Chessing234

Copy link
Copy Markdown
Contributor

keeping device-only oxygen rows is useful for vent studies. does the regenerated postgres/duckdb copy land in this pr too?

@Chessing234 Chessing234 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

filtering rn before the full outer join is the right fix — post-join where ce.rn = 1 was dropping device-only o2 rows when ce was null. mirrored across dialects; the comment in the bq file helps. a tiny regression query in concepts/documentation (device-only count before/after) would make the behavioral change obvious to reviewers.

@acnimma

acnimma commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

dialects included; skip regression query (nice-to-have, not required thank you

@Chessing234 Chessing234 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

following up on the dialect question above — the postgres copy in this pr isn't what the transpiler produces.

i ran the repo's own transpiler (sqlglot 30.11.0 from requirements-lock) over your bigquery source and diffed it against mimic-iv/concepts_postgres/measurement/oxygen_delivery.sql as committed here. the o2 comment block comes out of sqlglot as a single line, but this pr has it re-wrapped across 12 separate /* ... */ lines. the duckdb copy is byte-identical, so it looks like just the postgres file was touched by hand. running the same check against main gives IDENTICAL for this file, so the drift starts in this branch.

that file's header says it's automatically generated and not to be edited directly, and the generated-up-to-date job in transpile.yml diffs it against transpiler output — so this will come back as a failure once the checks actually run (the current ones were cancelled waiting for runners, not failed).

could you regenerate the postgres copy instead of editing it? the ce_stg3 logic itself looks right to me — moving rn = 1 ahead of the full outer join does retain the device-only rows, and the COALESCE on subject_id/charttime in stg means those rows keep their grouping keys.

one thing i'm not sure about and would want your read on: with device-only charttimes now retained, those rows land in the output with o2_flow null. worth confirming that's the intended shape for #1570 rather than something downstream consumers would need to filter.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Question about the number of patients in the oxygen_delivery table

2 participants