Skip to content

perf(postgres): avoid repeated parameter rescans - #1431

Open
Gordex2014 wants to merge 1 commit into
mainfrom
bugfix/1416-repeated-date-rescans
Open

Gordex2014 wants to merge 1 commit into
mainfrom
bugfix/1416-repeated-date-rescans

Conversation

@Gordex2014

@Gordex2014 Gordex2014 commented Sep 21, 2026

Copy link
Copy Markdown
Contributor

Closes #1416.

Summary

  • Fold eligible repeated occurrences of the same PostgreSQL search parameter into one resource-id membership test whose arms use INTERSECT.
  • Preserve FHIR AND semantics between occurrences, OR semantics within an occurrence, positional bind order, tenant scoping, and the previous fallback for ineligible shapes.
  • Add PostgreSQL integration coverage, a repeatable benchmark note, and full-corpus plan capture queries.

Why this fixes the reported plan

The previous SQL emitted one id IN (SELECT ...) condition per occurrence. PostgreSQL could plan the pair as nested semi-joins and rerun one date scan once per candidate from the other arm.

The new SQL intersects the occurrences' resource-id sets inside one membership test. Different indexed values on a multi-valued date can still satisfy different occurrences, so this preserves resource-level semantics without merging predicates onto one index row.

Evidence

The deterministic PostgreSQL 16 benchmark reproduced the nested-loop rescan at two sizes. On the 24k-Patient scale, the page changed from 33.4 s and 7.42 million buffers to 21.3 ms and 15,360 buffers. The count changed from 33.8 s to 18.6 ms. Every INTERSECT arm executed once, returned pages and counts matched the seed-derived expected results, and no capture spilled at 10 MB or 64 kB work_mem.

When the corpus doubled, page buffers grew by 3.915x before and 1.999x after. The committed benchmark document records the method, controls, raw figures, and limitations. The CI plan script captures both the emitted shape and the previous two-membership control on the full benchmark database.

The original manual-QA corpus from #1416 was not rerun. The deterministic corpus reproduces the same rescanning mechanism and verifies its removal, but it does not replace a final confirmation of the reported 1,268-Patient result on that original data.

Compatibility

  • Repeated occurrences remain ANDed; comma-separated values within one occurrence remain ORed.
  • A multi-valued Encounter.period can satisfy separate occurrences with separate indexed values.
  • search_count, both _total modes, cursor paging, offset paging, conditional criteria, and tenant isolation use the same result set.
  • Single occurrences and ineligible repeat shapes retain their previous SQL.

Checks

  • cargo fmt --all --check
  • 1,511 helios-persistence library tests
  • PostgreSQL regression, conditional criteria, If-Match, and 92-test search compatibility subset
  • cargo clippy -p helios-persistence --features postgres --all-targets with no new warnings
  • PostgreSQL 16 syntax and execution check for the new plan-capture section

Risk

The measured gain applies to the misestimated nested-rescan regime reported in #1416. When both repeated arms match a very large share of a resource type, PostgreSQL 16 may prefer a parallel hash-semi-join plan for the old conjunction, while the SetOp consume/deduplicate step runs in one process. The benchmark documentation calls out this boundary and the plan script includes the old and new shapes for comparison.

Visual explanation

Open the interactive explanation

Temporary review evidence expires 2026-09-26T20:55:20.497Z.

@Gordex2014
Gordex2014 marked this pull request as ready for review September 21, 2026 21:00
@codecov

codecov Bot commented Sep 21, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.83721% with 4 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
...ence/src/backends/postgres/search/query_builder.rs 98.78% 4 Missing ⚠️

📢 Thoughts on this report? Let us know!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perf(postgres): repeated date bounds trigger nested-loop rescans at corpus scale

1 participant