fix(search-index): oversized resources, reindex diagnostics, Mongo keyset receipts, rebuild banner

[mauripunzueta]

Summary

One branch for #1050, #1046 and #1065, because they share one gap: resources are stored in the primary but missing from the search index, and there is no reliable way to see which ones or why.

This is a draft. All four parts are implemented, one commit each. What remains is the full-corpus reconciliation re-run at the end of the test plan.

Closes #1050
Closes #1046
Closes #1065

Done

#1050: Elasticsearch nested-object limit

• ElasticsearchConfig::nested_objects_limit (default 50000), exposed as HFS_ELASTICSEARCH_NESTED_OBJECTS_LIMIT and written into the index template.
• Backend::initialize raises the limit on existing {prefix}_* indices that are below it. It only ever raises, sends index names in chunks, and is non-fatal: a missing privilege logs a warning instead of stopping the server.
• The setting was verified dynamic on Elasticsearch 8.15.0: PUT _settings on an open index was accepted, and a 28,192-nested document (the corpus maximum) then indexed. So existing deployments need no reindex of what already indexed.
• Sizing from the corpus: exactly 458 Provenance records exceed 10,000 target entries, matching the reindex errorCount: 458 in $reindex silently drops resources exceeding Elasticsearch's 10,000 nested-object limit — 458 Provenance unsearchable, errorCount is the only signal #1050. The maximum is 28,192; 176 more sit between 8,000 and 10,000.

#1050: reindex diagnostics and retry classification

• Every per-resource reindex error now carries retryable: true for the conditions REST answers with 503/504 (Unavailable, ConnectionFailed, PoolExhausted, Timeout), false otherwise. An error recorded without the field deserializes as retryable, so older state keeps its old behaviour.
• The Elasticsearch _bulk path classifies failures. A request that never arrived, a 429 or a 5xx is transient and reaches $reindex as Unavailable. A 4xx rejection of the document (the nested-object limit) stays Internal. create_many keeps its existing mapping.
• GET /$reindex-status/{job_id} lists the first 100 failing resources as error parameters (resourceType, resourceId, message, retryable), plus errorsOmitted and, for a failed job, errorMessage. errorCount is unchanged, so existing clients are unaffected.
• bulk-submit: avoid redundant deferred reindex work across manifests #1087's automatic reindex no longer retries a generation that completed with only permanent errors, because a rerun fails identically. It logs the count, the first failing Type/ids and the first error. Any transient error, a failed job or a panic still gets the single retry.

#1046: MongoDB receipt paging by keyset

• get_entry_results_page continues strictly after the last stored (file_url, line_number) and sorts in that order, which both receipt indexes end in. It returns EntryResultContinuation::Keyset with every receipt's stored identity, as SQLite and PostgreSQL have since perf(bulk-submit): replace SQL receipt OFFSET pagination with keyset cursors #986. The skip and the {line_number, file_url} sort, which no index could serve, are gone.
• A receipt missing file_url or line_number, or with a negative line, is an error instead of a default, because a cursor built from an invented identity would skip or repeat receipts. Every MongoDB receipt has carried both fields since Bulk Submit ($bulk-submit) is not available on MongoDB or S3 primaries #521.
• Schema v10 replaces idx_bulk_entry_results_outcome with idx_bulk_entry_results_outcome_line (adds file_url, line_number), so the composite sync's outcome = success pages are an index walk too. The prefix still serves outcome counts. The migration builds this index once at startup.
• The shared receipt paging contract now also runs on MongoDB (exact_sql_pages is renamed exact_keyset_pages), and a profiler guard asserts that filtered receipt pages have no blocking sort and no skip.

Measured on the #941 corpus (MongoDB 7.0, helios.bulk_entry_results, 18,955,865 receipts in one manifest, schema v9 indexes)

explain("executionStats"), page size 1000, same manifest scope, unfiltered (served by idx_bulk_entry_results_line):

query	plan	blocking sort	keys examined	time
before: skip 15000, sort {line_number, file_url}	FETCH <- SKIP <- SORT <- IXSCAN	yes	18,955,865	39,096 ms
after: first page, sort {file_url, line_number}	LIMIT <- FETCH <- IXSCAN	no	1,000	1 ms
after: keyset past receipt 9,000,000	LIMIT <- FETCH <- SORT_MERGE <- IXSCAN <- IXSCAN	no	1,001	9 ms

The old query examined every key in the manifest on every page and sorted in memory, independent of skip. The keyset page examines limit + 1 keys at any depth.

The composite sync (CompositeSubmitJobs::sync_ingested) filters outcome = success, the shape the v10 index exists for. Its plan was measured on 2,000,000 synthetic receipts (20 files × 100,000 lines, 95% success) in a scratch database on the same MongoDB 7.0 server, dropped afterwards:

indexes	query	plan	blocking sort	keys examined	time
v9	before: outcome=success, skip 15000, sort {line_number, file_url}	SKIP <- SORT <- FETCH <- IXSCAN (outcome)	yes	1,900,000	3,692 ms
v10	after: filtered, first page	LIMIT <- FETCH <- IXSCAN (outcome_line)	no	1,000	2 ms
v10	after: filtered, keyset past receipt 1,000,000	LIMIT <- FETCH <- SORT_MERGE <- IXSCAN <- IXSCAN (outcome_line)	no	1,001	3 ms
v10	after: unfiltered, keyset past receipt 1,000,000	same shape (line)	no	1,001	3 ms

Building idx_bulk_entry_results_outcome_line took 3,979 ms on 2M receipts, so it adds roughly 40 s to the one-time v10 migration on the #941 corpus. Without it, the filtered keyset query falls back to the line index with a residual outcome filter: still no sort, but keys examined grow with how rare the filtered outcome is.

#1065: "search index rebuilding" banner

PR #1082 (@angela-helios) already fixed the first half of #1065: the rail no longer shows placeholder zeros during a rebuild. It named the other half as follow-up, a reindex_jobs_active-style snapshot field fed from the reindex job registry and driving a banner. This implements that design:

• ReindexProgress records the tenant it rebuilds.
• DashboardSnapshot::reindex_active: Option<ReindexActivity> (running jobs, processed, total) sits next to import_jobs_active. It is None when nothing is rebuilding or no $reindex operation is wired.
• The REST dashboard provider fills it from ReindexOperation's in-memory job registry, never storage, so page loads stay constant-time (ui: 1h "FHIR Resources over Time" chart stays on "Waiting for the live figures…" during a large import — dashboard blocks on storage scans instead of degrading to approximate data #1078).
• Home shows a warning line in the live region: "Search index rebuilding — 42% (8,000 of 19,000 resources). Searches may miss stored resources until it finishes." While the rebuild is still counting, the line has no percentage, so it never shows a fabricated "0%". A running rebuild keeps the region on the fast refresh. The line is announced once, not on every percentage tick.
• Resources shows the same line under its page head.
• Strings are provided in en, es and de.

Context and coordination

• perf(persistence): parallelize bulk-ingest extraction, partial-index LIKE predicates #1093 (smunini, merged) owns the reindex performance work. This branch changes only error reporting and retry classification in search/reindex.rs, not the rebuild path, and leaves the begin/end_bulk_index_rebuild exit handling untouched.
• bulk-submit: avoid redundant deferred reindex work across manifests #1087's retry stays in place for transient failures. Its existing test injects Unavailable errors and passes unchanged.
• feat(cluster): make HFS cluster-capable behind a load balancer #269 (aacruzgon) also edits search/reindex.rs, so a small conflict there is likely.
• ui: a bulk-import deferred index rebuild gives no in-progress indication — Resource counts read 0 and the dashboard shows "Still gathering the live figures" at scale #1065 is assigned to @angela-helios. Her PR fix(ui): don't render fabricated zero counts while a deferred index rebuild degrades the snapshot (#1065) #1082 fixed the first half and described the second; this branch follows that design rather than inventing another, and I'll drop ui: a bulk-import deferred index rebuild gives no in-progress indication — Resource counts read 0 and the dashboard shows "Still gathering the live figures" at scale #1065 from this PR if she has already started on it.
• The composite sync's create_many error mapping is deliberately unchanged, so fix(persistence): bulk-submitted resources reach Elasticsearch and the receipt reports the ones that do not (#1007) #1049's receipt semantics are unaffected.

Test plan

• cargo test -p helios-persistence --features elasticsearch --lib backends::elasticsearch::schema::tests (2 passed)
• cargo test -p helios-persistence --features elasticsearch --test elasticsearch_tests config unit tests (2 passed)
• Integration tests against a real Elasticsearch (2 passed, 49.7 s): es_integration_resource_over_elasticsearch_default_nested_limit_indexes, es_integration_startup_raises_nested_limit_on_existing_index
• cargo check -p helios-rest and cargo check -p helios-hfs --features elasticsearch,mongodb (clean, 0 warnings)
• Reindex and ES unit tests: cargo test -p helios-persistence --features elasticsearch --lib -- search::reindex backends::elasticsearch::storage::tests (31 passed, including bulk-submit: avoid redundant deferred reindex work across manifests #1087's automatic_reindex_retries_failure_error_result_and_panic_once, unchanged)
• Negative control: classifying every writer error as transient makes automatic_reindex_does_not_retry_a_completion_with_only_permanent_errors fail (2 runs instead of 1)
• ES integration, now also asserting the nested rejection is permanent: es_integration_startup_raises_nested_limit_on_existing_index, es_integration_resource_over_elasticsearch_default_nested_limit_indexes, es_integration_reindex_page_* (4 passed)
• cargo clippy -p helios-persistence --features elasticsearch,mongodb --lib --tests: no findings in changed files
• cargo test -p helios-persistence --features mongodb --test mongodb_tests -- bulk_submit::test_receipt_paging_contract bulk_submit::test_receipt_pages_are_index_walks_without_a_blocking_sort bulk_submit::test_entry_results_from_two_files_do_not_collide (3 passed against a MongoDB testcontainer; each prints its Verified … line, so none skipped)
• Negative control: restoring the old {line_number, file_url} sort fails the contract on ordering and the plan guard on hasSortStage: true
• SQLite contract: backends::sqlite::bulk_submit::tests (45 passed); postgres_tests compiles with the renamed contract entry point
• Explain on the Manual test pass: MongoDB + Elasticsearch backend #941 corpus and on 2M synthetic receipts (tables above)
• cargo clippy -p helios-persistence --features mongodb,elasticsearch,postgres --lib --tests: no new findings (the two in touched files, sqlite/bulk_submit.rs:2584 and mongodb_tests.rs:3145, are pre-existing on main)
• helios-ui: lib 258 passed (including en/es/de key-set parity); rebuild_banner_http (rebuilding, still counting, and idle, on Home and Resources, plus the fast refresh while rebuilding); dashboard_pending_http 26, rail_counts_http 1, dashboard_view_all_types_http 4
• helios-rest dashboard::tests 38 passed (including reindex_activity_sums_only_the_tenants_running_jobs); helios-observability dashboard 52 passed; persistence search::reindex 30 passed
• cargo check -p helios-hfs --features elasticsearch,mongodb; cargo clippy -p helios-observability -p helios-rest -p helios-ui --lib --tests: no new findings (3 pre-existing on main)
• CI green on 2cfde9b94 (Test Rust, Linting, Code Coverage and codecov/patch, UI browser suite, search + Resources R4/R4B/R5/R6). The first run failed only because test_v7_to_v8_manifest_aware_submit_file_identity still expected schema v9. It now expects v10 and also checks the v7 → v10 receipt index swap.
• Full local runs: mongodb_tests 138, postgres_tests 184, elasticsearch_tests 110, all passed
• Re-run the Manual test pass: MongoDB + Elasticsearch backend #941 mongo-es corpus reconciliation: expect all 11,705 Provenance searchable

[codecov]

Codecov Report

❌ Patch coverage is 99.16201% with 3 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
crates/persistence/src/search/reindex.rs	99.55%	1 Missing ⚠️
crates/rest/src/config.rs	80.00%	1 Missing ⚠️
crates/ui/src/lib.rs	96.77%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!