ui: a bulk-import deferred index rebuild gives no in-progress indication — Resource counts read 0 and the dashboard shows "Still gathering the live figures" at scale

[angela-helios]

Symptom (reported by a tester on a ~6M-resource sqlite import)

After a bulk import completes, the hfs process sits at ~100% CPU and the UI looks broken: on Resources every type count shows 0, and the Home dashboard shows inaccurate stats with "Still gathering the live figures…" on the 1h window. Metrics look unavailable. Nothing tells the user why.

Root cause: the post-import deferred search-index rebuild

Bulk import uses fast-load — it stores resources without indexing, then rebuilds the search index in an async background job:

INFO helios_persistence::core::bulk_submit_worker: bulk fast-load: rebuilding deferred search indexes types=[...]
INFO helios_persistence::search::reindex: deferred-index rebuild started job_id=<id>

The job is CPU-bound (it pegs ~100% of one core the whole time) and builds ~20 index rows per resource, so it runs for a long time at scale (measured: 512k resources → 10.3M rows, ~35 min; 6M → ~100M+ rows → hours). Progress is observable at GET /$reindex-status/<jobId> (status / percentage / processed / total).

The data is not lost. Resources are stored; GET /[type]?_summary=count&_total=accurate returns correct numbers straight from the resources table during the rebuild (verified: 11,705 / 299,941 held throughout a reindex). Searches on already-reindexed types work; not-yet-reindexed types return incomplete/0 until the rebuild reaches them (processed in type order). Everything self-heals when $reindex-status → completed and the CPU frees.

Why the UI shows 0 / "gathering"

The Resources type-rail counts and the Home dashboard both read one async DashboardSnapshot (crates/rest/src/dashboard.rs): a TTL-cached count_by_types (GROUP BY on resources, #959) plus a per-type time-series query (count_deltas_by_bucket). Under the reindex's CPU/DB load at 6M scale those snapshot queries are slow / time out; on failure the snapshot is filled with zeros (the partial SnapshotState) or reports "still computing" (the "gathering the live figures" state). At 512k the snapshot computes fast enough that counts stay correct even mid-rebuild — so the breakage is scale-dependent and won't show on a small test DB.

Reproduction

1. Import enough resources that the reindex runs for a while (the effect scales with corpus size; most visible in the millions).
2. While GET /$reindex-status/<jobId> reads inprogress, open /ui/resources and /ui — counts read 0 and the dashboard shows "gathering". (On a small DB, trigger it directly with POST /$reindex and watch one core hit ~100%.)
3. Confirm the data is intact the whole time via GET /[type]?_summary=count&_total=accurate.

Suggested fixes

1. Surface the rebuild in the UI. A "search index rebuilding — X%" banner on the dashboard and Resources pages (fed by $reindex-status) so 0 counts + "gathering" read as "indexing, come back later" instead of a broken server.
2. Decouple the rail counts from the snapshot. The type-rail counts come from count_by_types (resources table) and are cheap and correct even during a rebuild; sourcing them directly (not via the failure-prone dashboard snapshot) keeps them accurate while the heavier time-series snapshot is still computing.

Environment: sqlite, all-features release at commit 2b1b0347a, Windows 11. Related: #959 (type-count TTL), #993 (rail fabricated 0).

[mauripunzueta]

@angela-helios — heads-up before you start on this. I'm working on #1050 in branch fix/1050-search-index-integrity, and it extends the same $reindex-status surface your suggested fix #1 would read: the reindex already records per-resource failures (type, id, Elasticsearch error), but to_parameters() drops them.

I'd like to pick this issue up in the same branch, so the rebuild banner can show progress and failures from one payload. Have you already started on it? If so, I'll leave it with you and only coordinate the shape of the status payload.

On your suggested fix #2: 909f38fa4 (#1078) reworked the dashboard snapshot after this was filed, so I'll first check whether the rail counts still depend on it. That part also overlaps #993 (the rail showing a fabricated 0).

[mauripunzueta]

@angela-helios — the second half of this issue, the "search index rebuilding — X%" banner, is now implemented in draft PR #1109. It follows the design your #1082 laid out, so please don't start on it separately:

• DashboardSnapshot::reindex_active: Option<ReindexActivity> (running jobs, processed, total) next to import_jobs_active, filled by the REST dashboard provider from ReindexOperation's in-memory job registry. It needs no storage read, so the ui: 1h "FHIR Resources over Time" chart stays on "Waiting for the live figures…" during a large import — dashboard blocks on storage scans instead of degrading to approximate data #1078 constant-time page load holds. ReindexProgress now records its tenant so the provider can filter.
• Home shows a notice--warn line in #dash-live, e.g. "Search index rebuilding — 42% (8,000 of 19,000 resources). Searches may miss stored resources until it finishes." While the rebuild is still counting, the line has no percentage, so it never shows a fabricated "0%". A running rebuild keeps the live region on the fast refresh. The line uses the quiet_notices mechanism, so it is announced once, not on every tick.
• Resources shows the same line under the page head.
• Strings are provided in en, es and de. The new crates/ui/tests/rebuild_banner_http.rs covers the rebuilding, counting and idle states on both pages.

Your rail_counts change is untouched. If you'd rather review this half or take it over, say so and I'll split it out of #1109.