fix(search-index): oversized resources, reindex diagnostics, Mongo keyset receipts, rebuild banner

.agents/skills/bulk-data-submit/SKILL.md

@@ -112,12 +112,12 @@ The backend capability splits into `BulkSubmitIngest` (the synchronous `BulkSubm
 - With `HFS_BULK_SUBMIT_DEFER_INDEXING=true` (bulk fast-load, #903, the default since #946), ingestion skips search-index and FTS writes. The worker requests an automatic full-type reindex after the manifest becomes terminal, so `$bulk-submit-status` can answer `200` while search is still incomplete.
 - Automatic deferred reindex requests share the coordinator owned by their `ReindexOperation` (#1087). One tenant has one active generation plus one pending, deduplicated type set. The process runs at most `W` automatic generations and retains at most `2W` tenant entries, where `W` is the existing `HFS_BULK_SUBMIT_WORKER_CONCURRENCY` value. Admission applies backpressure before it adds another tenant. HFS has no separate bulk-submit reindex-concurrency variable.
 - The coordinator releases and reacquires its execution permit between generations so another admitted tenant can progress. Separate `ReindexOperation` instances remain independent because they can have different writer and registry sets. Explicit `$reindex` jobs bypass this coordinator and can overlap automatic work.
-- A clean automatic generation ends `Completed` with no resource errors. Failure, panic, or an errorful completion gets one retry with the active and pending types. A second consecutive failure abandons that generation and logs the manual `$reindex` repair. Independently queued work that arrived during the retry still runs as a new generation with its own retry budget. Cancellation does not retry the cancelled active types, but independently queued pending types still run after the cancelled task has stopped writing.
+- A clean automatic generation ends `Completed` with no resource errors. Failure, panic, or a completion with *transient* resource errors (a backend that was unavailable, timed out, or answered Elasticsearch `429`/`5xx`) gets one retry with the active and pending types. A completion whose resource errors are all *permanent* — a document the search backend rejects outright, such as one over Elasticsearch's nested-object limit (#1050) — is not retried, because a rerun fails identically; it logs the error count and the first failing `Type/id`s instead. A second consecutive failure abandons that generation and logs the manual `$reindex` repair. Independently queued work that arrived during the retry still runs as a new generation with its own retry budget. Cancellation does not retry the cancelled active types, but independently queued pending types still run after the cancelled task has stopped writing.
 - Coordination and reindex job state are in memory and local to one HFS process. A restart loses pending work, and separate processes do not coordinate. Full-type scans remain in use, so a finite burst can still cause one active scan and one accumulated follow-up. Limiting work to successful manifest IDs was evaluated and deferred because the generic path lacks bounded receipt deduplication, current-resource handling for missing or deleted IDs, and consistent semantics for every composite target.
 - The coordination logic is common to standalone SQLite, PostgreSQL, and MongoDB plus the Elasticsearch composites that wire reindex. Current performance evidence is PostgreSQL-only; do not claim equivalent latency or database-work improvements for the other backends without measuring them. See `docs/deferred-reindex-coordination-benchmark.md`.
 - MongoDB ingests a batch, not an entry (#1000): one `find` resolves which ids already exist, then one `insert`/`update` command per collection. The per-entry path it replaced cost ~9 round trips per resource and ran at ~60–76 resources/s with the server two-thirds idle; batched it reaches ~720, and ~3 100 with indexing deferred. The flush is ordered commands, not one transaction — the per-entry path was not atomic across a batch either, and a batch-wide transaction would widen #1001 from one lost entry to a whole batch.
 - Composite deployments (primary + Elasticsearch, including the `mongo-es`/`s3-es` modes) must wrap the primary's job store with `composite_submit_jobs(...)`: ingestion runs on the primary, whose own indexing is offloaded, so without the wrapper a completed import is readable by id and invisible to every search (#882, and #1021 for the modes that were missed). Guard: `crates/hfs/tests/bulk_submit/run_composite_es_index_check.sh`.
-- On those composite deployments (#1007), the worker copies every manifest's ingested resources into the secondary search index *before* writing the manifest's receipt (not at `finish_manifest`, which no longer syncs — a manifest already terminal is never re-synced by a restart). A resource the secondary still rejects after its retries gets an entry result of `processing-error` in the receipt, with an OperationOutcome (`incomplete`) naming the `Type/id`, the rejecting backend, and `POST /{type}/$reindex` as the repair; the resource itself stays stored and readable by id, and `failed_entries` on the status counts it.
+- On those composite deployments (#1007), the worker copies every manifest's ingested resources into the secondary search index *before* writing the manifest's receipt (not at `finish_manifest`, which no longer syncs — a manifest already terminal is never re-synced by a restart). A resource the secondary still rejects after its retries gets an entry result of `processing-error` in the receipt, with an OperationOutcome (`incomplete`) naming the `Type/id`, the rejecting backend, and `POST /{type}/$reindex` as the repair; the resource itself stays stored and readable by id, and `failed_entries` on the status counts it. If the rejection is Elasticsearch's nested-object limit (`The number of nested documents has exceeded the allowed limit`), `$reindex` fails the same way until `HFS_ELASTICSEARCH_NESTED_OBJECTS_LIMIT` (default 50000, raised on existing indices at startup) is above that resource's nested value count (#1050). Each failed resource is listed, with its error and whether it is retryable, in `GET /$reindex-status/{job_id}`.
 - After that copy, the worker compares the primary's tenant-wide count against each secondary's for every resource type the manifest ingested. A mismatch is recorded as a `warning` OperationOutcome (also `incomplete`, naming both counts) in the manifest's `error` artifact and logged. This only runs when `HFS_COMPOSITE_SYNC_MODE` is `synchronous` or `hybrid`; under the default `asynchronous` mode the secondary's count reflects whatever had already drained from its queue rather than this manifest's sync, so the check is skipped and the receipt then only guarantees the resources committed on the primary.
 - Without `HFS_ELASTICSEARCH_WRITE_REFRESH=wait_for`, a small count difference can be a write not yet visible rather than a real gap; reconfirm with `GET /{type}?_summary=count` before treating it as drift. See "Verifying and repairing search drift" below.
 - `HFS_BULK_SUBMIT_DEFER_INDEXING` is read once at startup, not per submission. Repair a stale or missed index with `$reindex`.

.claude/skills/bulk-data-submit/SKILL.md

@@ -119,7 +119,7 @@ The backend capability splits into `BulkSubmitIngest` (the synchronous `BulkSubm
 - With `HFS_BULK_SUBMIT_DEFER_INDEXING=true` (bulk fast-load, #903 — **the default since #946**) ingestion skips the search-index and FTS writes and an automatic per-type reindex rebuilds them when each manifest finishes. Reads and history are complete throughout; search sees a manifest's resources once its reindex lands. That rebuild is started *after* the manifest is already terminal and is fire-and-forget (`bulk_submit_worker.rs` → `reindex.rs`, `tokio::spawn`), so `$bulk-submit-status` answers `200` while search is still incomplete, and the job lives only in an in-memory map — no column on `bulk_manifests` records that indexing is outstanding and nothing re-fires it at startup. A restart in that window is not recoverable on its own.
 - MongoDB ingests a batch, not an entry: one `find` resolves which of the batch's ids already exist, then one `insert` or `update` command per collection writes the whole batch (`backends/mongodb/bulk_ingest.rs`). Before #1000 each entry cost ~9 round trips of its own — a `read`, `create`'s second existence probe, the resource and history inserts, a search-index delete and insert, a transaction commit, the rollback record and the receipt — which pinned ingest at ~60–76 resources/s with `mongod` two-thirds idle. The batch flush is a sequence of commands rather than one transaction. Every command is retried on a transient driver error (`RetryableError`/`RetryableWriteError` label, I/O error, cleared pool — not a server-selection timeout) with 100 ms doubling backoff capped at 1 s over six attempts, checking the submission's cancel token before each sleep; a retry never duplicates what an earlier attempt landed (resources are re-read and matched on version + the batch's own `last_updated` + content, history and rollback rows dedupe on their unique keys, the search index is cleared before re-insert). When a stage outlives its retries the batch's entries get `processing-error` receipts with issue code `transient` and the file continues with its next batch, so `max_errors`/`continue_on_error` govern backend failures too (#1001); re-submitting the file converges. Only a receipt write that itself fails after retries still aborts the file. The manifest counters are a `$inc` and may over-count one batch if a retried attempt had actually landed — the receipts are authoritative.
 - **On a composite deployment (primary + Elasticsearch), the ingest engine does not reach the secondary by itself.** Ingestion runs on the *primary's* engine, and the primary deliberately skips its own indexing when search is offloaded — so `main.rs` wraps the primary's job store in `CompositeSubmitJobs`, which syncs each manifest's ingested resources into the secondary. Every composite mode must call `composite_submit_jobs(...)`; `mongo-es` and `s3-es` did not, and a completed import there was readable by id and invisible to every search — 15.27M of 15.28M resources on the reported deployment, with `GET` by id passing every smoke test (#1021). `crates/hfs/tests/bulk_submit/run_composite_es_index_check.sh` asserts the searchable count, not just readability, and is the guard against a fourth composite backend repeating it.
-- The sync itself runs *before* the manifest's receipt is written (#1007), as an explicit worker step — not at `finish_manifest`, which no longer syncs by itself, so a manifest that already reached a terminal state is never re-synced by a restart; repair it with `$reindex`. A resource the secondary still rejects after its retries gets an entry result of `processing-error` in the receipt, carrying an OperationOutcome (`incomplete`) that names the `Type/id`, the rejecting backend, and `POST /{type}/$reindex` as the repair; the resource itself stays stored and readable by id, and the status's `failed_entries` counts it.
+- The sync itself runs *before* the manifest's receipt is written (#1007), as an explicit worker step — not at `finish_manifest`, which no longer syncs by itself, so a manifest that already reached a terminal state is never re-synced by a restart; repair it with `$reindex`. A resource the secondary still rejects after its retries gets an entry result of `processing-error` in the receipt, carrying an OperationOutcome (`incomplete`) that names the `Type/id`, the rejecting backend, and `POST /{type}/$reindex` as the repair; the resource itself stays stored and readable by id, and the status's `failed_entries` counts it. If the rejection is Elasticsearch's nested-object limit (`The number of nested documents has exceeded the allowed limit`), `$reindex` fails the same way until `HFS_ELASTICSEARCH_NESTED_OBJECTS_LIMIT` (default 50000, raised on existing indices at startup) is above that resource's nested value count (#1050). Each failed resource is listed, with its error and whether it is retryable, in `GET /$reindex-status/{job_id}`.
 - After that copy, for every resource type the manifest ingested, the worker compares the primary's tenant-wide resource count against each secondary's. A mismatch is recorded as a `warning` OperationOutcome (also `incomplete`, naming both counts) in the manifest's `error` artifact and logged on the server. This check only runs when `HFS_COMPOSITE_SYNC_MODE` is `synchronous` or `hybrid`; under the default `asynchronous` mode the secondary's count reflects whatever had already drained from its queue rather than this manifest's own sync, so the check is skipped and the receipt then guarantees only that the resources committed on the primary.
 - Without `HFS_ELASTICSEARCH_WRITE_REFRESH=wait_for`, a small count difference can be a write that has not become visible yet rather than a real gap; reconfirm with `GET /{type}?_summary=count` before treating it as drift. See "Verifying and repairing search drift" below.
 - **Under deferred indexing the rebuild is the import.** On SQLite the fast-load

.claude/skills/run-hfs-server/SKILL.md

@@ -75,6 +75,7 @@ HFS_SERVER_PORT=3000 HFS_LOG_LEVEL=debug cargo run --bin hfs
 | `HFS_ELASTICSEARCH_PASSWORD` | none | Elasticsearch basic auth password |
 | `HFS_ELASTICSEARCH_REFRESH_INTERVAL` | `1s` | Index `refresh_interval` applied when an index is created (`-1` disables periodic refresh) |
 | `HFS_ELASTICSEARCH_WRITE_REFRESH` | `false` | `refresh` parameter on index/delete writes: `false`, `wait_for`, or `true` |
+| `HFS_ELASTICSEARCH_NESTED_OBJECTS_LIMIT` | `50000` | Index `mapping.nested_objects.limit`: max nested objects per document across all nested search-parameter fields. Set on new indices; raised at startup on existing indices below it |
 | `HFS_COMPOSITE_SYNC_MODE` | `asynchronous` | ES-backed composite write sync mode: asynchronous, synchronous, or hybrid |
 
 Use `HFS_COMPOSITE_SYNC_MODE=synchronous` **and** `HFS_ELASTICSEARCH_WRITE_REFRESH=wait_for` when callers need read-your-write search semantics, such as integration tests or bulk loads that immediately search. Either alone still leaves a window: synchronous mode only guarantees the document reached Elasticsearch, and it is not searchable until the next index refresh. See `crates/persistence/README.md` (Search visibility on Elasticsearch-backed composites).

README.md

@@ -301,6 +301,7 @@ compressed when the client sends `Accept-Encoding`.
 | `HFS_ELASTICSEARCH_PASSWORD` | *(none)* | ES basic auth password |
 | `HFS_ELASTICSEARCH_REFRESH_INTERVAL` | `1s` | Index `refresh_interval` applied when an index is created (`-1` disables periodic refresh) |
 | `HFS_ELASTICSEARCH_WRITE_REFRESH` | `false` | `refresh` parameter on index/delete writes: `false`, `wait_for`, or `true` |
+| `HFS_ELASTICSEARCH_NESTED_OBJECTS_LIMIT` | `50000` | Index `mapping.nested_objects.limit`: max nested objects per document across all nested search-parameter fields. Set on new indices; raised at startup on existing indices below it |
 
 **PostgreSQL** (used to assemble a connection when `HFS_DATABASE_URL` is not set)
 

book/src/configuration/environment-variables.md

@@ -57,6 +57,7 @@ are compressed when the client sends `Accept-Encoding`.
 | `HFS_ELASTICSEARCH_PASSWORD` | *(none)* | Elasticsearch basic auth password |
 | `HFS_ELASTICSEARCH_REFRESH_INTERVAL` | `1s` | Index `refresh_interval` applied when an index is created (`-1` disables periodic refresh) |
 | `HFS_ELASTICSEARCH_WRITE_REFRESH` | `false` | `refresh` parameter on index/delete writes: `false`, `wait_for`, or `true` |
+| `HFS_ELASTICSEARCH_NESTED_OBJECTS_LIMIT` | `50000` | Index `mapping.nested_objects.limit`: max nested objects per document across all nested search-parameter fields. Set on new indices; raised at startup on existing indices below it |
 | `HFS_S3_BUCKET` | `hfs` | S3 bucket name |
 | `HFS_S3_REGION` | *(AWS chain)* | AWS region override |
 | `HFS_S3_PREFIX` | *(none)* | Optional key prefix for all S3 object keys |

crates/hfs/README.md

@@ -99,6 +99,7 @@ Options:
 | `HFS_COMPOSITE_SYNC_MODE` | `asynchronous` | Composite-store write sync mode for ES-backed backends (`sqlite-elasticsearch`, `postgres-elasticsearch`, `mongodb-elasticsearch`, `s3-elasticsearch`). One of `asynchronous`, `synchronous`, `hybrid`. With `asynchronous` (default) the write returns as soon as the primary commits and the search backend is updated on a background worker — lowest latency, but a follow-up search can race the indexing. Use `synchronous` when callers need read-your-write semantics (e.g. integration tests, bulk-load flows that immediately search). Ignored when the storage backend has no search secondary. |
 | `HFS_ELASTICSEARCH_REFRESH_INTERVAL` | `1s` | Elasticsearch index `refresh_interval` for ES-backed backends. Controls how quickly indexed documents become searchable when no per-write refresh is requested. `-1` disables periodic refresh entirely. Applied when an index is created; indices that already exist keep their current setting. |
 | `HFS_ELASTICSEARCH_WRITE_REFRESH` | `false` | The `refresh` parameter applied to Elasticsearch index/delete operations. One of `false` (no per-write refresh), `wait_for` (block each write until the affected shards refresh), or `true` (force a refresh per write; expensive, low-volume deployments only). |
+| `HFS_ELASTICSEARCH_NESTED_OBJECTS_LIMIT` | `50000` | Elasticsearch `index.mapping.nested_objects.limit`: the maximum nested objects a single document may contain, summed across every nested search-parameter field. Elasticsearch's own default of 10000 rejects larger documents, so a resource with very many indexed values (for example a Synthea `Provenance` with 13,554 `target` references) is stored but never searchable. Applied to new indices through the index template, and raised at startup on existing indices that are below it; the setting is dynamic, so no reindex is needed for resources that already indexed. |
 
 Read-after-write search on an ES-backed composite needs **both**
 `HFS_COMPOSITE_SYNC_MODE=synchronous` and

crates/hfs/src/main.rs

@@ -2191,6 +2191,7 @@ async fn start_sqlite_elasticsearch(
         fhir_version: config.default_fhir_version,
         refresh_interval: config.elasticsearch_refresh_interval.clone(),
         write_refresh: es_write_refresh_from_config(&config)?,
+        nested_objects_limit: config.elasticsearch_nested_objects_limit,
         ..Default::default()
     };
 
@@ -2485,6 +2486,7 @@ async fn start_postgres_elasticsearch(
         fhir_version: config.default_fhir_version,
         refresh_interval: config.elasticsearch_refresh_interval.clone(),
         write_refresh: es_write_refresh_from_config(&config)?,
+        nested_objects_limit: config.elasticsearch_nested_objects_limit,
         ..Default::default()
     };
 
@@ -2696,6 +2698,7 @@ async fn start_mongodb_elasticsearch(
         fhir_version: config.default_fhir_version,
         refresh_interval: config.elasticsearch_refresh_interval.clone(),
         write_refresh: es_write_refresh_from_config(&config)?,
+        nested_objects_limit: config.elasticsearch_nested_objects_limit,
         ..Default::default()
     };
 
@@ -3108,6 +3111,7 @@ async fn start_s3_elasticsearch(
         fhir_version: config.default_fhir_version,
         refresh_interval: config.elasticsearch_refresh_interval.clone(),
         write_refresh: es_write_refresh_from_config(&config)?,
+        nested_objects_limit: config.elasticsearch_nested_objects_limit,
         ..Default::default()
     };