Problem
A search-indexing pipeline run whose DatasetSelector returns zero datasets still reaches RunWriter.commit() (Pipeline.run commits unconditionally after a zero-iteration loop). Both transactional rebuild writers then treat "empty selection" as "the index should be empty":
InPlaceRebuild — commit's registry-membership sweep computes departedSources = indexed − selected. With an empty selection that is every source, so the sweep issues delete-by-filter against the whole live collection. A populated index is wiped. This path is new and has no test.BlueGreenRebuild — swaps the name alias to the freshly built (empty) collection and drops the previous one, so the live index goes empty. This is currently intended and tested (blue-green-rebuild.test.ts "publishes an empty collection for an empty run", carried over from the old rebuild()).
The danger is that "zero datasets" is not always a genuine empty registry — a transient Dataset Register outage, an auth blip, or a selection-criteria bug can return an empty result set, and a single such run then destroys production search until the next good rebuild.
Why it matters
The object-index use case (#534) runs on a schedule against the Register over potentially millions of objects. A one-off empty selection from a flaky upstream should be a no-op, not a full wipe.
Decision needed
Is "empty selection ⇒ empty index" ever the intended outcome, or should an empty (or implausibly small) selection be refused?
Options:
- Guard in the pipeline / writers: skip the destructive parts of
commit (In-place membership sweep, Blue/green alias swap + drop-previous) when selectedSources() is empty, and surface it as a skipped run rather than a wipe. Update the Blue/green "empty run" test to assert the live index is left intact.
- Sanity threshold: refuse to sweep/swap when the selection shrank beyond a configured fraction of the currently-indexed source count (guards against partial-outage under-selection, not just fully-empty).
- Accept as-is and document that an empty selection wipes the index, making it the deployment's responsibility to never run the pipeline against an empty/degraded Register.
Preference: option 1 as the floor (empty selection is never a legitimate reason to delete a live index), with option 2 as a follow-up if partial under-selection proves to be a real failure mode.
References
Problem
A search-indexing pipeline run whose
DatasetSelectorreturns zero datasets still reachesRunWriter.commit()(Pipeline.runcommits unconditionally after a zero-iteration loop). Both transactional rebuild writers then treat "empty selection" as "the index should be empty":InPlaceRebuild—commit's registry-membership sweep computesdepartedSources = indexed − selected. With an empty selection that is every source, so the sweep issues delete-by-filter against the whole live collection. A populated index is wiped. This path is new and has no test.BlueGreenRebuild— swaps thenamealias to the freshly built (empty) collection and drops the previous one, so the live index goes empty. This is currently intended and tested (blue-green-rebuild.test.ts"publishes an empty collection for an empty run", carried over from the oldrebuild()).The danger is that "zero datasets" is not always a genuine empty registry — a transient Dataset Register outage, an auth blip, or a selection-criteria bug can return an empty result set, and a single such run then destroys production search until the next good rebuild.
Why it matters
The object-index use case (#534) runs on a schedule against the Register over potentially millions of objects. A one-off empty selection from a flaky upstream should be a no-op, not a full wipe.
Decision needed
Is "empty selection ⇒ empty index" ever the intended outcome, or should an empty (or implausibly small) selection be refused?
Options:
commit(In-place membership sweep, Blue/green alias swap + drop-previous) whenselectedSources()is empty, and surface it as a skipped run rather than a wipe. Update the Blue/green "empty run" test to assert the live index is left intact.Preference: option 1 as the floor (empty selection is never a legitimate reason to delete a live index), with option 2 as a follow-up if partial under-selection proves to be a real failure mode.
References
packages/pipeline/src/pipeline.tsrun()— unconditionalcommit().packages/search-typesense/src/in-place-rebuild.tscommit(membership sweep) andsrc/sweep.tsdepartedSources.