fix(e2e): harden ObsessionDB live e2e against replica lag (supersedes #220)

[KeKs0r]

Summary

The obsessiondb job on main has failed on four consecutive SHAs (256ec62, 3cc768d, fedbf56, 7fe44e5). This PR replaces #220. It keeps #220's sound parts, fixes a second ingest failure mode that #220 misses, and closes gaps that would let the dictionary and mv_replay tests keep failing. It targets main and touches the same lines as #220 and #218, so it is meant to land instead of them, not stacked on them.

Root pattern: the tests that fail are the ones where a request can reach a different replica than the previous one. That happens with the stateless executor, the dictionary's HTTP source, and fire-and-forget chunk inserts. A single waitForTable/waitForRows success only shows that one replica caught up.

Review of #220

#220 change	Verdict
waitForTable after journal CREATE in ensure()	Correct but incomplete. It fixes the UNKNOWN_TABLE in run 36370187217. But the ingest failures on fedbf56 and 3cc768d were a DOMException TimeoutError, not UNKNOWN_TABLE. That error comes from run_finished being hard-capped at AbortSignal.timeout(5000) even on a healthy, uncancelled run. So #220 would still leave ingest red on the run that is red most often. Also, run_started had no retry at all, so one request to a lagging replica fails the whole run even after ensure() saw the table on another replica.
Dictionary: one SYSTEM RELOAD DICTIONARY, then poll dictGet	Gap. The reload runs once, outside the poll. If that single reload's HTTP source request reaches a replica that hasn't seen the INSERT yet, the dictionary caches "" for its 300s lifetime. Polling dictGet for 10s can't recover from that.
mv_replay: poll target until rows.length === BUCKETS	Possibly ineffective. The target read already used select_sequential_consistency = 1. Buckets 0 and 1 missing is at least as consistent with chunk INSERT…SELECTs reading the source on a lagging replica and writing 0 rows. In that case the data is permanently missing, and the poll just times out. It fails correctly, but the flake stays. Waiting on length, not values, also loses the diff.
text-index: raise the two timeouts to 120s/60s	Hides the cause. The escape test makes ~94 sequential remote requests. #218's batching removes the cause, and the quoted-literal test has the same pattern (18 requests).

What this PR changes

• packages/clickhouse/src/e2e-testkit.ts: new pollUntil(read, predicate). It repeats the whole observation until the predicate holds, then returns the last value so the caller's expect still shows a real diff. It retries reads that throw, such as a transient UNKNOWN_TABLE. It is re-exported from the CLI testkit.
• packages/plugin-ingest/src/journal.ts: keeps fix(e2e): stabilize ObsessionDB live e2e flakes on main #220's waitForTable after CREATE.
• packages/plugin-ingest/src/executor.ts (product change; changeset included):
  • run_started/run_finished now use the same bounded retry as stream facts (4 attempts, 250ms base). Run facts are deterministic and appends use insert_deduplication_token, so retrying is safe.
  • Terminal facts (work_finished, run_finished) are bounded by the live run's deadline. The fixed 5s grace now applies only after cancellation or budget exhaustion; once the run is interrupted, behavior is unchanged. This is what fixes the TimeoutError.
  • New unit tests: a slow (5.5s) terminal append on a live run still lands, and a transient run-fact failure is retried. All four fail on main; one reproduces the exact CI TimeoutError: The operation timed out.
• packages/plugin-ingest/src/ingest.e2e.test.ts: the journal is created through ensure() in beforeAll, like the raw and destination tables, so its propagation overlaps setup instead of racing the first run. Hooks get 60s timeouts; bun's default is 5s.
• packages/cli/src/test/migrate-dictionary.e2e.test.ts:
  • The dictionary's HTTP source URL adds select_sequential_consistency=1, so whichever replica serves a load catches up with the seed INSERT before reading.
  • Every poll attempt runs SYSTEM RELOAD DICTIONARY + dictGet, both after seeding and after CREATE OR REPLACE.
• packages/plugin-backfill/src/mv-replay-plan.e2e.test.ts:
  • Chunk SQL gets select_sequential_consistency = 1, so no chunk replays an empty source partition.
  • The target is polled until it equals the expected rows.
  • The assertion message includes per-chunk writtenRows, so a future failure shows whether data never landed (0) or was only slow to appear.
• packages/cli/src/test/text-index.e2e.test.ts: the escape checks are batched into one UNION ALL query (the same approach as test(cli): batch text-index escape checks in one query #218), and so are the quoted-literal checks. Both tests get the 60s timeout the other live tests use.
• Docs: plugins/ingest.md now describes the terminal-append limit accurately.

Confidence

• High: text-index (the cause is removed), ingest TimeoutError (reproduced in a unit test, gone after the fix), dictionary (the stale-cache path is removed).
• Medium–high: ingest UNKNOWN_TABLE and mv_replay. These are layered defences against replica lag; nothing can prove a replica is up to date from the client side. If mv_replay still fails, the message now shows which of the two causes it is.

How tested

• bunx turbo run typecheck lint --filter=@chkit/clickhouse --filter=@chkit/plugin-ingest --filter=@chkit/plugin-backfill --filter=chkit: 14/14 tasks pass.
• bunx turbo run test for the same packages against a local single-node ClickHouse 26.10 with the CI config and seeded fixtures: clickhouse 47, plugin-backfill 131, chkit 310, plugin-ingest 61 pass, 0 fail. The batched escape test now takes about 140ms.
• I checked the mv_replay diagnostic by forcing a mismatch. It printed rows written per chunk: {…: 1, …}.
• Not run against ObsessionDB (no secrets here). The post-merge obsessiondb job is the real check.

Needs a human

• The executor change is a runtime behavior change. A hung terminal journal write on a live run is now bounded by the run's duration budget (default 1h) instead of 5s. This matches how ensure, run_started, and batch_committed are already bounded, and the post-interruption grace is unchanged.
• readCheckpoint is still not retried. A replica that lags after both ensure() and run_started would still fail that stream. I left it out to keep the product change small.
• Other CLI live tests still inherit the package's 15s --concurrent default. Only text-index hit it in the logs.
• If this lands, fix(e2e): stabilize ObsessionDB live e2e flakes on main #220 and test(cli): batch text-index escape checks in one query #218 can be closed. I have not closed them.