fix(e2e): copy every async-backfill source row on ObsessionDB

[KeKs0r]

Summary

The obsessiondb job on 9632090 (#226) failed async-backfill.e2e.test.ts:146: executeBackfill reported every chunk done, the source count was 2000, and the target had 500 (SOURCE_ROWS / 4). Resume and replay in the same file then copied all 2000 rows, so the plan already covered every partition. A stateless chunk INSERT…SELECT landed on a replica that had not attached the other partitions; with parallel replicas on, that scan finishes successfully with 0 written rows. Reusing the plan id would also reuse executeBackfill's query id, so a retry would read the old 0-row query_log line.

The harness now waits until a session can see all 2000 rows across 4 partitions (syncing replicas when SYSTEM SYNC REPLICA is granted, and falling back on ACCESS_DENIED without requiring system.replicas), runs chunk reads with enable_parallel_replicas = 0, and reissues any chunk that finished with 0 written rows under a new plan id. The test still asserts the target count is exactly 2000.

The obsessiondb job is main-only (if: push && main), so PR CI cannot prove the live ObsessionDB path. Single-node verify still runs this e2e.

Test plan

• bun test src/async-backfill.test.ts src/mv-replay-visibility.test.ts
• async-backfill.e2e.test.ts against local ClickHouse (full copy, resume, replay)
• obsessiondb job on main after merge

  

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-28T17:33:54.513026Z	25ee010	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

chkit/packages/plugin-backfill/src/async-backfill.e2e.test.ts

Lines 307 to 311 in 25ee010

	const result = await executeBackfill({
	executor: runExecutor,
	planId: `${plan.planId}-resume`,
	chunks: plan.chunks,
	buildQuery: ({ id }) => sqlForChunk(id),

Retry empty chunks in the resume and replay cases

When SYSTEM SYNC REPLICA is denied, or the readiness samples repeatedly hit the same current replica, this stateless execution can still land on a stale replica and successfully finish a chunk with writtenRows === 0. The full-backfill case explicitly retries that outcome, but this resume case—and the analogous replayFailed case below—does not, so settledTargetRows() times out below 2,000 and the main-only ObsessionDB job can still fail intermittently. Apply the same new-plan-id empty-chunk replay logic to these paths.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".