fix(pglite-sync): don't filter move-in messages by LSN#883
Conversation
Move-in messages from Electric's tagged_subqueries feature don't have an LSN header because they come from direct DB queries, not replication. Previously these messages were incorrectly skipped as "already seen" because the missing LSN defaulted to 0. This checks for the is_move_in header and bypasses LSN filtering for move-in messages, ensuring rows moving into a shape due to subquery condition changes are properly synced. Fixes electric-sql/electric#3769
Move-in data from tagged_subqueries can overlap with initial sync data, causing duplicate key errors. This adds ON CONFLICT DO UPDATE handling specifically for move-in inserts. - Add primaryKey param to applyInsertsToTable - Use ON CONFLICT DO UPDATE for move-in inserts - Update changeset description
|
Added a second commit to handle duplicate key errors that can occur when move-in data overlaps with initial sync data. When a row "moves in" to a shape, the server sends an INSERT. But if that row was already synced during initial sync, we get a duplicate key error. For move-in inserts specifically, use |
|
This would be really valuable for our team. We're using PGlite with Electric sync and this implementation would enable a workflow we've been waiting for. We'll be trying out @jbingen's branch internally. Thanks for the work on this PR! |
|
This looks good @jbingen, thank you for the contribution! To clarify: this fixes move-ins under subquery mode, but doesn't address move-outs (which need tag handling/matching etc.). That's fine as an incremental step toward subquery support — just want to be clear it isn't full support yet, since we don't handle move-outs, stale rows will be left in the db after one. Happy to merge — only thing I'd want first is a basic test. Should be easy with the mock stream in |
Summary
Move-in messages from Electric's
tagged_subqueriesfeature don't have anlsnheader because they come from direct database queries, not from the PostgreSQL replication stream. Previously, these messages were incorrectly skipped as "already seen" because the missing LSN defaulted to 0.This fix checks for the
is_move_inheader and bypasses LSN filtering for move-in messages, ensuring that rows moving into a shape due to subquery condition changes are properly synced to the client.Problem
When using shapes with subqueries (e.g.,
WHERE id IN (SELECT user_id FROM workspace_members WHERE workspace_id = $1)), new rows that "move in" to match the subquery weren't being synced:headers.is_move_in: truebut NOheaders.lsn(since it's from a query, not replication)lsn <= lastCommittedLsnForShapeevaluates to0 <= Xwhich is true after initial syncSolution
Test case
id IN (SELECT user_id FROM workspace_members WHERE workspace_id = ...))