Skip to content

fix(pglite-sync): don't filter move-in messages by LSN#883

Merged
msfstef merged 4 commits into
electric-sql:mainfrom
jbingen:fix/move-in-lsn-filtering
Jul 1, 2026
Merged

fix(pglite-sync): don't filter move-in messages by LSN#883
msfstef merged 4 commits into
electric-sql:mainfrom
jbingen:fix/move-in-lsn-filtering

Conversation

@jbingen

@jbingen jbingen commented Jan 25, 2026

Copy link
Copy Markdown
Contributor

Summary

Move-in messages from Electric's tagged_subqueries feature don't have an lsn header because they come from direct database queries, not from the PostgreSQL replication stream. Previously, these messages were incorrectly skipped as "already seen" because the missing LSN defaulted to 0.

This fix checks for the is_move_in header and bypasses LSN filtering for move-in messages, ensuring that rows moving into a shape due to subquery condition changes are properly synced to the client.

Problem

When using shapes with subqueries (e.g., WHERE id IN (SELECT user_id FROM workspace_members WHERE workspace_id = $1)), new rows that "move in" to match the subquery weren't being synced:

  1. Electric server correctly processes the move-in and sends an INSERT message
  2. The INSERT message has headers.is_move_in: true but NO headers.lsn (since it's from a query, not replication)
  3. pglite-sync defaults missing LSN to 0
  4. The check lsn <= lastCommittedLsnForShape evaluates to 0 <= X which is true after initial sync
  5. The move-in message is skipped as "already seen"

Solution

const isMoveIn = (message.headers as Record<string, unknown>).is_move_in === true

if (!isMoveIn && lsn <= lastCommittedLsnForShape) {
  // Skip only if NOT a move-in message
  return
}

Test case

  1. Create a shape with a subquery WHERE clause (e.g., profiles where id IN (SELECT user_id FROM workspace_members WHERE workspace_id = ...))
  2. Complete initial sync
  3. Add a new row to the inner table that causes a row to "move in" to the outer shape
  4. Before fix: the moved-in row is not synced to PGlite
  5. After fix: the moved-in row is correctly synced

Move-in messages from Electric's tagged_subqueries feature don't have
an LSN header because they come from direct DB queries, not replication.
Previously these messages were incorrectly skipped as "already seen"
because the missing LSN defaulted to 0.

This checks for the is_move_in header and bypasses LSN filtering for
move-in messages, ensuring rows moving into a shape due to subquery
condition changes are properly synced.

Fixes electric-sql/electric#3769
Move-in data from tagged_subqueries can overlap with initial sync data,
causing duplicate key errors. This adds ON CONFLICT DO UPDATE handling
specifically for move-in inserts.

- Add primaryKey param to applyInsertsToTable
- Use ON CONFLICT DO UPDATE for move-in inserts
- Update changeset description
@jbingen

jbingen commented Jan 25, 2026

Copy link
Copy Markdown
Contributor Author

Added a second commit to handle duplicate key errors that can occur when move-in data overlaps with initial sync data.

When a row "moves in" to a shape, the server sends an INSERT. But if that row was already synced during initial sync, we get a duplicate key error.

For move-in inserts specifically, use ON CONFLICT DO UPDATE to upsert instead of plain INSERT. This only applies when is_move_in: true - regular inserts still use plain INSERT to catch any unexpected duplicates as errors.

@leoSouthwick-KE

Copy link
Copy Markdown

This would be really valuable for our team. We're using PGlite with Electric sync and this implementation would enable a workflow we've been waiting for.

We'll be trying out @jbingen's branch internally. Thanks for the work on this PR!

@msfstef

msfstef commented Jun 30, 2026

Copy link
Copy Markdown
Collaborator

This looks good @jbingen, thank you for the contribution!

To clarify: this fixes move-ins under subquery mode, but doesn't address move-outs (which need tag handling/matching etc.). That's fine as an incremental step toward subquery support — just want to be clear it isn't full support yet, since we don't handle move-outs, stale rows will be left in the db after one.

Happy to merge — only thing I'd want first is a basic test. Should be easy with the mock stream in sync.test.ts: an is_move_in insert with no lsn after initial sync, and one for the ON CONFLICT overlap.

@msfstef msfstef merged commit af37b02 into electric-sql:main Jul 1, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants