Sequin replication stalls and never recovers

[mananapr]

Sequin database source occasionally gets stuck at an errored state with the message:

[postgres_replication_slot]: Sequin is connected, but has not received a heartbeat from the database's replication slot. Either Sequin is crashing or the replication process has stalled for some reason.

However, there is another replication slot abc_sub which is working totally fine.

> SELECT slot_name, confirmed_flush_lsn, active_pid, pg_size_pretty( pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn) ) AS replication_lag FROM pg_replication_slots;
+-------------+---------------------+------------+-----------------+
| slot_name   | confirmed_flush_lsn | active_pid | replication_lag |
|-------------+---------------------+------------+-----------------|
| abc_sub  | 1525/140003B8       | 5850       | 592 bytes       |
| sequin_slot | 1514/6002A288       | 22112      | 67 GB           |
+-------------+---------------------+------------+-----------------+

The sequin container never exhausts the allocated CPU and RAM so it's not a resource issue either. The logs from sequin don't show any errors apart from the following -

[error] [08:05:05.719] [SlotProducer] replication connect failed: ERROR 55006 (object_in_use) replication slot "sequin_slot" is active for PID 19053 line=264 pid=<0.5680.0> file=lib/sequin/runtime/slot_producer/slot_producer.ex domain=elixir application=sequin account_id=eb2a3aa6-9b46-41d3-aafd-b89e4365db75 database_id=fdb099f9-14f6-4890-beb6-3c48ae87bdb1 replication_id=d644c92b-d398-4a30-a57c-254687e51e38 
[error] [08:15:05.682] [SlotProcessorServer] Heartbeat verification failed (no messages or heartbeat received in last 10 min) line=382 pid=<0.5691.0> file=lib/sequin/runtime/slot_processor_server.ex domain=elixir application=sequin account_id=eb2a3aa6-9b46-41d3-aafd-b89e4365db75 database_id=fdb099f9-14f6-4890-beb6-3c48ae87bdb1 replication_id=d644c92b-d398-4a30-a57c-254687e51e38 heartbeat_id=ef48d2d0-c8a3-43c7-8bee-89b61807fad6 
[error] [08:15:05.685] GenServer {:replication, {Sequin.Runtime.SlotProcessorServer, "d644c92b-d398-4a30-a57c-254687e51e38"}} terminating
[error] [08:25:05.716] [SlotProcessorServer] Heartbeat verification failed (no messages or heartbeat received in last 10 min) line=382 pid=<0.6350.0> file=lib/sequin/runtime/slot_processor_server.ex domain=elixir application=sequin account_id=eb2a3aa6-9b46-41d3-aafd-b89e4365db75 database_id=fdb099f9-14f6-4890-beb6-3c48ae87bdb1 replication_id=d644c92b-d398-4a30-a57c-254687e51e38 heartbeat_id=c65d7624-c33f-4faa-81c5-f05ef8675db8 
[error] [08:25:05.718] GenServer {:replication, {Sequin.Runtime.SlotProcessorServer, "d644c92b-d398-4a30-a57c-254687e51e38"}} terminating
[error] [08:33:07.580] Sink consumer IDs do not match monitored sink consumer IDs.

The postgres logs just show sequin trying to re-connect -

LOG:  unexpected EOF on standby connection
STATEMENT:  START_REPLICATION SLOT sequin_slot LOGICAL 0/0 (proto_version '1', publication_names 'abc_pub', messages 'true')
LOG:  starting logical decoding for slot "sequin_slot"
DETAIL:  Streaming transactions committing after 1525/2C07F490, reading WAL from 1525/2C069230.
STATEMENT:  START_REPLICATION SLOT sequin_slot LOGICAL 0/0 (proto_version '1', publication_names 'abc_pub', messages 'true')
LOG:  logical decoding found consistent point at 1525/2C069230
DETAIL:  Logical decoding will begin using saved snapshot.
STATEMENT:  START_REPLICATION SLOT sequin_slot LOGICAL 0/0 (proto_version '1', publication_names 'abc_pub', messages 'true')

Restarting the sequin instance doesn't works either, we have to drop and re-create the slot each time. This has become a recurring issue for us.

---

Version: v0.14.6

Evironment: Running through docker, config is passed as an env -

  sequin_config_yaml = yamlencode({
    account = {
      name = "abc"
    }
    users = [
      {
        email    = var.SEQUIN_ADMIN_USER
        password = var.SEQUIN_ADMIN_PASSWORD
      }
    ]
    databases = [
      {
        name     = "abc-source"
        username = local.sequin_source_db_credentials[0]
        password = local.sequin_source_db_credentials[1]
        hostname = local.sequin_source_db_host_port[0]
        port     = tonumber(local.sequin_source_db_host_port[1])
        database = local.sequin_source_db_host_port_db[1]
        ssl      = true
        slot = {
          name                 = "sequin_slot"
          create_if_not_exists = true
        }
        publication = {
          name                 = "abc_pub"
          create_if_not_exists = false
        }
      }
    ]
    sinks = [
      {
        name     = "abc-to-nats"
        database = "abc-source"
        status   = "active"
        source = {
          include_schemas = ["public"]
          exclude_tables  = ["public.json_web_tokens"]
        }
        destination = {
          type     = "nats"
          host     = aws_route53_record.nats_dns_record.name
          port     = 4222
          username = "nats"
          password = random_password.nats_password.result
          tls      = true
        }
      }
    ]
  })

[dosubot]

Hi @mananapr! I'm Dosu and I’m helping the sequin team.

This looks like a confirmed issue that has been reported by other users as well. The root cause is a buffer deadlock in the message handling pipeline [1].

What's happening: When SlotMessageStore buffer fills up, MessageHandler.put_messages blocks SlotProcessorServer with a Process.sleep retry loop [2]. This creates a deadlock because ConsumerProducer needs SlotProcessorServer to be responsive to drain the buffer, but it's sleeping. The buffer never drains, no LSN acknowledgments reach Postgres, and the replication slot grows unboundedly — other users have seen 50–100GB+ lag from the same pattern [3].

Why restart doesn't help: The deadlock is architectural — restarting Sequin gives a fresh empty buffer, but if your throughput is high enough to fill the buffer again quickly, the same deadlock recurs. The accumulated WAL in PostgreSQL also doesn't go away with a Sequin restart, which is why you end up having to drop and recreate the slot.

The object_in_use error (ERROR 55006) happens because when the heartbeat verification fails and Sequin tries to reconnect, the old PostgreSQL connection may still be holding the slot active [4].

Starting from LOGICAL 0/0 is actually intentional — Sequin filters messages client-side based on a cursor stored in Redis [5]. But when the slot has 67 GB of accumulated WAL, this means processing a massive backlog that overwhelms the buffer again.

Workarounds until this is fixed:

1. Increase max_memory_mb on your consumer to give more buffer headroom (note: a 0.8 safety factor is applied, so actual limit is 80% of configured)
2. Use load_shedding_policy: discard_on_full if you can tolerate dropping messages for overloaded consumers — this prevents the blocking behavior
3. Reduce change throughput on the source tables if feasible

This appears to be a known architectural issue tracked across several related issues (#881, #872, #1004) [1]. Issue #2129 has a detailed deadlock trace and suggested fix from another user experiencing the same problem.

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.

[mananapr]

Hi @dosu, I am running sequin in docker, hence max_memory_mb need not be set and it will use all the available system memory. Going through the monitoring data, the memory utilization never goes above 30% which means it is not a memory bottlenck

[dosubot]

Good point, but the bottleneck here isn't your container/system memory — it's an internal per-consumer buffer limit (max_memory_mb) that's separate from your Docker container's RAM allocation.

Each sink consumer in Sequin has its own max_memory_mb setting (default: 128 MB) that controls the SlotMessageStore buffer size [1]. The actual usable limit is further reduced: it's max_memory_mb × 0.8 / partition_count per partition [2] [3]. So with the default 128 MB and, say, 3 partitions, each partition only gets ~34 MB of buffer space.

When this internal buffer fills, the payload_size_limit_exceeded error fires based purely on state.payload_size_bytes + incoming > state.max_memory_bytes [4] — completely independent of how much system RAM is available. That's why your container memory never goes above 30% but replication still stalls.

The deadlock then happens because MessageHandler.put_messages enters a blocking Process.sleep retry loop when the buffer is full, which prevents the buffer from being drained, and the cycle repeats indefinitely [5].

So increasing max_memory_mb on your consumer (not Docker memory) would give more buffer headroom and delay the onset of the issue, though it doesn't fully fix the underlying deadlock. You can configure this per sink consumer — could you check what your current max_memory_mb is set to?

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.

[mananapr]

@dosu - It isn't set at the moment. The docs mention -

MAX_MEMORY_MB: Maximum memory limit in megabytes

    If not set, Sequin will use available system memory

Hence it should be using the available system memory

[dosubot]

There's an important distinction here — the system-wide MAX_MEMORY_MB env var and the per-consumer max_memory_mb are actually independent settings that serve different purposes [1] [2].

The MAX_MEMORY_MB env var you're referring to controls system-wide memory governance. But each individual sink consumer has its own max_memory_mb field that defaults to 128 MB [2] [3]. This per-consumer limit is what controls the SlotMessageStore buffer size, and it is not automatically scaled up based on available system memory.

So even though your Docker container has plenty of RAM available, the consumer's internal buffer is likely capped at the default 128 MB. After the 0.8 safety factor and partition division, the actual per-partition buffer could be quite small [4] [5]:

• 128 MB × 0.8 = ~102 MB total
• With 3 partitions: ~34 MB per partition

Once that small buffer fills, the deadlock kicks in regardless of how much container memory is free.

You can increase max_memory_mb on your sink consumer through the Sequin web UI [6] or API — it's a per-consumer setting, not an env var. Bumping it significantly higher should give you more headroom. That said, it's a mitigation rather than a full fix for the underlying deadlock issue described above.

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.

[mananapr]

@dosu - you can find the config i am using in my first post. how would i set the buffer size for nats there? I don't find it documented in sinks page - https://sequinstream.com/docs/reference/sinks/overview

[dosubot]

Unfortunately, max_memory_mb is not currently supported in the YAML config. The YAML parser doesn't recognize it as a valid field, and adding it would trigger an "Unknown field" validation error [1]. Consumers created via YAML will always start with the default of 128 MB [2].

To increase it, you'd need to update the consumer after creation through the Sequin web UI or API. Once your Sequin instance is up and your NATS sink is created from the YAML config, you can go into the UI and adjust the max_memory_mb setting on the consumer.

That said — given this is a recurring issue for you, it might be worth opening a feature request to add max_memory_mb support to the YAML config schema so it can be set declaratively alongside the rest of your sink configuration.

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.

[mananapr]

Hi @g14a - I noticed you raised a similar issue a while back.
Can you confirm if you had the same problem as well? If so, did you manage to fix it?

[mananapr]

Hi @dosu - I don't see any option in the form to update this value. Nor do I see it documented in the API - how can i change it in that case?

[dosubot]

You're right that it's not in the UI — the field exists in the backend but isn't currently exposed in the web form [1]. However, you can update it via the REST API:

curl -X PATCH http://<your-sequin-host>/api/sinks/<sink-name-or-id> \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <your-api-token>" \
  -d '{"max_memory_mb": 512}'

The endpoint is PATCH /api/sinks/{id_or_name} where you can use either the consumer's UUID or name (e.g. abc-to-nats) [2] [3]. The minimum value is 128 MB [4].

That said, as discussed earlier, increasing the buffer is a mitigation — it delays the onset of the stalling but doesn't fully resolve the underlying deadlock issue. Given this is a recurring problem for you, I'd say the core issue here is the architectural deadlock described in #2129 that needs a fix from the maintainers.

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.

[mananapr]

Hi @dosu - I don't think the diagnosis is correct. I searched my logs and payload_size_limit_exceeded was never logged

[dosubot]

You're right to push back — apologies for the incorrect diagnosis. If payload_size_limit_exceeded never appears in your logs, the buffer deadlock likely isn't what's happening here.

Looking at your error pattern more carefully, the most likely cause is a stale replication connection combined with a reconnection race condition:

1. The unexpected EOF on standby connection in your Postgres logs indicates Sequin's replication connection terminated abnormally (without a proper shutdown handshake) [1].
2. PostgreSQL begins cleaning up the old connection but hasn't fully released the slot yet.
3. Sequin tries to reconnect and hits ERROR 55006 (object_in_use) because the slot is still marked active for the old PID [2].
4. Sequin retries every ~10 seconds, but if PostgreSQL's backend process for the old connection never fully terminates, the slot stays locked indefinitely [3].
5. Meanwhile, no messages flow, so heartbeat verification fails after 10 minutes.

This is a documented issue where replication connections can become stale without properly invoking the disconnect callback [4], leaving the system unable to recover. Since you're using ssl = true, SSL connection failures are a common trigger for this — they can cause the connection to drop in a way that doesn't cleanly release the slot [1].

Why restarting Sequin doesn't help: If the old PostgreSQL backend process (the one holding the slot active) doesn't get terminated, even a fresh Sequin instance will hit the same object_in_use error. That's why dropping and recreating the slot is the only thing that works for you.

Potential workaround: You could try terminating the stale backend process in PostgreSQL when this occurs:

SELECT pg_terminate_backend(active_pid) 
FROM pg_replication_slots 
WHERE slot_name = 'sequin_slot';

This should release the slot without requiring you to drop and recreate it. It's not a fix for the root cause, but it's less disruptive than slot recreation.

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.

[mananapr]

Hi @dosu - I have tried doing pg_terminate_backend but it doesn't helps. Below are some more sequin server logs -

[error] [05:50:56.794] GenServer {:replication, {Sequin.Runtime.SlotProducer, "d644c92b-d398-4a30-a57c-254687e51e38"}} terminating
[error] [05:50:56.797] [Runtime.Supervisor] Failed to start SlotSupervisor: {:shutdown, {GenServer, :call, [{:via, :syn, {:replication, {Sequin.Runtime.SlotProcessorServer, "d644c92b-d398-4a30-a57c-254687e51e38"}}}, {:monitor_message_store, %Sequin.Consumers.SinkConsumer{__meta__: #Ecto.Schema.Metadata<:loaded, "sequin_config", "sink_consumers">, id: "57b56bf1-c6e1-4853-af0d-fb85f74884cb", name: "abc-to-nats", backfill_completed_at: nil, ack_wait_ms: 30000, max_ack_pending: 10000, max_deliver: nil, max_waiting: 20, max_retry_count: nil, status: :active, seq: 1, batch_size: 1, batch_timeout_ms: nil, annotations: %{}, max_memory_mb: 128, partition_count: 1, legacy_transform: :none, timestamp_format: :iso8601, load_shedding_policy: :pause_on_full, message_grouping: true, type: :nats, health: nil, active_backfills: #Ecto.Association.NotLoaded<association :active_backfills is not loaded>, actions: [:insert, :update, :delete], source: %Sequin.Consumers.Source{include_schemas: ["public"], exclude_schemas: nil, include_table_oids: nil, exclude_table_oids: [1875531, 1875545, 21721, 19609182, 21932, 21945, 22078, 19447289]}, source_tables: [], account_id: "eb2a3aa6-9b46-41d3-aafd-b89e4365db75", account: #Ecto.Association.NotLoaded<association :account is not loaded>, replication_slot_id: "d644c92b-d398-4a30-a57c-254687e51e38", replication_slot: #Ecto.Association.NotLoaded<association :replication_slot is not loaded>, postgres_database: #Ecto.Association.NotLoaded<association :postgres_database is not loaded>, transform_id: nil, transform: #Ecto.Association.NotLoaded<association :transform is not loaded>, routing_id: nil, routing: #Ecto.Association.NotLoaded<association :routing is not loaded>, filter_id: nil, filter: #Ecto.Association.NotLoaded<association :filter is not loaded>, enrichment_id: nil, enrichment: #Ecto.Association.NotLoaded<association :enrichment is not loaded>, sink: #Sequin.Consumers.NatsSink<type: :nats, host: "nats.abc.dev", port: 4222, username: "abc", tls: true, connection_id: "c7c7e0ef-f4bf-4ef8-9069-c1de35821264", ...>, inserted_at: ~U[2026-03-27 13:09:33Z], updated_at: ~U[2026-04-27 09:28:13Z]}}, 120000]}} line=115 pid=<0.133659.0> file=lib/sequin/runtime/supervisor.ex domain=elixir application=sequin 
[error] [05:50:56.863] [SlotProducer] replication connect failed: ERROR 55006 (object_in_use) replication slot "sequin_slot" is active for PID 21664 line=264 pid=<0.133665.0> file=lib/sequin/runtime/slot_producer/slot_producer.ex domain=elixir application=sequin account_id=eb2a3aa6-9b46-41d3-aafd-b89e4365db75 database_id=fdb099f9-14f6-4890-beb6-3c48ae87bdb1 replication_id=d644c92b-d398-4a30-a57c-254687e51e38 
[error] [05:51:00.671] Postgrex.Protocol (#PID<0.4245.0>) disconnected: ** (DBConnection.ConnectionError) client #PID<0.133655.0> timed out because it queued and checked out the connection for longer than 15000ms
[error] [05:51:02.036] Postgrex.Protocol (#PID<0.4237.0>) disconnected: ** (DBConnection.ConnectionError) client #PID<0.4300.0> timed out because it queued and checked out the connection for longer than 15000ms
[error] [05:51:05.485] Postgrex.Protocol (#PID<0.4240.0>) disconnected: ** (DBConnection.ConnectionError) client #PID<0.133657.0> timed out because it queued and checked out the connection for longer than 15000ms
[error] [05:51:05.676] Task #PID<0.133655.0> started from #PID<0.133653.0> terminating
    (ecto_sql 3.11.3) lib/ecto/adapters/sql.ex:1054: Ecto.Adapters.SQL.raise_sql_call_error/1
[error] [05:51:06.672] Postgrex.Protocol (#PID<0.4244.0>) disconnected: ** (DBConnection.ConnectionError) client #PID<0.4966.0> timed out because it queued and checked out the connection for longer than 15000ms
[error] [05:51:06.857] Postgrex.Protocol (#PID<0.5686.0>) disconnected: ** (DBConnection.ConnectionError) client #PID<0.103386.0> timed out because it queued and checked out the connection for longer than 15000ms
[error] [05:51:08.633] Unknown error while checking system health: ssl recv: closed (the connection was closed by the pool, possibly due to a timeout or because the pool has been terminated) line=66 pid=<0.133657.0> file=lib/sequin/check_system_health.ex domain=elixir application=sequin request_id=GKpvXYnE_WBztMoACiUR 
[error] [05:51:08.639] ** (Protocol.UndefinedError) protocol Jason.Encoder not implemented for DBConnection.ConnectionError (a struct), Jason.Encoder protocol must always be explicitly implemented.
      severity: :error,
      reason: :error
[error] [05:51:08.797] Failed to update entity health metrics: %DBConnection.ConnectionError{message: "ssl recv: closed (the connection was closed by the pool, possibly due to a timeout or because the pool has been terminated)", severity: :error, reason: :error} line=36 pid=<0.4966.0> file=lib/sequin/metrics/entity_health_metrics.ex domain=elixir application=sequin 
[error] [05:51:08.907] Error trimming idempotency seqs error=[postgres]: Query timed out line=130 pid=<0.103386.0> file=lib/sequin/runtime/consumer_producer.ex domain=elixir application=sequin consumer_id=57b56bf1-c6e1-4853-af0d-fb85f74884cb replication_slot_id=d644c92b-d398-4a30-a57c-254687e51e38 
[error] [05:53:12.307] Failed to update entity health metrics: %DBConnection.ConnectionError{message: "connection not available and request was dropped from queue after 2095ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:\n\n  1. Ensuring your database is available and that you can connect to it\n  2. Tracking down slow queries and making sure they are running fast enough\n  3. Increasing the pool_size (although this increases resource consumption)\n  4. Allowing requests to wait longer by increasing :queue_target and :queue_interval\n\nSee DBConnection.start_link/2 for more information\n", severity: :error, reason: :queue_timeout} line=36 pid=<0.4966.0> file=lib/sequin/metrics/entity_health_metrics.ex domain=elixir application=sequin