Skip to content

feat(live-messages): follow PostgreSQL, Redis and Kafka messages live - #3981

Merged
datlechin merged 5 commits into
TableProApp:mainfrom
J2TeamNNL:feat/live-messages
Nov 1, 2026
Merged

datlechin merged 5 commits into
TableProApp:mainfrom
J2TeamNNL:feat/live-messages

Conversation

@J2TeamNNL

@J2TeamNNL J2TeamNNL commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • New module Live Messages, off by default (id liveMessages). With it on, Database > Live Messages… opens a sheet that takes channels or topics, then a Live tab that lists messages as they arrive: time, channel or topic and payload, plus pattern on Redis and partition, offset and key on Kafka. With it off, no menu item, observer or connection is left behind.
  • The tab has Pause and Resume, Clear, a filter field, Copy (selected rows, or all, as tab-separated text, also Command-C), Export as CSV, and a field to send a test message (NOTIFY, PUBLISH, or a produce on Kafka).
  • The tab keeps the newest 10,000 messages. Pause freezes the list only, so messages keep arriving and resuming loses nothing up to the cap.
  • A subscription always uses connections of its own, opened like the routine debugger's: never the session's query connection, never pooled, no statement timeout. Closing the tab, turning the module off or quitting closes them.
  • A dropped connection adds a "Connection lost" row and subscribes again with a backoff of 1 s doubling to 30 s. A subscribe that fails on the first try (a wrong topic, say) ends the tab with the error instead of retrying.
  • PluginKit: a new optional protocol PluginLiveMessaging with PluginLiveSubscription and PluginLiveSessionSource. It is a separate protocol, like PluginRoutineDebugging, so it adds no requirement to existing drivers. No kit bump: main is already at 41, one past the 40 that v0.84.0-beta.1 shipped.
  • PostgreSQL: runs LISTEN per channel (quoted as an identifier), then reads PQnotifies after polling the socket in 250 ms slices on the connection's own queue. NOTIFY goes through pg_notify($1, $2). PGlite opts out, since a second session opens a different database. Redshift and CockroachDB do not conform.
  • Redis: SUBSCRIBE and PSUBSCRIBE on a dedicated connection, read by polling the socket and draining hiredis's reader, never through a hiredis read timeout, which would leave the context unusable. PUBLISH opens a second connection on first use, since a subscribed connection refuses it. Cluster subscribes on one node, which gets every classic PUBLISH.
  • Kafka: lists each partition's end offset, then fetches forward per partition with plain Fetch requests and no consumer group, so no committed offset moves. A non-UTF-8 key or payload reads as hex, as in the topic grid.

Defaults I picked: newest rows at the top, so a new message is in sight without scrolling; on Kafka, a reconnect starts at the end again and the gap row marks it.

Docs: TableProApp/docs#157 (draft, merges after this one).

Test plan

  • New tests: LiveMessageBufferTests (cap, ids, gap, filter), LiveMessageControllerTests (pause keeps messages, a drop leaves one gap row and resubscribes after the backoff, a first failure ends the tab, stop closes the subscription), LiveMessageExportTests, LiveMessagesModuleResidueTests (no menu item while off, toggling adds and removes it), PostgreSQLListenStatementsTests, RedisLivePushTests.
  • Live, against PostgreSQL 17, Redis 7 and Kafka 3.9 in Docker over an SSH tunnel, by loading the built plugin bundles in a small harness that calls subscribe with a session source of fresh drivers:
    • PostgreSQL: a NOTIFY from psql arrived about 200 ms after it was sent, a channel named Audit Log worked, and pg_terminate_backend ended the stream with an error, which the tab answers with a gap row and a resubscribe.
    • Redis: PUBLISH from redis-cli to a channel and to a PSUBSCRIBE pattern both arrived within 100 ms, the test PUBLISH went out on a second connection, and CLIENT KILL TYPE pubsub ended the stream with an error.
    • Kafka, a 2-partition topic: the 3 messages produced before the follow were skipped, new ones arrived with partition, offset and key, and kafka-consumer-groups --list was empty before and after.
  • Not checked here: the Live tab on screen. This Mac's shell has no Screen Recording permission, so there are no screenshots. verify.sh plugins was not run locally (low disk); the Kafka plugin was built on its own.

Fixes #3614

@datlechin
datlechin merged commit 3c7c801 into TableProApp:main Nov 1, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Live message view for PostgreSQL LISTEN/NOTIFY, Redis pub/sub and Kafka topics

2 participants