|
| 1 | +# Trigger.dev Real-Time Service |
| 2 | + |
| 3 | +A high-performance Go service that provides real-time streaming of task run updates via Server-Sent Events (SSE) using PostgreSQL logical replication. |
| 4 | + |
| 5 | +## Features |
| 6 | + |
| 7 | +- **Low Latency**: p95 latency ≤ 300ms from WAL commit to client receive |
| 8 | +- **Scalable**: Supports 400k+ concurrent SSE connections |
| 9 | +- **Efficient**: Single PostgreSQL replication slot with REPLICA IDENTITY FULL |
| 10 | +- **Flexible Filtering**: Subscribe by run_id, env_id, tags, or time windows |
| 11 | +- **Resilient**: Automatic reconnection with exponential backoff |
| 12 | + |
| 13 | +## Architecture |
| 14 | + |
| 15 | +- **Single Process**: Vertical scaling approach with in-memory state |
| 16 | +- **Logical Replication**: Consumes PostgreSQL WAL via pgoutput format |
| 17 | +- **SSE Streaming**: HTTP/2 Server-Sent Events for real-time updates |
| 18 | +- **Memory Indexes**: Fast lookups by run_id, env_id, and tags |
| 19 | + |
| 20 | +## Configuration |
| 21 | + |
| 22 | +Environment variables: |
| 23 | + |
| 24 | +- `DATABASE_URL`: PostgreSQL connection string |
| 25 | +- `PORT`: HTTP server port (default: 8080) |
| 26 | +- `REPLICATION_SLOT`: Logical replication slot name |
| 27 | +- `PUBLICATION_NAME`: PostgreSQL publication name |
| 28 | + |
| 29 | +## API Endpoints |
| 30 | + |
| 31 | +### Stream Task Runs |
| 32 | + |
| 33 | +``` |
| 34 | +GET /v1/runs/stream?filter=<json> |
| 35 | +``` |
| 36 | + |
| 37 | +Filter examples: |
| 38 | +```json |
| 39 | +{ |
| 40 | + "run_id": "123e4567-e89b-12d3-a456-426614174000", |
| 41 | + "env_id": "123e4567-e89b-12d3-a456-426614174001", |
| 42 | + "tags": ["tag1", "tag2"], |
| 43 | + "created_at": "2025-06-01T00:00:00Z" |
| 44 | +} |
| 45 | +``` |
| 46 | + |
| 47 | +### Health Check |
| 48 | + |
| 49 | +``` |
| 50 | +GET /health |
| 51 | +``` |
| 52 | + |
| 53 | +## Event Types |
| 54 | + |
| 55 | +- `initial`: Full current state sent once per run on new stream |
| 56 | +- `delta`: Partial updates with changed fields |
| 57 | +- `keepalive`: Sent every 15 seconds to maintain connection |
| 58 | + |
| 59 | +## Client Protocol |
| 60 | + |
| 61 | +- **Headers**: `Accept: text/event-stream`, `Last-Event-Id` for replay |
| 62 | +- **Reconnection**: Exponential backoff with jitter |
| 63 | +- **Back-pressure**: Connections dropped if write buffer > 64KB |
| 64 | + |
| 65 | +## Performance Targets |
| 66 | + |
| 67 | +- **Latency**: p95 ≤ 300ms from WAL to client |
| 68 | +- **Capacity**: 400k concurrent connections |
| 69 | +- **Memory**: ≤ 3KB per connection + 200B per run |
| 70 | +- **Cost**: ≤ $1000/month infrastructure |
| 71 | + |
| 72 | +## Deployment |
| 73 | + |
| 74 | +```bash |
| 75 | +# Build |
| 76 | +go build -o trigger-realtime-service . |
| 77 | + |
| 78 | +# Run |
| 79 | +./trigger-realtime-service |
| 80 | + |
| 81 | +# Docker |
| 82 | +docker build -t trigger-realtime-service . |
| 83 | +docker run -p 8080:8080 trigger-realtime-service |
| 84 | +``` |
| 85 | + |
| 86 | +## Database Setup |
| 87 | + |
| 88 | +The service automatically creates the required PostgreSQL publication and replication slot: |
| 89 | + |
| 90 | +```sql |
| 91 | +-- Publication for task_run table |
| 92 | +CREATE PUBLICATION trigger_realtime_pub FOR TABLE task_run |
| 93 | +WITH (publish = 'insert,update,delete'); |
| 94 | + |
| 95 | +-- Set replica identity to include full row data |
| 96 | +ALTER TABLE task_run REPLICA IDENTITY FULL; |
| 97 | + |
| 98 | +-- Replication slot (created automatically) |
| 99 | +SELECT pg_create_logical_replication_slot('trigger_realtime_slot', 'pgoutput'); |
| 100 | +``` |
| 101 | + |
| 102 | +## Monitoring |
| 103 | + |
| 104 | +- Health endpoint provides service status and warmup state |
| 105 | +- Logs include replication lag and connection metrics |
| 106 | +- Built-in keepalive prevents connection timeouts |
| 107 | + |
| 108 | +## Integration |
| 109 | + |
| 110 | +This service is designed to integrate with the existing Trigger.dev platform: |
| 111 | + |
| 112 | +- Replaces Electric SQL for real-time updates |
| 113 | +- Compatible with existing SDK subscription patterns |
| 114 | +- Maintains the same client-side API surface |
| 115 | +- Provides better performance and lower operational overhead |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?v=4&size=40){kind=avatar}
0 commit comments