Job Queue System

A distributed job queue system built as a pnpm monorepo. Jobs are created via a REST API, persisted in PostgreSQL, queued in Redis via BullMQ, and processed asynchronously by a worker service. All services are containerized with Docker.

Architecture

                    ┌─────────────────┐
                    │   API Gateway   │  :3000
                    │  (rate limit,   │
                    │   CORS, morgan) │
                    └────────┬────────┘
                             │ reverse proxy
               ┌─────────────┴──────────────┐
               │                            │
               ▼                            ▼
    ┌─────────────────┐           ┌──────────────────┐
    │  Auth Service   │  :3002    │   Job Service    │  :3001
    │  POST /register │           │   POST /jobs     │
    │  POST /login    │           │   GET  /jobs     │
    │  → JWT token    │           │   GET  /jobs/:id │
    └─────────────────┘           │   DELETE /jobs/:id│
                                  └────────┬─────────┘
                                           │ enqueue
                                           ▼
                                    ┌─────────────┐
                                    │    Redis    │
                                    │   (BullMQ)  │
                                    └──────┬──────┘
                                           │ consume
                                           ▼
                                    ┌─────────────┐
                                    │   Worker    │
                                    │  (BullMQ    │
                                    │  processor) │
                                    └─────────────┘
                                           │
                                           ▼
                                    ┌─────────────┐
                                    │  PostgreSQL  │
                                    │  (Prisma)   │
                                    └─────────────┘

Monorepo Structure

apps/
  api-gateway/   — reverse proxy with rate limiting, CORS, logging (port 3000)
  auth-service/  — JWT authentication: register + login (port 3002)
  job-service/   — REST API for creating and managing jobs (port 3001)
  worker/        — BullMQ processor that executes queued jobs
packages/
  database/      — shared Prisma client (@jqs/database)
  common/        — shared types, pipes, and constants (@jqs/common)

Tech Stack

Layer	Technology
Framework	NestJS
Queue	BullMQ + Redis
Database	PostgreSQL + Prisma
Auth	JWT (HS256)
Validation	Zod
API Docs	Swagger / OpenAPI
Containerization	Docker + Docker Compose
Package manager	pnpm workspaces

Data Model

User — email/password authentication, owns jobs

Job — tracks the full lifecycle of a queued task:

• status: PENDING → RUNNING → COMPLETED / FAILED / RETRYING
• priority: LOW / NORMAL / HIGH
• runAt: schedule a job for future execution
• attempts / maxAttempts: automatic retry with exponential backoff
• payload: arbitrary JSON passed to the worker

JobLog — structured log entries attached to a job

Getting Started

Prerequisites

• Docker + Docker Compose
• Node.js 20+ and pnpm (for local development)

Run with Docker

# Copy environment file and adjust values if needed
cp .env.example .env   # or edit .env directly

# Build images and start all services
docker compose up --build

The migrator service runs prisma migrate deploy before any other service comes up. It runs on every docker compose up, not just the first — on subsequent starts it simply finds no pending migrations and exits 0.

Services available after startup:

Service	URL
API Gateway	http://localhost:3000
Auth Service	http://localhost:3002
Job Service	http://localhost:3001
Job Service Swagger	http://localhost:3001/docs
Auth Service Swagger	http://localhost:3002/docs

The direct service ports (3001, 3002) and their Swagger docs are exposed for local development only — see Known Limitations.

Local Development

# Install dependencies, then generate + build the shared @jqs/database package
pnpm install
pnpm db:generate
pnpm --filter @jqs/database build

# Start infrastructure (postgres + redis)
docker compose up postgres redis -d

# Set DATABASE_URL in apps/*/. env files to use localhost instead of Docker DNS
# Then start all services with hot reload
pnpm dev

Environment Variables

The root .env is read by Docker Compose. Copy and adjust as needed:

POSTGRES_USER=jqs
POSTGRES_PASSWORD=jqspw
POSTGRES_DB=jqsdb
JWT_SECRET=dev-only-change-me-openssl-rand-base64-32

Each service also reads its own .env for local development (not used in Docker).

API Overview

All job endpoints require a Bearer token from the login response.

Auth

POST /auth/register   { email, password }         → 201
POST /auth/login      { email, password }         → { accessToken }

Jobs

POST   /jobs          { name, payload, priority?, runAt?, maxAttempts? }  → Job
GET    /jobs                                                               → Job[]
GET    /jobs/:id                                                           → Job + logs
DELETE /jobs/:id                                                           → Job

Example

# Register and login
curl -s -X POST http://localhost:3000/auth/register \
  -H 'Content-Type: application/json' \
  -d '{"email":"test@example.com","password":"password123"}'

TOKEN=$(curl -s -X POST http://localhost:3000/auth/login \
  -H 'Content-Type: application/json' \
  -d '{"email":"test@example.com","password":"password123"}' \
  | grep -o '"accessToken":"[^"]*' | cut -d'"' -f4)

# Create a job
curl -s -X POST http://localhost:3000/jobs \
  -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"name":"send-email","payload":{"to":"a@b.de"},"priority":"HIGH"}'

A Postman collection is included at postman-collection.json — the login request automatically saves the token to a collection variable.

Useful Commands

pnpm install              # install all workspace dependencies
pnpm dev                  # start all services with hot reload (parallel)
pnpm build                # build all services

pnpm db:generate          # regenerate Prisma client after schema changes
pnpm db:migrate           # run migrations (dev mode, interactive)
pnpm db:studio            # open Prisma Studio

docker compose up --build         # rebuild and start everything
docker compose up postgres redis  # start infrastructure only
docker compose logs -f worker     # follow worker logs

Job Flow

1. Client sends POST /jobs with a JWT token to the gateway
2. Gateway proxies to job-service, which validates the token
3. job-service creates a Job record in PostgreSQL (PENDING) before enqueuing it in Redis — deliberately in that order: if the enqueue fails, the row is still there and recoverable. The reverse order would leave a queued job whose data was never persisted — unrecoverable, since the payload only existed in the request
4. Worker picks up the job, updates status to RUNNING, executes it
5. On success: status → COMPLETED
6. On failure with retries remaining: status → RETRYING, BullMQ retries with exponential backoff
7. On final failure: status → FAILED

All state transitions are logged in JobLog.

Known Limitations

This project prioritizes demonstrating queue mechanics and service boundaries over production hardening. Known gaps:

• Dual write without a transaction. The DB insert and the Redis enqueue (step 3 above) are two separate systems — a crash between them leaves an orphaned PENDING job that's never enqueued, and there's no reconciliation mechanism to detect or recover it. A periodic reconciliation job (re-enqueue stale PENDING rows) or a transactional outbox pattern would close this gap.
• At-least-once delivery without idempotency. BullMQ retries mean a job can execute more than once — e.g. the worker finishes the actual work but dies before the COMPLETED status update, BullMQ marks the job stalled, and it gets picked up and re-executed. Real handlers would need an idempotency key to make repeated execution safe.
• Auth is duplicated across services. Each service runs its own JWT guard rather than relying solely on the gateway, because the service ports (3001, 3002) are directly reachable and not just proxied. For production this means closing the direct ports and/or extracting the guard into a shared package.
• No refresh token. The access token expires after 7 days with no rotation mechanism.
• DELETE /jobs/:id doesn't remove the job from the queue. It only deletes the Postgres row; a PENDING/RETRYING job already sitting in BullMQ/Redis still gets picked up later (the worker then just logs a warning and skips it, since the DB row is gone — no crash, but wasted processing). Fixing this also requires passing an explicit jobId: job.id option to jobQueue.add(...) in create() — right now BullMQ assigns its own internal id, decoupled from the Postgres job.id, so there's no cheap way to look the queued job back up by id to remove it.
• All services share one Postgres user. migrator, auth-service, job-service, and worker all connect as the same POSTGRES_USER superuser — no least-privilege separation (e.g. worker can DELETE from User, auth-service can touch Job). Fixing this needs at least two roles: a migration owner with full DDL rights, and per-service runtime roles with GRANTs scoped to what each service actually touches — those grants aren't managed by prisma migrate itself and would need separate upkeep.
• BullMQ version is pinned. attemptsMade semantics differ between BullMQ v4 and v5, so the dependency is pinned rather than left on a floating range.

Note: The worker simulates job execution — it does not perform any real work (no emails sent, no reports generated, etc.). The name and payload fields are logged and the job is marked as completed after a short artificial delay. This project demonstrates the infrastructure and queue mechanics, not domain-specific job logic.