feat: Work queue core — leased work items, with token cleanup and retention as periodic kinds #278

Parent

#276

What to build

The durable work queue, end to end, with maintenance as its first callers. Refresh-token cleanup and the queue's own retention purge stop being private timers. Every worker schedules them as unscoped periodic work items. A worker claims one, runs it under a lease, and records the outcome in WorkItems. A worker that dies, pauses or shuts down never leaves an item stuck in Running.

Everything follows #276's Implementation Decisions. In short:

• Table: the WorkItems entity, with Fluent configuration, HasComment, a migration and the regenerated docs/schema.sql.
• Producing: IWorkQueue.EnqueueAsync is the only insert path, with opt-in DedupKey coalescing.
• Store:
  • a claim over FOR UPDATE SKIP LOCKED that increments Attempt;
  • a 2 s heartbeat that extends the 60 s lease and writes the progress snapshot, fenced by Id, Attempt and Running;
  • lease recovery with the 3-stall limit;
  • release on graceful shutdown without a stall;
  • Failed with the error on a handler exception, with no retry.
• Executor: one item at a time per worker, first in, first out.
• Scheduler: one IWorkSchedule per periodic kind; due-ness derived from the table, so overdue work runs right after downtime.
• Kinds: a WorkKind enum, IWorkPayload with a static abstract kind, IWorkHandler<TPayload> in the Worker, and a registry that fails startup unless every kind has exactly one payload and exactly one handler. No switch on the kind.
• First kinds: TokenCleanup replaces TokenCleanupService's timer, and WorkItemRetention deletes finished items older than the retention period.
• Settings: poll 5 s, heartbeat 2 s, lease 60 s, 3 stalls, retention 30 days, token cleanup 24 h, retention purge 24 h.

Scope routing by storage config is not in this ticket. Both kinds here are unscoped; routing ships with its first caller, the crawl ticket.

Acceptance criteria

• Token cleanup runs as a TokenCleanup work item on its interval, and TokenCleanupService's own timer is gone.
• Finished items older than the retention period are deleted by a WorkItemRetention item.
• Two workers that find the same periodic item due produce one queued item; the second insert is absorbed.
• Contract tests on real PostgreSQL (integration project):
  • concurrent claimers never get the same item;
  • after a reclaim, the old attempt's heartbeat and completion affect no rows;
  • an expired lease puts the item back in Queued with a stall, and the 3rd stall fails it as stalled;
  • DedupKey coalesces queued items only.
• Executor and scheduler unit tests with a substituted store and FakeTimeProvider:
  • the heartbeat cadence;
  • a fenced heartbeat (0 rows) cancels the handler;
  • shutdown releases the item without a stall;
  • an exception fails the item;
  • only one item runs at a time;
  • due-ness is computed from the last finished item.
• Both kinds have a test that a second run converges. The existing token cleanup tests move onto its handler.
• Startup fails for a kind without a payload or a handler, or with two of either.
• Architecture rules: only IWorkQueue inserts work items, and handlers live only in the Worker. Each is shown failing on a deliberate violation.
• An ADR records the hand-rolled PostgreSQL work queue over a library, citing the evaluation on feat: Work queue — API requests Worker jobs, admin sees queued and running work #273.
• PostgreSQL-only store members carry // Stryker disable all with ADR 0005 rows; everything else reaches 100% mutation score.
• CLAUDE.md describes the work queue, and the Worker section no longer describes the token cleanup timer.

First caller

The TokenCleanup and WorkItemRetention kinds.

Design

Decided with the maintainer on 2026-10-18 (#276, Implementation Decisions). The library evaluation is in #273.