spec: Work queue — durable API/Worker work items with scope routing, leases and a live admin view #276

Problem Statement

As the admin of an Anichron install, I cannot ask the Worker to do anything, and I cannot see what it is doing.

• The API and the Worker share only the database. The Worker finds work only by crawling on its own timer, so a newly added storage config waits for the next crawl, up to 4 h.
• The Worker's queue is an in-memory, per-process channel. Nobody can see it, and it is lost on restart.
• Maintenance runs on separate timers, and nothing records when it last ran or whether it failed.
• Upcoming features need the API to request work that only the Worker can do, because only the Worker has the proxy mount read-write. The first is the orphaned-proxy sweep and selected deletion (feat(worker): Orphaned proxy file sweeper (proxies ↔ DB reconciliation) #174).
• A future admin view in Anichron.UI should show what is queued, what is running with its progress, and what finished or failed.

Solution

A work queue: a durable WorkItems table that both hosts use through one enqueue path.

• Producers:
  • The API enqueues admin requests through per-kind endpoints.
  • Every Worker schedules the periodic work (crawls and maintenance) for the scopes it serves. "Due" is derived from the table itself, so no host holds timer state.
• Claiming: a Worker claims items it is allowed to serve (scope routing by storage config), runs one at a time, and keeps a lease alive with a heartbeat.
• Progress: the heartbeat also carries a progress snapshot and picks up cancel requests.
• Failure handling:
  • a dead or paused Worker is fenced off and its item recovered;
  • a graceful shutdown hands the item back;
  • a handler exception fails the item with its error.
• Admin view: the admin lists items, follows them live over Server-Sent Events, and cancels queued or running ones.

The first callers are the two maintenance jobs: refresh-token cleanup and the queue's own retention purge. The crawl then moves onto the queue, one item per storage config. #174 builds on the queue afterwards with its sweep and deletion kinds.

User Stories

Admin: seeing work

1. As an admin, I want to list work items filtered by status and kind, paged, so that I can see what is queued, running, finished and failed.
2. As an admin, I want each item to show its kind, scope (storage config), requester (or "scheduled"), timestamps, the claiming worker and its attempt, so that I know who asked for what and where it ran.
3. As an admin, I want a running item to show a progress snapshot, done of total with a short note, so that the UI can draw a progress bar.
4. As an admin, I want a running item whose total is unknown, such as a directory walk, to show its done count and note, so that the UI can draw an indeterminate bar.
5. As an admin, I want live updates over Server-Sent Events, so that the view follows progress and status changes without reloading.
6. As an admin, I want a page reload to re-read the current state and resubscribe, so that nothing is lost when I reload.
7. As an admin, I want the stream to resume from the last event after a dropped connection, so that I don't miss changes.
8. As an admin, I want a succeeded item to show its full results summary, so that I can see what it did.
9. As an admin, I want a cancelled item to show its partial results, clearly marked as partial, so that the work done before the cancel isn't lost.
10. As an admin, I want a failed item to show its error and its last progress snapshot, but no partial results, so that an incomplete result cannot pass for a complete one.
11. As an admin, I want an item whose Worker died to come back to the queue on its own, and to fail as "stalled" after 3 such recoveries, so that a crash never leaves an item "running" forever and an item that keeps killing workers stops.
12. As an admin, I want to see every user's items, showing only config ids and counts and never media, so that I can manage the install without gaining media access.

Admin: controlling work

13. As an admin, I want to cancel a queued item and have it end as Cancelled immediately, so that unwanted work never starts.
14. As an admin, I want to cancel a running item and have its handler stop at the next heartbeat, so that a long job can be stopped.
15. As an admin, I want cancelling an already finished item to answer with a conflict, so that the UI can tell me why nothing happened.
16. As an admin, I want a request for a coalescing kind that is already waiting to return the waiting item rather than create a second, so that double clicks and duplicate schedules never queue the same work twice.
17. As an admin, I want requests whose payloads differ, such as two deletions of different files, to stay separate items, so that no request is silently dropped.

Operator: deployment

18. As an operator, I want the existing deployment shapes (per-user workers and the all-user worker) to keep working unchanged, so that the queue needs no new containers and no Docker socket.
19. As an operator running per-user workers, I want each worker to claim only items scoped to its own user's storage configs, plus unscoped items, so that a worker never touches a folder it hasn't mounted.
20. As an operator running an all-user worker, I want it to claim every item, so that one container serves everyone.
21. As an operator, I want crawling and maintenance to keep running while the API is down, so that the API is not a single point of failure for background work.
22. As an operator, I want periodic work to run straight away after downtime because it is overdue, so that restarts don't push a crawl out by a full interval.
23. As an operator, I want a deploy or restart to hand a running item back to the queue without counting as a stall, so that routine deploys during a long crawl never fail it.
24. As an operator, I want every interval and timeout configurable, with the defaults below, so that I can tune the install without code changes.
25. As an operator, I want finished items and their result rows purged after 30 days, so that the table and the result tables don't grow without bound.
26. As an operator, I want items of a deleted storage config to disappear with it, so that no work is attempted for a config that no longer exists.

Developer: adding work

27. As a developer, I want to add a kind by adding one enum value, a payload record, a handler and its DI registration, so that the queue, claimer, scheduler and admin list never change for a new kind.
28. As a developer, I want startup to fail when a kind lacks exactly one payload and exactly one handler, so that a missing handler is caught at deploy, not when an item is claimed.
29. As a developer, I want a periodic kind to declare its schedule through its own schedule class, so that the scheduler never switches on the kind.
30. As a developer, I want my handler to receive the typed payload, a progress sink and a cancellation token, so that it needs no knowledge of leases, heartbeats or fencing.
31. As a developer, I want a documented handler contract (each run starts from scratch and converges, and side effects are safe to repeat), so that at-least-once delivery is safe.
32. As a developer, I want large, selectable results in a kind-specific table tagged with the item and attempt, so that readers see only the attempt that finished and a zombie attempt's writes stay invisible.
33. As a developer, I want a kind to opt into coalescing by providing a dedup key, so that coalescing is a per-kind choice.
34. As a developer, I want architecture rules that keep insertion in the enqueue path, handlers in the Worker, and the API to cancel-only writes, so that write ownership cannot drift.

Crawl migration

35. As an admin, I want each storage config's crawl to appear as its own work item with progress (files found, processed, failed), so that I can see which library is being crawled and how far along it is.
36. As an operator, I want a crawl to be due per config at the configured crawl interval, so that crawl timing behaves as today.
37. As an operator, I want the crawl's per-file concurrency to stay as configured, so that ingestion throughput doesn't change.
38. As a developer, I want the Worker's own crawl timer and the token cleanup timer removed once their kinds run on the queue, so that there is one execution model.

Implementation Decisions

All decisions below were made by the maintainer in a grilling session on 2026-10-18. The libraries considered (Hangfire, TickerQ, Quartz.NET, Wolverine, Rebus and MassTransit) and why each was ruled out are in #273.

Hand-rolled, in our model

• One WorkItems entity in the EF model with Fluent configuration and HasComment, so it appears in docs/schema.sql. All times are NodaTime.
• Columns:
  • identity and request: Id, Kind (enum persisted as text), Payload (jsonb), StorageConfigId (nullable, cascade on config delete), DedupKey (nullable), RequestedBy (nullable, null when scheduled), RequestedAt;
  • execution: Status, Attempt, Stalls, ClaimedBy, StartedAt, LeaseUntil, FinishedAt, UpdatedAt, CancelRequestedAt (nullable);
  • progress and outcome: ProgressDone, ProgressTotal (nullable), ProgressNote, Result (jsonb summary), Error.
• Statuses are Queued, Running, Succeeded, Failed and Cancelled.
• Indexes:
  • a claim index on queued items in request order;
  • a partial unique index on DedupKey where the status is Queued and the key is not null;
  • an index serving lease recovery, on running items by LeaseUntil;
  • an index serving the admin list and the SSE re-query, by UpdatedAt.

Kinds and payloads

• A WorkKind enum. Each payload record in Core implements IWorkPayload and declares its kind as a static abstract member.
• Each handler in the Worker implements IWorkHandler<TPayload>.
• Payloads are serialised with System.Text.Json using NodaTime-aware settings.
• Dispatch goes through a registry built from DI, never a switch on the kind. At startup the registry checks that every WorkKind value has exactly one payload type and exactly one handler.
• This spec's kinds are TokenCleanup, WorkItemRetention and Crawl. feat(worker): Orphaned proxy file sweeper (proxies ↔ DB reconciliation) #174 adds the sweep and deletion kinds.

Producers

• IWorkQueue.EnqueueAsync (Core) is the only code that inserts an item.
• The API enqueues through per-kind endpoints, with typed payloads and validation. Admins cannot enqueue internal kinds.
• Every Worker runs the same scheduler for the scopes it serves:
  • a per-user worker: its user's configs, plus unscoped kinds;
  • an all-user worker: all configs, plus unscoped kinds.
• Due-ness: each periodic kind supplies an IWorkSchedule. An item is due when no item with the same dedup key is queued or running, and the last finished one is older than the interval. No timer state is kept, so overdue work runs right after downtime.
• Coalescing is opt-in through DedupKey:
  • periodic kinds set it, e.g. crawl:<config>, token-cleanup, work-item-retention;
  • when two schedulers race, the losing insert is absorbed by the partial unique index;
  • a coalesced enqueue returns the waiting item's id.

Execution (Worker)

• Routing: scope is the nullable StorageConfigId. A worker claims queued items whose config it serves, or that are unscoped.
• One item at a time per worker, first in, first out by RequestedAt. File-level concurrency inside a crawl is unchanged.
• Claiming: a single UPDATE over a FOR UPDATE SKIP LOCKED sub-select. It sets Running, ClaimedBy, StartedAt and the lease, and increments Attempt.
• Heartbeat every 2 s per running item: one UPDATE fenced by Id, Attempt and Running. It extends the lease (60 s), writes the latest progress snapshot (overwritten, no history) and UpdatedAt, and returns CancelRequestedAt.
  • 0 rows: the worker has been fenced off, so it cancels the handler and writes nothing more.
  • A cancel request: the worker cancels the handler, and the item ends Cancelled with partial results kept.
• Lease recovery, run by every worker's poll: a Running item with an expired lease goes back to Queued and Stalls increments. At 3 stalls it becomes Failed with a "stalled" error.
• Graceful shutdown releases the running item to Queued without incrementing Stalls.
• A handler exception marks the item Failed with the error. There is no automatic retry.
• All terminal transitions are fenced by Attempt.
• Handler contract:
  • each run starts from scratch and converges;
  • side effects are safe to repeat;
  • result rows are written tagged with (WorkItemId, Attempt), and readers read only the finished attempt's rows: Succeeded, or Cancelled marked partial. A Failed item shows no partial results.
• ClaimedBy is the worker's identity for display: its WORKER__USER value or all, plus the host name.

Maintenance as work items

• Refresh-token cleanup becomes the unscoped periodic kind TokenCleanup, still executed in the Worker (CLAUDE.md's existing exception for that one identity write stands). Its own timer is removed.
• WorkItemRetention is an unscoped periodic kind. It deletes finished items, and their result rows, older than the retention period.

Crawl migration

• Crawl is a periodic kind scoped to one storage config. Its handler runs today's crawl of that config through the existing ingestion pipeline and reports files found, processed and failed as progress.
• The Worker's own crawl timer is removed. Crawling on startup comes from due-ness.
• Per-file retry and failure tracking stay with feat(worker): Per-item retry with exponential backoff and persistent failure tracking #132.

Admin API (admin role only, under /api/v1/admin/work-items)

• List: filter by status and kind, paged.
• Detail by id.
• SSE stream: every 2 s it re-queries items with UpdatedAt newer than the last event and pushes each changed item. Event ids allow Last-Event-ID resumption. The UI consumes it with fetch streaming so the bearer token is sent as a header.
• Cancel:
  • a queued item becomes Cancelled atomically, guarded by status;
  • a running item gets CancelRequestedAt;
  • a finished item returns 409;
  • an unknown id returns 404.
• Enqueue endpoints are per kind and are added with the kinds that need them. None of this spec's kinds is admin-enqueueable; feat(worker): Orphaned proxy file sweeper (proxies ↔ DB reconciliation) #174 adds the first.

Write ownership, enforced by architecture rules

• Only IWorkQueue inserts work items.
• The API's only other write is cancel.
• Every other state change belongs to the Worker's executor.
• Handlers live only in the Worker.

Settings, all configurable:

ADR: the first ticket records "hand-rolled PostgreSQL work queue over a library" as an ADR in docs/adr/, with the library evaluation from #273.

Testing Decisions

A good test asserts observable behaviour at the seam: what the store was asked to do, what the endpoint returned, what the database holds. It never asserts private steps, and its expected values come from the spec, never recomputed the way the code does.

• Executor and scheduler, against a substituted store interface, with time driven by FakeTimeProvider. Prior art: the token cleanup service tests and the signalling time provider test double. Covered:

  • heartbeat cadence;
  • fenced → handler cancelled; cancel request → handler cancelled;
  • shutdown → release without a stall; exception → Failed;
  • one item at a time;
  • due-ness from the last finished item;
  • scheduling only within the worker's own scopes.

• PostgreSQL store, in a new integration test project running a real PostgreSQL through Testcontainers, in CI as well. Contract tests:

  • two concurrent claimers never get the same item;
  • scope routing;
  • a zombie attempt's heartbeat and completion affect no rows after a reclaim;
  • lease expiry recovery and the 3-stall limit;
  • DedupKey coalescing for queued items only;
  • cascade on config delete;
  • status-guarded cancel.

  The integration project is outside the Stryker run, and its PostgreSQL-only members get // Stryker disable all with ADR 0005 rows, as CLAUDE.md requires. This is new test infrastructure: the project's SDK setup and the CI job (added to the CI gate's needs) are part of the first ticket.

• Handlers, tested directly with substituted repositories and MockFileSystem. Each kind includes a "second run converges" test. The existing token cleanup tests move onto its handler. The crawl handler reuses the crawl and ingestion pipeline tests' fixtures.

• Registry: a missing or duplicate payload or handler for a kind fails startup.

• API endpoints (list, detail, cancel status codes, SSE emitting changed items and resuming from Last-Event-ID), in the existing cheap endpoint-test style, with time faked for SSE.

• Architecture rules for the write-ownership decisions, each shown failing on a deliberate violation.

• Mutation testing at 100% applies to everything outside the PostgreSQL-only store members.

Out of Scope

• The orphaned-proxy sweep and selected deletion, their enqueue endpoint and their result table (feat(worker): Orphaned proxy file sweeper (proxies ↔ DB reconciliation) #174, which builds on this).
• The Anichron.UI admin view itself; this spec delivers the API it reads.
• An admin "crawl now" endpoint; a later per-kind enqueue endpoint.
• Per-file retry and failure tracking inside a crawl (feat(worker): Per-item retry with exponential backoff and persistent failure tracking #132).
• PostgreSQL LISTEN/NOTIFY. It can be added later for faster pickup or SSE without changing the progress sink or the UI contract.
• WebSockets and SignalR.
• Priorities beyond first in, first out.
• Automatic retry of failed items.
• A coordinator that starts or manages containers.
• Regenerating missing proxies for stored assets (the "third source" noted on feat(worker): Orphaned proxy file sweeper (proxies ↔ DB reconciliation) #174).

Further Notes

• Order: this spec's tickets come first. feat(worker): Orphaned proxy file sweeper (proxies ↔ DB reconciliation) #174 is blocked by this issue and is rewritten on top of it, keeping its 2026-10-18 decision: report-only, manual scans, 1 h grace period, a re-check before deletion, and deletion items scoped to the config their directory belongs to.
• Deployment: the shared /data/proxies mount and the per-user and all-user worker shapes stay as they are.
• Design: decided by the maintainer on 2026-10-18 (grilling); every decision is in Implementation Decisions above. The options weighed are in feat: Work queue — API requests Worker jobs, admin sees queued and running work #273.

First caller

The TokenCleanup and WorkItemRetention kinds, which replace the token cleanup timer in the same PR as the queue. Next come the Crawl kind, then #174's sweep and deletion, which adds the first admin enqueue endpoint.

Design

Decided by the maintainer in a grilling session on 2026-10-18. The decisions are recorded in Implementation Decisions above, and the library evaluation is in #273. The first ticket adds the ADR.