feat: PostgreSQL integration tests and the work queue core (#277, #278)

.github/workflows/ci.yml

@@ -34,7 +34,7 @@ jobs:
         run: dotnet build src/
 
       - name: Test
-        run: dotnet test src/ --no-build --collect:"XPlat Code Coverage" --results-directory coverage --settings src/coverlet.runsettings
+        run: dotnet test src/ --no-build -p:TestSuite=Unit --collect:"XPlat Code Coverage" --results-directory coverage --settings src/coverlet.runsettings
 
       - name: Generate coverage report
         uses: danielpalme/ReportGenerator-GitHub-Action@v5
@@ -91,6 +91,21 @@ jobs:
       - name: Schema documentation is current
         run: scripts/derive-db-docs.sh --check
 
+  integration:
+    name: Integration tests
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+    steps:
+      - uses: actions/checkout@v7
+
+      - uses: actions/setup-dotnet@v6
+        with:
+          dotnet-version: '10.0.x'
+
+      - name: Integration tests
+        run: dotnet test src/ -p:TestSuite=Integration
+
   mutation:
     name: Mutation
     runs-on: ubuntu-latest
@@ -204,7 +219,7 @@ jobs:
   gate:
     name: CI gate
     runs-on: ubuntu-latest
-    needs: [checks, mutation, docker]
+    needs: [checks, integration, mutation, docker]
     if: always()
     permissions:
       contents: read

CLAUDE.md

@@ -29,17 +29,27 @@ dotnet run --project src/Anichron.Worker
 ## Testing
 
 ```bash
-# Run all tests
+# Run the unit tests (the default suite)
 dotnet test
 
+# Run the integration tests (needs a running Docker daemon)
+dotnet test -p:TestSuite=Integration
+
+# Run both
+dotnet test -p:TestSuite=All
+
 # Run tests for a specific project
 dotnet test src/SomeProject.Tests.Unit/SomeProject.Tests.Unit.csproj
 
 # Run a single test
 dotnet test --filter "FullyQualifiedName=Namespace.ClassName.MethodName"
 ```
 
-Test projects must be named `*.Tests.Unit` — `Directory.Build.props` auto-applies the test SDK to any project matching that pattern: xUnit v3, **NSubstitute** (not Moq), FluentAssertions, `System.IO.Abstractions.TestingHelpers`, and coverlet. Internal members are exposed to test projects via `InternalsVisibleTo`.
+An integration test project, or a filter matching integration tests, runs nothing without `-p:TestSuite=Integration` (or `All`).
+
+Test projects are recognised by name: `*.Tests.Unit` for unit tests and `*.Tests.Integration` for integration tests. `Directory.Build.props` applies the test SDK to both: xUnit v3, **NSubstitute** (not Moq), FluentAssertions, `System.IO.Abstractions.TestingHelpers`, and coverlet. `Directory.Build.targets` makes only the projects in the selected `TestSuite` (`Unit` by default, `Integration` or `All`) test projects, so `dotnet test` and Stryker never run the integration suite. Internal members are exposed to test projects via `InternalsVisibleTo`.
+
+**Unit or integration?** A test belongs in `Anichron.Tests.Integration` only when its subject is behaviour that only real PostgreSQL has: `ExecuteUpdate`/`ExecuteDelete`, transactions, advisory locks, row locking, constraint and partial-index enforcement. Everything else is a unit test, including a query's SQL shape (assert on `ToQueryString()` with `AnichronDbContexts.NpgsqlModel()`). The integration project starts one PostgreSQL container per run (`PostgresServer`, the same image as `src/docker-compose.yml`). Each test gets its own database, from `CreateMigratedDatabaseAsync` (real migrations) or `CreateDatabaseAsync` (empty). Its CI job is `Integration tests`, and it runs outside mutation testing, so the members it covers keep their `// Stryker disable all` and ADR 0005 rows.
 
 Shared test infrastructure lives in `Anichron.Testing`, referenced by the test projects only and tested by `Anichron.Testing.Tests.Unit`; it is excluded from mutation (ADR 0005, rule 4). Use it instead of a private copy:
 
@@ -85,11 +95,12 @@ Four projects with strict separation of concerns:
 | `Anichron.Core` | Single source of truth: domain models, `AnichronDbContext`, all Fluent API config, shared utilities (XXHash64, path parsing). Zero dependencies on other projects (every other project references Core, so a reference back would be circular and fail the build). |
 | `Anichron.Infrastructure` | Shared host plumbing used by API and Worker: configuration (`DatabaseConfiguration` reads the connection from `POSTGRES_CONNECTION__*` env vars or Docker secrets; INI bootstrap; `ConfigurationList`), migration under a PostgreSQL advisory lock, and the API's read-only stores (`IProxyStore`, `IOriginalStore`). Each host wires its own DI. |
 | `Anichron.API` | ASP.NET Core Minimal APIs (the only web project: `libraries-are-not-web` keeps ASP.NET Core out of Core, Infrastructure and Worker, and the two hosts never use each other) — owns all identity/auth/admin writes (users, storage configs, refresh tokens, invites, lockout); read-only for media data. Media endpoints so far: proxy serving (#43) and On This Day (`GET /api/v1/flashbacks/on-this-day`, #38); originals (#44) are deferred past v1.0. All routes prefixed `/api/v1/`. |
-| `Anichron.Worker` | `BackgroundService`s — NAS crawling, EXIF extraction, FFmpeg transcoding, proxy generation, expired-refresh-token cleanup; reconciliation is planned (Epic 4) |
+| `Anichron.Worker` | `BackgroundService`s — NAS crawling, EXIF extraction, FFmpeg transcoding, proxy generation, and the work queue's executor and scheduler (maintenance such as expired-refresh-token cleanup runs as work items); reconciliation is planned (Epic 4) |
 
 **Write ownership is split by domain, not centralized.**
-- **Worker is the only writer of *media* data** — `MediaAsset`, `Metadata`, `ProxyFile`, `Burst`, plus all reconciliation and soft-deletes happen here. The API never writes media: outside its DI wiring it never touches `AnichronDbContext` (architecture rule `api-reaches-data-through-repositories`), and the media write repository stays out of it (`only-worker-writes-media`). Its one identity-side write is `TokenCleanupService` deleting expired refresh tokens.
+- **Worker is the only writer of *media* data** — `MediaAsset`, `Metadata`, `ProxyFile`, `Burst`, plus all reconciliation and soft-deletes happen here. The API never writes media: outside its DI wiring it never touches `AnichronDbContext` (architecture rule `api-reaches-data-through-repositories`), and the media write repository stays out of it (`only-worker-writes-media`). Its one identity-side write is the `TokenCleanup` work item's handler deleting expired refresh tokens.
 - **API owns all *identity/auth/admin* writes** — registration, login/refresh-token issuance and revocation, lockout state, invites, and admin user/storage-config CRUD all persist through API services (`AuthService`, `AdminUserService`, `AdminStorageConfigService`, `TokenService`, etc.) via `IUnitOfWork.SaveChangesAsync`. The API is read-only *only with respect to media*. The one exception is `AssetInteraction` (star/like/hide, #46): per-user state, to be written by the API.
+- **Work items** are inserted only through `IWorkQueue` (architecture rule `only-work-queue-inserts-work-items`); every later state change belongs to the Worker's executor, and handlers live only in the Worker (`work-handlers-live-in-worker`). See "Work queue" below.
 - Both hosts apply EF migrations at startup, serialised by a PostgreSQL advisory lock (`MigrateWithAdvisoryLockAsync`).
 
 A planned `Anichron.UI` (NextJS/React) for the Instagram Story-style flashback experience is documented but not yet in the solution.
@@ -107,7 +118,7 @@ into PostgreSQL and therefore into `docs/schema.sql` and any schema-doc tool poi
 database. If you find yourself about to explain *why* a column exists in a Markdown file, put it in
 a `HasComment` instead — see `docs/adr/0004-schema-documentation.md`.
 
-**Nine entities**, all EF Core config via **Fluent API only** (architecture rule `domain-has-no-data-annotations`). Columns are
+**Ten entities**, all EF Core config via **Fluent API only** (architecture rule `domain-has-no-data-annotations`). Columns are
 **PascalCase** — the only explicitly-named column in the whole model is `Invites.xmin`. Table names
 do not always match entity names (`AssetInteraction` → `Interactions`, `UserStorageConfig` →
 `StorageConfigs`):
@@ -119,8 +130,9 @@ do not always match entity names (`AssetInteraction` → `Interactions`, `UserSt
 - `AssetInteraction` — per-user state (starred, liked, hidden, view count, last viewed); unique on `(UserId, AssetId)`
 - `UserStorageConfig` — NAS root paths per user; `RootPath` is **globally unique** — a path cannot be assigned to more than one storage config across all users. Paths are stored canonical (absolute, `.`/`..`/`//` collapsed, no trailing slash) via `StoragePath.Canonicalize`, so `/p` and `/p/` cannot both be assigned. A `RootPath` must be a folder strictly inside `StoragePath.OriginalsRoot` (`/data/originals`), never the root itself, and may not contain or be contained by another config's root (`StoragePath.Overlaps`) — nested roots would index the same files twice
 - `User` — owns storage configs and interactions
-- `RefreshToken` — issued and revoked by the API, expired rows deleted by the Worker's `TokenCleanupService`; `TokenHash` unique
+- `RefreshToken` — issued and revoked by the API, expired rows deleted by the Worker's `TokenCleanup` work item; `TokenHash` unique
 - `Invite` — admin-issued registration tokens; `TokenHash` unique, and the only entity carrying an optimistic-concurrency token (`xmin`)
+- `WorkItem` — one durable unit of Worker work (`Kind`, jsonb `Payload`, optional `DedupKey`, status, lease, attempt, progress snapshot, result or error); see "Work queue"
 
 ### Key EF Core / Database Decisions
 
@@ -179,6 +191,16 @@ Each user has a private set of `UserStorageConfig` entries pointing to NAS roots
 
 **Asset lists are read through `IAssetViewReader`** (`Anichron.Core/Data/Repository/AssetViewReader.cs`, #253). It always applies `VisibleTo` and `NotHiddenFor` to the caller, then the caller's `IAssetSelection` (e.g. `OnThisDay`, which filters only on `MediaAsset` columns), then left-joins the caller's own interaction row, all projected into a flat `AssetView` in one SQL statement. A new list view adds a selection class; it never changes the reader, and no API type touches a media entity (architecture rule `api-reads-media-through-views`). Grouping and ordering (`FlashbackYears`: years newest first, starred first, then capture time) happen in memory on the result.
 
+### Work queue
+
+Durable work for the Worker lives in `WorkItems` (#276, ADR 0008). Today its kinds are periodic maintenance (`TokenCleanup`, `WorkItemRetention`); the crawl (#281) and admin requests (#174) follow.
+
+- **Producing:** `IWorkQueue.EnqueueAsync` (`EfWorkQueue`, Core) is the only insert. A payload with a `DedupKey` coalesces: while an item with that key is `Queued`, enqueueing returns it (a partial unique index absorbs racing inserts).
+- **Scheduling:** every Worker runs `WorkScheduler`. Each poll (`Worker:WorkPollIntervalSeconds`, 5 s) it recovers expired leases, then asks each `IWorkSchedule` whether its kind is due: nothing queued or running with its key and nothing finished within its interval. No timer state is kept, so overdue work runs right after downtime.
+- **Executing:** `WorkExecutor` claims the oldest queued item (`FOR UPDATE SKIP LOCKED`), one at a time, under a lease (`WorkLeaseSeconds`, 60 s) that a heartbeat (`WorkHeartbeatSeconds`, 2 s) extends while writing the progress snapshot. Every write about a running item is fenced by `(Id, Attempt)`. A fenced heartbeat cancels the handler and writes nothing more; a shutdown releases the item without a stall; a handler exception fails it with no retry. An expired lease requeues the item and counts a stall; the `WorkMaxStalls`th (3) fails it as stalled. An item handed back (release or recovery) while another with its key is already queued ends `Cancelled` as superseded, since the queued one does the same work. A Worker claims only the kinds it knows, so a newer kind waits for a newer Worker. `WorkItemRetention` deletes items finished more than `WorkItemRetentionDays` (30) ago.
+- **Adding a kind:** a `WorkKind` value, a payload record in Core implementing `IWorkPayload` (static `Kind`, optional `DedupKey`), an `IWorkHandler<TPayload>` in the Worker registered with `AddWorkHandler`, and for a periodic kind a `WorkSchedule<TPayload>` subclass. `WorkHandlerRegistry` fails startup unless every kind has exactly one payload and one handler; nothing switches on the kind.
+- **Handler contract:** delivery is at least once. Each run starts from scratch and converges, side effects are safe to repeat, and the returned summary becomes `Result`.
+
 ### Worker Processing Model
 
 Processing uses a bounded-concurrency `Channel<T>` pipeline. The crawler produces file paths; N consumer tasks process them concurrently. Default concurrency: `Worker:MaxConcurrentFiles = 4`. Processing is always idempotent — files already in the DB (matched by `(StorageConfigId, ContentHash)`) skip extraction and proxy generation; only the row's `FilePath`/`LastSeenOnNas` are updated, and a file whose bytes changed at a stored path updates that row in place (see Reconciliation above).

docs/adr/0005-mutation-scope-excludes-unkillable-and-wiring-code.md

@@ -86,11 +86,15 @@ The directive is left bare — the reason lives here, not in the code.
 | --- | --- |
 | `AnichronDbContext.ExecuteInTransactionAsync` (both overloads) | Commit versus no commit is only observable on a provider with real transactions. |
 | `EfRefreshTokenRepository.RevokeAllActiveByUserIdAsync` and `DeleteExpiredAsync` | `ExecuteUpdate`/`ExecuteDelete` are not supported by the EF InMemory provider. |
+| `EfWorkQueue.TryInsertAsync` | Absorbing a lost race needs the partial unique index on `DedupKey`, which the InMemory provider does not enforce. The rest of `EnqueueAsync` is in scope. |
+| `EfWorkItemStore` members except `IsDueAsync`, and its `Running`/`Expired` filters | The claim is raw SQL with `FOR UPDATE SKIP LOCKED`; every transition is an `ExecuteUpdate`/`ExecuteDelete` fenced by `(Id, Attempt)`. `IsDueAsync` is plain LINQ and in scope. |
 
 These are **not** tested against a second SQL engine such as SQLite. A substitute engine
 diverges from PostgreSQL in exactly the places this model relies on (partial-index filters,
 `xmin`, NodaTime types, collation), so a green test there would claim coverage it does not
-have. They stay excluded until tests run against real PostgreSQL.
+have. Where they are tested, it is against real PostgreSQL in `Anichron.Tests.Integration`
+(`DeleteExpiredAsync` and `MigrateWithAdvisoryLockAsync` so far). That project is outside the
+Stryker run, so the members stay excluded from mutation.
 
 `WorkerSettings.FfmpegPath` is **not** in that list. It is a real default and is tested. `RootPath`
 lost its default in #221: it names one user's folder inside `/data/originals`, so no value is right
@@ -107,7 +111,7 @@ for every deployment. The proxy root is no longer a setting at all since #43: it
   `ServiceCollection` and resolve the configured options.
 
 - **`Log.*` hides four mutants that existing tests do kill**: the two log calls in
-  `LoggingMiddleware`, `TokenCleanupService`'s deleted-count log, and `PwnedPasswordClient`'s
+  `LoggingMiddleware`, `TokenCleanupService`'s deleted-count log (now `TokenCleanupHandler`'s), and `PwnedPasswordClient`'s
   "check unavailable" warning. Accepted — the tests stay and still guard those lines; they just no
   longer count toward the score. `LoggingMiddleware` is left with almost no mutants of its own as a
   result, which is expected for a class whose only behaviour is logging.

docs/adr/0008-hand-rolled-postgresql-work-queue.md

@@ -0,0 +1,43 @@
+# A hand-rolled PostgreSQL work queue, not a job library
+
+Status: accepted (2026-10-18, #276).
+
+Work the Worker does on request or on a schedule is a row in `WorkItems`, in our EF model. A Worker
+claims a row with `FOR UPDATE SKIP LOCKED`, holds it under a lease that a heartbeat extends, and
+records the outcome in the same row. No job library is used.
+
+## What it has to do
+
+The API and the Worker share only the database. The queue has to let the API ask for work that only
+a Worker can do, route a request to a Worker that has the right folder mounted (a per-user worker
+sees one user's originals), recover an item whose Worker died, and give an admin a list of what is
+queued, running and finished, with progress, from our own API.
+
+## Libraries considered
+
+The full evaluation is on #273. In short:
+
+| Library | Why not |
+| --- | --- |
+| Hangfire + Hangfire.PostgreSql | Feasible, but its schema lives outside the EF model: not in `docs/schema.sql`, no `HasComment` (ADR 0004), created at startup beside our advisory-locked migrations. Jobs are serialised method calls on types both hosts share, so a rename breaks queued items. Newtonsoft.Json and `DateTime` at its boundary. No progress model, and per-job queue routing on PostgreSQL was unverified. |
+| TickerQ | Enqueue-in-one-host, run-in-another and scope routing are undocumented; EF-only crash recovery unclear. |
+| Quartz.NET (clustered) | A scheduler, not a work queue: any node runs any job, so routing would need one scheduler per user. No progress. |
+| Wolverine | A messaging framework and programming model for both hosts; no job-status model to list. |
+| Rebus | A message bus without a status view. |
+| MassTransit | Commercial since 2026. |
+
+## Consequences
+
+- One table, one model: `WorkItems` is in `docs/schema.sql`, its columns carry comments, and every
+  time is NodaTime.
+- We own the concurrency code. The claim, the fenced transitions and lease recovery are PostgreSQL-only,
+  so they carry `// Stryker disable all` (ADR 0005) and are covered instead by contract tests on real
+  PostgreSQL in `Anichron.Tests.Integration`: concurrent claimers, skip-locked, zombie attempts, the
+  stall limit, release without a stall and dedup coalescing.
+- Delivery is at least once. A handler starts from scratch and converges, and its side effects are
+  safe to repeat. Every write a Worker makes about a running item is fenced by `(Id, Attempt)`, so an
+  attempt that lost its lease changes nothing.
+- Due-ness comes from the table (nothing queued or running with the kind's key, nothing finished
+  within its interval), so no host keeps timer state and overdue work runs right after downtime.
+- `LISTEN/NOTIFY`, priorities and automatic retry are left out; each can be added without changing
+  the handler contract.