feat(worker): Orphaned proxy file sweeper (proxies ↔ DB reconciliation)

Why this is separate from #33

#33 ("Implement periodic reconciliation scan") reconciles DB ↔ NAS originals — soft-delete rows whose file vanished, re-link moved files by hash. This issue is the other direction: proxies on local SSD ↔ DB. Different source of truth, different scan, same epic ("Data Health").

Problem

/data/proxies accumulates files that no row references, and nothing in the solution ever deletes a file. A repo-wide grep for File.Delete / Directory.Delete returns zero hits (the only .Delete( is a cookie). There is no code path that can remove a proxy file, ever.

Two independent sources of orphans:

1. Process death mid-ingestion. Since #188, an in-process failure is compensated by IngestionPipelineRunner (deletes registered proxies + .tmp siblings, removes the directory only if empty), and retries land in the same content-keyed directory. What compensation cannot reach is a hard kill / power loss / OOM: completed proxies with no ProxyFile row, and stray *.tmp files from ProxyStagingWriter. A clean shutdown also deliberately keeps proxies already completed in that run.

2. Cascade deletes. Deleting a UserStorageConfig cascades MediaAsset → ProxyFile rows. Every row disappears; every byte stays on disk permanently. No in-process fix touches this path.

A sweeper is the only thing that covers both.

What makes this tractable

The expected proxy layout is fully derivable without a join:

IProxyDirectoryStrategy / TwoLevelHexShardStrategy (src/Anichron.Worker/Ingestion/Proxy/ProxyShardStrategy.cs): {contentHash[..2]}/{storageConfigId:N}-{contentHash} — one directory per (storage config, content hash), not per asset id.
Expected filenames: thumbnail.jpg, preview.jpg, blurhash.txt (images) and video_720p.mp4 (video), from the FileName of the generators in ImageProxyGenerators.cs / VideoProxyGenerators.cs.
ProxyFile.ProxyPath stores the path relative to the proxy root (default /data/proxies).
Any *.tmp (ProxyStagingWriter.TemporaryFileSuffix) older than a grace period is reclaimable.

So a sweep can enumerate shard directories and diff them against the (StorageConfigId, ContentHash) pairs, including soft-deleted rows, plus the ProxyPath set.

⚠️ Until the partial unique index in #159 lands, two MediaAsset rows can share one directory. Keying the diff on (config, hash) rather than on asset id handles that correctly; deleting a directory just because one of its rows is gone does not.

Trap to avoid

Both MediaAsset and ProxyFile carry soft-delete global query filters (AnichronDbContext). A naive EF query returns no proxy rows for soft-deleted assets, so a sweeper would classify their still-valid files as orphans and delete them — breaking the soft-delete/revive path that #33 depends on. The sweep must use IgnoreQueryFilters().

Open questions

Dry-run / report-only mode first, before it is allowed to delete anything?
Grace period before deleting a directory (guard against deleting a shard for an asset mid-ingestion by a concurrent consumer)?
Does it run on the crawl timer, or its own slower schedule?

Soft-delete missing files after a crawl, and auto-restore them when they return #33 — the DB ↔ NAS direction, same epic
fix(worker): Ingestion pipeline owns its failures — no orphaned proxy files #188 / feat(worker): Ingestion pipeline owns failure — no orphaned proxy files #175 — the in-process compensation half (done); this issue is the out-of-process remainder
Code & architecture review — findings tracker + content_hash/multi-user analysis #159 — config-scoped partial unique index on (StorageConfigId, ContentHash)
feat(worker): Per-item retry with exponential backoff and persistent failure tracking #132 — per-item retry and persistent failure tracking, the adjacent consumer-level concern

Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com

First caller

The admin endpoints under /api/v1/admin/proxy-sweeps (POST /, GET /{sweepId}/orphans, POST /{sweepId}/deletions) and the Worker's ProxySweep and ProxyDeletion handlers ship together in one PR. The Anichron.UI admin screen is a later caller.

Design

Approved note: #174 (comment). It is amended by the approval comment (#174 (comment)), and the decisions are in #174 (comment) and #174 (comment). The Open questions above are answered by those decisions.

refactor(worker): Express the staging-artifact name rule once

A third source: a losing attempt deletes the winner's file (left open by #263)

#263 stopped cleanup from deleting another attempt's files in most races, but one case is left. The fix needs a ref check, which belongs here.

What #263 fixed. Proxies live at content-derived paths, so two concurrent attempts on the same content (or two HEICs sharing one paired MOV) write to the same final path. #263 closed most of the overlap:

temporary files are named per attempt (<proxy>.<attempt id>.tmp), and cleanup removes only its own;
a unique index on ProxyFiles (AssetId, ProxyType) lets only one row win;
an attempt that loses a uniqueness race (LostIngestionRace) releases its paired MOV's registrations before failing;
a successful self-heal commit releases the MOV's registrations too.

Still open. An attempt can rename its proxy onto the shared path and then fail for some other reason, such as a DB timeout, a dropped connection, or an exception in a later step, while a concurrent attempt has already committed the row for that path. IngestionPipelineRunner compensation then deletes every proxy the loser registered, including the file the winner's committed ProxyFile row points to. The row stays and the file is gone, so the API serves a 404 for that proxy until something regenerates it.

This was already true for image and video proxies before #263; the paired MOV's WebVideo is just one more instance. It is rare in practice, since it needs a real race plus an unrelated failure in the loser.

Why it fits here. Compensation can't tell "my file" from "the same bytes the winner also wrote" without asking the DB. Two options:

Ref-checked deletion in compensation: before deleting a registered proxy, check whether any ProxyFile row references that path, using IgnoreQueryFilters() (see the trap above), and skip it if so.
Sweeper-side repair: find ProxyFile rows whose file is missing, and either regenerate them or drop the row so the next crawl self-heals. (PairedMovProxyWriter already self-heals a MOV that lacks its row, but not a row that lacks its file.)

Note on *.tmp: temporary files are now named <proxy>.<attempt id>.tmp. They still end in ProxyStagingWriter.TemporaryFileSuffix, so the grace-period rule above still applies.

Objection

Asked to: build an orphaned-proxy sweeper in the Worker that deletes files under /data/proxies which no row references (and, per the comment above, also handle a losing attempt deleting the winner's file).

Why it does not fit: this is a legacy issue (≤ #261) with no ## First caller or ## Design part. The trigger check in docs/agents/design-process.md says a design note is needed:

Hard to reverse. It would be the first code that deletes proxies outside one attempt's own compensation. A wrong orphan rule destroys valid proxies, and an existing asset never regenerates a missing one: IdempotencyCheckMiddleware returns without calling next on a stored match (IdempotencyCheckMiddleware.cs:28). Only a paired MOV's WebVideo self-heals.
Spans processes on a shared mount. Every per-user worker mounts /data/proxies read-write (src/docker-compose.yml:104). Per-worker sweepers would race each other and race other workers' ingestion. A storage config removed by AdminStorageConfigService.DeleteAsync cascades its rows away and leaves no worker that owns its directories.
New abstraction. It adds a third BackgroundService with its own schedule and settings.
The issue's open questions are unanswered: report-only first, the grace period, and the schedule.
Parts of the body are stale:
- blurhash.txt is no longer written (refactor(api): Narrow read-only storage interfaces instead of ReadOnlyFileSystem #251). Copies written before refactor(api): Narrow read-only storage interfaces instead of ReadOnlyFileSystem #251 are now dead files inside live directories, so a filename allow-list or a directory-level diff misses them.
- The Code & architecture review — findings tracker + content_hash/multi-user analysis #159 warning no longer applies, because the index has landed.
- The comment's "drop the row so the next crawl self-heals" does not hold for image or video proxies.

Options: write the design note first (posted below), or implement with the implementer's own answers to the open questions, which settles hard-to-reverse choices without review.

Recommendation: design note first. Moving to needs-decision, because the note marks the deletion rule as hard to reverse and asks the maintainer three questions.

Design note

Problem and constraints:

Files under /data/proxies that no ProxyFiles row references are never reclaimed. They come from:
- a hard kill, power loss or OOM mid-ingestion: completed proxies with no row, plus *.<attempt>.tmp files;
- a clean shutdown, which keeps proxies already completed in that run;
- the cascade that follows UserStorageConfig (or User) deletion;
- blurhash.txt files written before refactor(api): Narrow read-only storage interfaces instead of ReadOnlyFileSystem #251.
Every per-user worker mounts /data/proxies read-write (src/docker-compose.yml:104), so N workers share one tree.
A deleted config's directories belong to no worker. Its id is still encoded in the directory name ({hash[..2]}/{configId:N}-{hash}), but no row is left.
Soft-deleted assets keep their proxies for Soft-delete missing files after a crawl, and auto-restore them when they return #33's revive path. The global query filters on MediaAsset and ProxyFile would hide those rows, so every read here uses IgnoreQueryFilters().
An in-flight attempt makes things appear before its row commits:
- the directory, as soon as ProxyStagingWriter calls CreateDirectory;
- the temp file;
- the renamed final file.
The sweeper must not delete any of these while that attempt can still commit.
Losing a valid proxy is not self-healing. IdempotencyCheckMiddleware never calls next for a stored asset (IdempotencyCheckMiddleware.cs:28); only a paired MOV's WebVideo is regenerated.

Options:

One global, file-level sweep, elected by an advisory lock.
- Each run, every worker tries pg_try_advisory_lock on a dedicated id. The pattern is MigrateWithAdvisoryLockAsync, but non-blocking and with its own constant in PostgresConstants. The losers skip the run.
- The holder walks the proxy root and deletes every file that meets both conditions:
  - no ProxyFiles.ProxyPath matches its relative path (IgnoreQueryFilters());
  - its mtime is older than the grace period.
  It then removes directories that are empty and also older than the grace period.
- The rows are read per top-level shard (ProxyPath LIKE 'ab/%'), so memory stays at about 1/256 of the table.
- Covers: crash debris, .tmp files, cascade orphans and stale blurhash.txt, all with one rule. It needs no filename list and no parsing of directory names.
- Costs: one directory walk and one indexed read per shard, per run. Correctness depends on ProxyPath being stored exactly as the relative path on disk. That already holds, since ProxyStagingWriter is the only writer.
Per-worker, directory-level sweep keyed on (config, hash), as the issue body proposes.
- Each worker parses directory names back into (config, hash). That inverts IProxyDirectoryStrategy, so the path rule would no longer live in one place.
- It deletes a directory when no row has that pair, and checks files inside live directories against the generators' FileName list.
- Costs:
  - N workers sweep the same tree concurrently.
  - Deciding which worker reclaims a deleted config's directories needs an extra rule ("any worker may delete a directory whose config id has no row").
  - The filename list duplicates generator knowledge and silently goes stale (it already has: blurhash.txt).
Change the pattern: make config deletion explicit.
- DELETE …/storage-configs/{id} deactivates the config instead of removing it. The Worker purges the config's directories, then deletes the rows.
- Costs:
  - It changes an admin API contract and the cascade.
  - It still needs option 1 or 2 for crash debris, so it adds a mechanism without removing one.
  - It is worth doing only if immediate disk reclaim after a config deletion becomes a requirement.

Recommendation: option 1.

Shared mount: the advisory lock makes the sweep single-writer across all workers.
Orphaned configs: a global, row-based rule needs no idea of an owning worker, so they are covered.
Path knowledge: a file-level diff against ProxyPath reuses the rule the writer already applied, instead of re-deriving it.
In-flight attempts: the grace period protects them. The window between the final rename and the row commit is milliseconds to minutes (one transaction after the remaining steps). A grace period in hours is orders of magnitude larger.
Residual case: the sweeper removes an empty, old directory just as an attempt reuses it, before that attempt writes its temp file. The attempt then fails on write, compensates, and the next crawl retries it. That is acceptable, and it stays out of reach of the grace period only for a directory older than the grace period that has been empty all along.

The "third source" (comment above: a losing attempt deletes the winner's file): out of scope here; it belongs in a separate issue.

The sweeper only removes files, and this case needs a file recreated. Dropping the row doesn't help, because the asset is stored and its proxies are never regenerated (IdempotencyCheckMiddleware.cs:28).
A ref check in compensation narrows the window but cannot close it. The winner can commit after the check, and the loser often failed because the database was unreachable. If added, it must fail safe: when the check errors, keep the file and leave it to this sweeper.
The real fix is regenerating a missing proxy for a stored asset, a pipeline change of its own.

First caller: ProxySweepService, a Worker BackgroundService shaped like TokenCleanupService.

It uses a PeriodicTimer on TimeProvider (so tests drive time), a scoped read of ProxyFiles paths per shard, and IFileSystem for the walk.
New WorkerSettings: ProxySweepIntervalHours and ProxySweepGraceHours, plus ProxySweepMode if the maintainer picks report-only first.
The read is a new method on a Worker-side repository that returns ProxyPath strings for one shard prefix with IgnoreQueryFilters(). No media type crosses into the sweeper.
Tests drive it with MockFileSystem and a substituted repository. The lock and the LIKE read are PostgreSQL-only, so they get // Stryker disable all and an ADR 0005 row, per CLAUDE.md.

Hard to reverse: yes. Deleted proxies are not regenerated for stored assets, so a wrong orphan rule causes permanent 404s until the asset is re-ingested. No schema or API contract changes.

Decision needed:

Report-only first?
- Options:
  - (a) ship with ProxySweepMode=Report, which logs what would be deleted, and switch to Delete in a later release;
  - (b) ship deleting, behind the grace period.
- Recommendation: (a) for one release, since the cost of a wrong rule is permanent.
Grace period.
- Options: (a) 24 h; (b) 1 h; (c) equal to CrawlIntervalHours.
- Recommendation: (a) 24 h. It is far above any single attempt's duration, including a long video transcode. Disk reclaim isn't urgent.
Schedule.
- Options: (a) its own timer, ProxySweepIntervalHours; (b) after each crawl.
- Recommendation: (a), default 24 h, with the first run one interval after startup rather than immediately. Startup is when crash debris from the previous run is freshest, but it is also when the grace period protects nothing older. Waiting one interval keeps the first sweep away from the startup crawl's ingestion burst.

Decision

Maintainer, 2026-10-18, on the questions in the design note above:

Report-only, with manual deletion. A sweep only reports orphans. Deletion is an explicit admin action on selected entries from a report: a button once Anichron.UI exists, an admin API endpoint until then. Nothing is ever deleted unattended.
Grace period: 1 h at scan time, plus a re-check at delete time. With a human reviewing before anything is deleted, the grace period only keeps in-flight attempts out of the report. Safety at deletion comes from re-checking each selected file just before it is removed: it must still be unreferenced (IgnoreQueryFilters()) and unchanged in size and mtime since the report. Otherwise it is skipped.
Schedule: manual only. A sweep runs when an admin requests one. There is no timer and no interval setting.
Mechanism: admin API + Worker. The API records requests in the database, and the Worker, the only process with the proxy mount read-write, does the file work.

These answers supersede the recommendation in the note above (a timer-driven sweep that deletes on its own). A revised design note follows. Because the decision makes the change span the API and the Worker and adds schema and an admin API contract, the revised note needs architect approval before ready-for-agent, so the issue moves to needs-design.

Out of scope, as the first note says: a losing attempt deleting the winner's file needs a missing proxy regenerated, which is a separate issue.

Design note (revised after the decision)

Problem and constraints:

An admin requests a sweep, reviews the orphans it found, and deletes selected ones. The orphan rule is unchanged from the first note: files that no ProxyFiles.ProxyPath references (IgnoreQueryFilters()) and that are older than 1 h.
The API cannot touch proxy files:
- its mount is :ro;
- the api-reads-no-file-system architecture rule forbids it, and ADR 0007 records why;
- IProxyStore has no write method.
So every file operation runs in the Worker, and the API only records requests and reads results.
Write ownership stays split as in CLAUDE.md: the API writes admin data, and the Worker is the only writer of media and of proxy files.
Several workers share /data/proxies, so exactly one of them must run a given request.
Admins get no media access. A report lists only files no row references, so it exposes paths (config id and content hash), never an asset.

Options:

Request tables written by the API, result tables written by the Worker.
- ProxySweepRequests (API writes): Id, RequestedBy, RequestedAt.
- ProxySweeps (Worker writes):
  - RequestId (PK/FK), Status (Running, Reported, Failed);
  - StartedAt, CompletedAt;
  - OrphanCount, OrphanBytes.
- ProxyOrphans (Worker writes): Id, SweepId, ProxyPath, SizeBytes, LastWriteTime.
- ProxyDeletionRequests (API writes): Id, OrphanId (unique), RequestedBy, RequestedAt.
- ProxyDeletions (Worker writes):
  - RequestId, Outcome (Deleted, SkippedReferenced, SkippedChanged, Missing, Failed);
  - ProcessedAt.
- Costs: five tables. Every table has exactly one writing host, so ownership can be checked by an architecture rule.
One job table per kind, with phase ownership.
- ProxySweeps is inserted by the API and updated by the Worker. ProxyOrphans carries DeletionRequestedAt (set by the API) and DeletedAt/Outcome (set by the Worker).
- Racing updates are guarded by xmin, as on Invites.
- Costs: two tables, but both hosts write the same rows. The ownership split becomes a rule about columns, which no architecture rule can check.
PostgreSQL LISTEN/NOTIFY instead of polling for pickup, on top of either of the above.
- Costs: a long-lived dedicated connection in the Worker, plus reconnect handling. That is a lot of machinery for an action an admin takes a few times a year.

Recommendation: option 1 with polling.

One writer per table keeps CLAUDE.md's ownership split mechanical, and it can be pinned by an architecture rule (only-worker-writes-proxy-sweep-results).
The extra tables are small and append-only.
Pickup is a poll every ProxySweepPollSeconds (default 30), which is fine for a manual action.

How it runs:

Claim. A Worker claims an unclaimed request by inserting its ProxySweeps / ProxyDeletions row. The primary key makes exactly one worker win, and the others hit the unique violation and skip. This is the same "loser skips on the unique index" pattern as LostIngestionRace, so no advisory lock is needed.
Sweep.
- The Worker walks the proxy root and reads ProxyPaths one top-level shard at a time (LIKE 'ab/%').
- It writes each orphan older than 1 h as a ProxyOrphans row, then sets Reported.
- Orphaned empty directories are not listed. They are removed when they become empty after a deletion, or by a later sweep's deletion in the same directory.
Deletion. Per request, the Worker re-checks that ProxyPath is still unreferenced (IgnoreQueryFilters()) and that size and mtime equal the report:
- if both hold, it deletes the file, removes the parent and shard directories if they are empty, and records Deleted;
- otherwise it records the matching Skipped…/Missing outcome.
A .tmp file is treated like any other orphan.
Admin endpoints under /api/v1/admin/proxy-sweeps, admin role only:
- POST / returns 202 with the request id;
- GET /{id} returns the status, the totals, and a paged list of orphans with each deletion's outcome;
- POST /{id}/deletions takes { "orphanIds": [...] } or { "all": true } and returns 202. all exists because a deleted storage config can orphan tens of thousands of files.
Unknown ids return 404.

First caller: the three admin endpoints, and the Worker's ProxySweepService that serves them, ship together.

The UI button is a later caller of the same endpoints.
ProxySweepService is a BackgroundService on a PeriodicTimer driven by TimeProvider, shaped like TokenCleanupService. Its pieces are tested with MockFileSystem and substituted repositories:
- the scan;
- the re-check, against an independent expected outcome per case: referenced, changed, missing, deleted;
- the claim.
PostgreSQL-only code gets // Stryker disable all and an ADR 0005 row, per CLAUDE.md:
- the unique-violation claim;
- the shard LIKE read.

Hard to reverse:

Five tables and a public admin API contract.
File deletion stays manual and re-checked, so a wrong orphan rule produces a wrong report, not lost proxies.

Left out on purpose:

Retention of old sweep reports. A follow-up can purge reports older than N days once there is a need.
Any timer.
Regeneration of missing proxies (separate issue).

Superseded

The revised design note above (one-off request and result tables for the sweep) is superseded. API → Worker work requests become a general work queue, designed in #273, and #174 is its first caller. The decision of 2026-10-18 (report-only, manual scan, 1 h grace, re-check before delete) stands; only the transport changes. This issue's note gets rewritten on top of #273 once that design is settled.

The work queue is now specified in #276 (it replaces #273, now closed), and this issue is blocked by #276.

feat(worker): Update the row in place when a file's content changes a…

Note from the 2026-10-24 architecture review (#302): StorageConfigSchedules<TPayload> (src/Anichron.Worker/Work/StorageConfigSchedules.cs) schedules every per-config kind at Worker:CrawlIntervalHours. That is right for Crawl and BurstDetection, which the #296 design note put on the crawl interval. A per-config kind added here would inherit it without saying so. When this issue adds one, give the schedule source its interval per registration (for example as a constructor argument) instead of reading the crawl setting.

🤖 Generated with Claude Code

Constraint from the #303 design (#303 (comment), decision 5): proxy files live in a folder derived from the config and the content hash, so an active row and a soft-deleted row can reference the same proxy files. That happens when a path holder takes content a now soft-deleted row already had (OverwrittenByMove, and #301's path-wins auto-restore). Deleting a soft-deleted row's files (purge, orphan sweep) must first check that no other row references the same ProxyPath.

🤖 Generated with Claude Code

Design note (rewritten on the work queue, #276)

This replaces the revised note of 2026-10-18 (superseded). The maintainer's decision of 2026-10-18 stands:

sweeps are report-only;
deletion is manual, on selected entries;
the grace period is 1 h at scan time;
each file is re-checked before it is deleted;
there is no timer.

The #276 spec adds one more decided point: deletion items are scoped to the config their directory belongs to.

Problem and constraints:

Only the Worker can touch proxy files. The API mounts them :ro, and api-reads-no-file-system (ADR 0007) forbids file access from the API. So the API only enqueues and reads. Requests travel as work items (IWorkQueue), and the API never writes media or proxy files.
The orphan rule is unchanged: a regular file under StoragePath.ProxiesRoot that no ProxyFiles.ProxyPath references, with IgnoreQueryFilters(), last written more than 1 h before the scan. Because the rule asks the database about each file and is not a filename allow-list, it also covers stray *.tmp files (ProxyStagingWriter.TemporaryFileSuffix) and the blurhash.txt files left by refactor(api): Narrow read-only storage interfaces instead of ReadOnlyFileSystem #251.
An active row and a soft-deleted row can share proxy files (fix(worker): A file overwritten with content stored elsewhere leaves its row stale #303 decision 5). "Referenced" therefore means referenced by any row, soft-deleted ones included.
A proxy directory is {hash[..2]}/{configId:N}-{hash} (IProxyDirectoryStrategy), so every orphan's config can be read from its path. Orphans of a cascade-deleted config name a config that no longer exists. Since config ids are never reused, no crawl will ever write under that prefix again.
A Worker runs one item at a time. Scoping a live config's deletion to that config means the deletion never overlaps that config's crawl on the Worker serving it.
ProxyFiles.ProxyPath has no index today. Both the per-shard scan read and the re-check before each delete need one.
Admins get no media access. A report lists paths (config id, content hash, file name, size, mtime), never an asset or its owner's files.

Options:

Orphans in a table only the Worker writes, keyed to the sweep's work item.
- ProxyOrphans:
  - Id;
  - SweepId, an FK to WorkItems.Id with cascade, so WorkItemRetention purges reports after 30 days with no new setting;
  - ProxyPath, StorageConfigId (parsed from the path, nullable, no FK, because the config may be gone or may go later), SizeBytes, LastWriteTime;
  - DeletionOutcome (null until processed: Deleted, SkippedReferenced, SkippedChanged, Missing, Failed) and ProcessedAt;
  - a unique index on (SweepId, ProxyPath).
- Plus a non-unique index on ProxyFiles.ProxyPath, with text_pattern_ops so the shard prefix LIKE 'ab/%' uses it.
- Cost: one table, one index, and an architecture rule: outside the Worker, only a Core reader (IProxyOrphanReader) depends on ProxyOrphan, the same shape as work-items-change-only-through-queue-or-cancel.
Orphans in the sweep item's Result JSON, deletion payloads carry paths.
- Cost:
  - no schema change;
  - but Result is unbounded: a deleted config orphans every file it had, often tens of thousands;
  - the admin list and the SSE stream (WorkItemEvents, every 2 s) carry that Result, and paging happens in memory;
  - a deletion's per-file outcome scatters across the deletion items' Results, so "what happened to this orphan" becomes a join over JSON.
Change the pattern: let a work item own child rows generically, for example WorkItemEntries(WorkItemId, Key, Data jsonb) for any kind with a large report.
- Cost: a generic table with no column comments of its own (ADR 0004) and untyped data, built for a single caller. Option 1 can grow into this if a second kind ever needs it.

Recommendation: option 1. It keeps one writer per table, gives paged reads with an index, and its retention comes for free from the work item FK. Decided by the maintainer on 2026-10-26, see below.

How it runs:

Kinds. Two new kinds, ProxySweep and ProxyDeletion, both admin-requested only. Neither has a schedule, so no IWorkScheduleSource is registered and the crawl-interval note from Architecture review due (10 PRs since arch-review/2026-10-18) #302 does not apply.
Sweep. The sweep is ProxySweepPayload(): unscoped, with dedup key proxy-sweep, so a double click coalesces.
- Walk. The handler walks the proxy root one top-level shard at a time. For each shard it reads the referenced ProxyPaths with IgnoreQueryFilters() and the LIKE 'ab/%' prefix.
- Record. It inserts every unreferenced file older than 1 h as an orphan, using ON CONFLICT DO NOTHING on (SweepId, ProxyPath), so a stalled and re-run attempt converges.
- Progress. Progress counts shards done, out of the shards found.
- Result. Result is { orphans, bytes, temporaryFiles, configsGone }.
- Not reported. Empty directories are left out of the report. A deletion removes a directory once it becomes empty.
Deletion. The deletion is ProxyDeletionPayload(Guid SweepId, Guid? StorageConfigId, IReadOnlyList<Guid>? OrphanIds). A null OrphanIds means every orphan of that sweep and scope.
- Scope.
  - Scoped to StorageConfigId while that config exists.
  - Unscoped, so any Worker may run it, for orphans of a config that no longer exists. Decided 2026-10-26.
- Re-check. For each orphan not processed yet, the handler re-checks just before deleting:
  - Referenced: still unreferenced by any row (IgnoreQueryFilters()), else SkippedReferenced.
  - Changed: size and mtime unchanged since the report, else SkippedChanged.
  - Missing: the file still exists, else Missing.
- Delete. If all three hold, it deletes the file and removes its directory and shard directory if they are now empty, never recursively, then records Deleted.
- Repeat runs. Orphans that already have an outcome are skipped, so a repeat converges.
- Result. Result is the count per outcome.
Admin endpoints, under /api/v1/admin/proxy-sweeps, admin role only:
- POST / enqueues a sweep and returns 202 { workItemId }. Status and progress come from the existing work-item endpoints and SSE.
- GET /{sweepId}/orphans?outcome=&storageConfigId=&page= returns a paged list of orphans with their outcomes. It returns 404 for an unknown id or one that is not a sweep, and 409 while the sweep has not succeeded.
- POST /{sweepId}/deletions, with { "orphanIds": [...] } (at most 1000) or { "all": true }:
  - The API groups the selection by StorageConfigId, nulling configs that no longer exist, and enqueues one deletion item per group.
  - It returns 202 { workItemIds }.
  - Ids that are not orphans of this sweep return 400, and an unknown sweep 404.

First caller: the three endpoints and both handlers ship together, so every piece of the interface is exercised.

Unit tests. The scan and the re-check are unit-tested with MockFileSystem, substituted repositories and a Timeline clock, each case with its own expected outcome:
- referenced by an active row, and by a soft-deleted row only;
- changed size, changed mtime;
- missing;
- younger than 1 h;
- a .tmp file;
- a gone config.
Integration tests. These cover the PostgreSQL-only parts: the ON CONFLICT insert, the prefix read using the index (EXPLAIN, as FindByHashesOrPathsAsync does), and the cascade on retention. Those members carry // Stryker disable all and ADR 0005 rows.
Later caller. The Anichron.UI admin screen is a later caller of the same endpoints.

Left out on purpose:

regenerating proxies whose file is missing (the third source in the 2026-10-17 comment, out of scope per spec: Work queue — durable API/Worker work items with scope routing, leases and a live admin view #276);
any timer;
reporting empty directories;
a per-file undo.

Residual risk: a deletion re-checks and then unlinks, so there is a small window between the two. If a second Worker serves the same live config and its crawl renames a new proxy onto that exact path inside the window, that proxy is lost until regeneration exists. The supported deployments have one Worker per config (per-user workers, or one all-user worker), so this needs two Workers serving one config.

Hard to reverse:

the ProxyOrphans table and the ProxyFiles.ProxyPath index;
the admin API contract under /api/v1/admin/proxy-sweeps;
two WorkKind values. Their payloads are persisted, so renaming them would break queued items.

Deletion stays manual and re-checked, so a wrong orphan rule produces a wrong report, not lost proxies.

Decision needed: where the report lives (options 1–3), and how orphans of a deleted config are scoped. Both were decided on 2026-10-26, see the next comment.

🤖 Generated with Claude Code

Decision

The maintainer decided these on 2026-10-26, on the design note above:

Where the report lives: option 1, the ProxyOrphans table that only the Worker writes. Its FK to the sweep's work item cascades, so WorkItemRetention purges old reports.
Orphans of a deleted storage config are deleted by an unscoped ProxyDeletion item that any Worker may run. Orphans of a live config stay scoped to that config, as the spec: Work queue — durable API/Worker work items with scope routing, leases and a live admin view #276 spec says.

The 2026-10-18 decision (report only, manual sweeps, 1 h grace period, a re-check before delete) is unchanged.

The note still needs architect approval from another session before ready-for-agent, so the issue stays needs-design.

🤖 Generated with Claude Code

Design approved. The work-queue note (#174 (comment)) puts each kind of write with one owner. It keeps deletion manual and re-checked, so a wrong rule produces a wrong report and loses no files. Its first caller, the three endpoints with both handlers, uses every part of the interface.

Checked against master (c0eb001):

ProxyFiles.ProxyPath still has no index (docs/schema.sql), so the note's index is needed.
The API mounts /data/proxies read-only and the Worker read-write (src/docker-compose.yml:64, :104).
ProxyStagingWriter always writes a fresh temporary file and moves it over the final file (File.Move(..., overwrite: true)). A proxy that is re-ingested therefore always gets a new mtime, so the 1 h grace period and the size/mtime re-check hold.
Giving ProxyOrphan a navigation property to WorkItem is allowed: work-items-change-only-through-queue-or-cancel exempts the Domain namespace and AnichronDbContext.
feat(worker,api): A Live Photo's video is never a memory of its own #321 and feat(worker): A Live Photo is reconciled as one unit #331 change which rows reference which proxies, but they never change how proxy paths are built. The orphan rule compares paths against ProxyPath, so neither issue affects it.

Amendments (the implementer follows these where they differ from the note):

Never remove top-level shard folders ({hash[..2]}/). A deletion removes only the {configId:N}-{hash} folder, and only once it is empty. That folder belongs to a single config, so no other Worker writes into it. A shard folder is shared by every config. An unscoped deletion (of a config that no longer exists) can run while another Worker's crawl is creating a folder in the same shard. If the deletion removed the shard folder at that moment, the crawl would fail. There are at most 256 shard folders, so keeping them costs nothing.
POST /{sweepId}/deletions returns 409 while the sweep has not succeeded, the same as GET /{sweepId}/orphans.
A scoped deletion is deleted with its config. A WorkItems.StorageConfigId row cascades on config delete, so deleting a config removes its queued or running deletion item. Orphans it had not processed yet keep a null outcome. When the admin asks again, the API sees the config is gone and enqueues an unscoped item, which processes them. This is accepted; tests cover how the API scopes the second request.

The issue's old Open questions section is answered by the decisions of 2026-10-18 and 2026-10-26.

🤖 Generated with Claude Code

feat(worker,api): Orphaned proxy files are swept on request and delet…

Navigation Menu

Repository navigation

feat(worker): Orphaned proxy file sweeper (proxies ↔ DB reconciliation) #174

Why this is separate from #33

Problem

What makes this tractable

Trap to avoid

Open questions

Related

First caller

Design

A third source: a losing attempt deletes the winner's file (left open by #263)

Objection

Design note

Decision

Design note (revised after the decision)

Superseded

Design note (rewritten on the work queue, #276)

Decision

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

feat(worker): Orphaned proxy file sweeper (proxies ↔ DB reconciliation) #174

Description

Why this is separate from #33

Problem

What makes this tractable

Trap to avoid

Open questions

Related

First caller

Design

Activity

bitbiter-dev commented on Oct 17, 2026

A third source: a losing attempt deletes the winner's file (left open by #263)

bitbiter-dev commented on Oct 18, 2026

Objection

bitbiter-dev commented on Oct 18, 2026

Design note

bitbiter-dev commented on Oct 18, 2026

Decision

bitbiter-dev commented on Oct 18, 2026

Design note (revised after the decision)

bitbiter-dev commented on Oct 18, 2026

Superseded

bitbiter-dev commented on Oct 18, 2026

bitbiter-dev commented on Oct 23, 2026

bitbiter-dev commented on Oct 25, 2026

bitbiter-dev commented on Oct 26, 2026

Design note (rewritten on the work queue, #276)

bitbiter-dev commented on Oct 26, 2026

Decision

bitbiter-dev commented on Oct 27, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions