Repository navigation
refactor(worker): Classify each crawled file once, then one handler per outcome #254
Copy link
Copy link
Closed
Labels
ready-for-agentFully specified, ready for an AFK agentFully specified, ready for an AFK agent
Description
From the 2026-10-16 architecture review of master after #234.
Problem
The ingestion pipeline is a linear chain built for new files. Handling of content the database already knows has been bolted onto it one branch at a time:
LivePhotoPairingMiddleware(order 15) attaches a late MOV to a stored HEIC and saves.IdempotencyCheckMiddleware(order 20) skips known content. With Re-link moved/renamed files and refresh LastSeenOnNas on every crawl #35 (PR feat: Re-link moved files and refresh LastSeenOnNas on every crawl (#35) #249) it also refreshesLastSeenOnNas, re-links moved rows, saves each Live Photo half separately, and catches FilePath unique violations. To stop a failed save being retried, it puts the row's old values back.PersistenceMiddleware(order 60) inserts, and handles the content-hash unique race with its own copy of the "is this unique violation" check.
That is three places that save, each with its own conflict handling, and a decision ("what is this file to us?") spread across several middlewares. The next issues add more branches to the same places: #250 (content changed at a known path), #33 and #34 (missing and reappearing files).
Design
Classify each crawled item once, right after hashing, then hand it to one handler per outcome.
- A classifier loads the relevant rows in one go (by content hash and by path, for both Live Photo halves) and returns a closed set of outcomes:
NewUnchangedMovedDuplicateCopyContentChanged(Update the row when a file's content changes at the same path #250)CompletesLivePhotoPair
NewandContentChangedcontinue through EXIF extraction, proxies and persistence. Every other outcome goes to a small handler that updates the row and stops the item.- Saving and unique-violation handling live in one place, e.g. a single
IsUniqueViolation(DbUpdateException, string indexName)next toMediaAssetIndexNames. No handler restores entity values to work around EF change tracking. - Behaviour stays exactly as Re-link moved/renamed files and refresh LastSeenOnNas on every crawl #35 and fix(worker): Pair live photos whose HEIC and MOV arrive in different crawls #217 (late Live Photo pairing) define it. Their tests are the safety net and keep passing with only seam changes.
Related: #135 (declarative middleware ordering). This issue reduces how many middlewares need ordering at all.
Acceptance
- One classifier decides the outcome; no other middleware re-queries "is this content known?".
- One helper recognises unique violations by index name; the duplicated checks are gone.
- All existing pairing, idempotency, move and persistence behaviours hold, including the concurrent-consumer races.
- Update the row when a file's content changes at the same path #250 can be implemented as one new outcome plus its handler, with no new branches in other middlewares. Say so in the PR, with the planned shape.
-
scripts/mutation-changed.shat 100%.
Metadata
Metadata
Assignees
Labels
ready-for-agentFully specified, ready for an AFK agentFully specified, ready for an AFK agent
Activity
refactor(worker): Classify each crawled file once, then one handler p…