Skip to content

refactor(worker): Classify each crawled file once, then one handler per outcome #254

Closed

Description

@bitbiter-dev

From the 2026-10-16 architecture review of master after #234.

Problem

The ingestion pipeline is a linear chain built for new files. Handling of content the database already knows has been bolted onto it one branch at a time:

That is three places that save, each with its own conflict handling, and a decision ("what is this file to us?") spread across several middlewares. The next issues add more branches to the same places: #250 (content changed at a known path), #33 and #34 (missing and reappearing files).

Design

Classify each crawled item once, right after hashing, then hand it to one handler per outcome.

Related: #135 (declarative middleware ordering). This issue reduces how many middlewares need ordering at all.

Acceptance

  • One classifier decides the outcome; no other middleware re-queries "is this content known?".
  • One helper recognises unique violations by index name; the duplicated checks are gone.
  • All existing pairing, idempotency, move and persistence behaviours hold, including the concurrent-consumer races.
  • Update the row when a file's content changes at the same path #250 can be implemented as one new outcome plus its handler, with no new branches in other middlewares. Say so in the PR, with the planned shape.
  • scripts/mutation-changed.sh at 100%.

Blocked by #35 (PR #249). Blocks #250.

Metadata

Metadata

Assignees

No one assigned

    Labels

    ready-for-agentFully specified, ready for an AFK agent

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions