[bitbiter-dev]

Closes #281. Part of #276.

What changes

Crawling is now visible, controllable work: each storage config's crawl is a Crawl work item, and the Worker's own crawl timer is gone.

Scope routing

• WorkItems.StorageConfigId (nullable, cascading on config delete) is set from a new optional IWorkPayload.StorageConfigId.
• A per-user Worker (WORKER__USER) claims unscoped items and items of its user's configs. An all-user Worker claims everything.
• The served user is WorkerState.ResolvedUserId, read at each claim and each scheduler tick and never cached at construction, because hosted services are built before WorkerInitializer resolves it.

Scheduling

• WorkScheduler now asks each IWorkScheduleSource for its schedules at every tick.
• A fixed WorkSchedule<TPayload> is its own only source, so token cleanup and retention behave as before.
• CrawlSchedules returns one crawl:<config id> schedule per active config the Worker serves, due Worker:CrawlIntervalHours (4 h) after the last crawl of that config finished.
• The first crawl after a start comes from due-ness, so a restart that outlasted the interval crawls straight away.

The crawl

• CrawlHandler runs today's crawl of the config through FileIngestionPipeline, with unchanged per-file concurrency, exclusions and failure handling.
• A missing or inactive config is skipped, and the result says why.
• A missing root fails only that config's item, so fix(worker): One storage config with a missing root stops the whole Worker #282's isolation still holds.
• A shutdown propagates, and the executor releases the item without a stall.
• The pipeline counts items found, processed and failed (a Live Photo pair is one item) under one lock, and reports them as progress with no total until the walk ends. The final counts are the item's result.

Merge with #36 (decided with the maintainer)

#36 (#291) reported duplicate content once, after the crawl loop had crawled every config. That loop no longer exists, so each Crawl item now reports for its own config:

• Within-config copies: from this Worker's tally for that config.
• Cross-config pairs: pairs between this config and the same user's other active configs.

The counts go into the crawl's result (Duplicates), and a failed report is logged without failing the crawl. A pair across two configs is reported by each of their crawls.

Deliberately not here

• Cancel (feat: Cancel queued and running work items #280) and the admin list and SSE (feat(api): Admin work-item list, detail and live SSE stream #279).
• One item at a time per Worker: this follows spec: Work queue — durable API/Worker work items with scope routing, leases and a live admin view #276, so a long crawl delays token cleanup and retention on the same Worker until it finishes. Both tolerate the delay.

Verification

• dotnet test -p:TestSuite=All after merging master: 1,475 unit and 35 integration tests, all passing.
• New contract tests on real PostgreSQL:
  • a per-user Worker claims only its configs and unscoped items, and an all-user Worker claims everything;
  • deleting a config removes its items and fences the running one;
  • enqueue stores the scope.
• Unit tests:
  • the executor claims for the user resolved at that moment;
  • CrawlSchedules picks the per-user or all-user configs, with the user resolved after construction;
  • the crawl handler: skips, a failing root, shutdown, and the duplicate report and its failure;
  • pipeline counts and progress, including a failed item;
  • a second crawl over the same folder converges on the same counts.
• Mutation testing: a scripts/mutation-changed.sh run over every production file changed before the merge reached 100.00%. The first CI sweep after the merge scored 99.86%, with two survivors in the reworked SharedContentReporter. Both are killed in ca0bb0b: the config ids are a Union, whose order a test pins, and a crawl with only cross-config pairs is now checked to be summarised.
• mutation-guard.sh, derive-db-docs.sh --check and dotnet format --verify-no-changes pass.

🤖 Generated with Claude Code