Repository navigation
feat(worker): Crawl as a work item per storage config, with scope routing #281
Description
Parent
What to build
Crawling becomes visible, controllable work. Each storage config's crawl is a Crawl work item, due at the crawl interval, with live progress: files found, processed and failed.
Scope routing ships here, because the crawl is its first caller:
- a per-user worker (
WORKER__USER) claims only items scoped to its user's storage configs, plus unscoped items; - an all-user worker claims everything.
The Worker's own crawl timer is removed. A crawl on startup now comes from due-ness.
Acceptance criteria
-
Crawlis a periodic kind scoped to one storage config, withDedupKeycrawl:<config id>, due after the existing crawl interval (default 4 h). - Each worker schedules crawls only for the configs it serves. An inactive config is not scheduled.
- The claim filters by scope: per-user workers take their own configs and unscoped items, all-user workers take everything. Covered by a contract test on real PostgreSQL with two workers of different scopes.
- The crawl handler runs today's crawl of that config through the existing ingestion pipeline, with unchanged per-file concurrency, crawl exclusions and failure handling. It reports files found, processed and failed as progress (total unknown while walking).
- The Worker's own crawl loop and its timer are gone.
- After a restart that outlasted the crawl interval, the crawl runs straight away.
- A graceful shutdown mid-crawl releases the item without a stall, and the next run converges. Ingestion is already idempotent by content hash.
- A test shows that a second run of the crawl handler converges.
- CLAUDE.md's Worker sections describe crawling as a work item.
- 100% mutation score outside PostgreSQL-only members, which get ADR 0005 rows.
First caller
The Crawl kind, which is also the first caller of scope routing.
Design
Decided with the maintainer on 2026-10-18 (#276: crawls are work items, scope by storage config, one item at a time per worker). Per-file retry stays with #132.
Activity
github-actions commentedon Oct 18, 2026
Moved back to
needs-triage:docs/agents/design-process.md("What every issue carries") requires these parts beforeready-for-agent:Add them, then label the issue
ready-for-agentagain.bitbiter-dev commentedon Oct 18, 2026
Note from #282
#282 stops one storage config's failed crawl from taking the Worker down.
Worker.CrawlAllAsyncnow logs the failure and moves on to the next config. One crash path is still open, and this issue should close it when the crawl moves onto the queue:GetAllActiveAsync/GetActiveByUserIdAsyncruns outside the per-config catch. If it throws (the database is unreachable, for example), the exception escapesRecurringWork.RunAsync, faults theBackgroundService, and the defaultStopHoststops the Worker, which then crash-loops underrestart: unless-stopped.With the queue, scheduling crawls is the scheduler's job, so a failing query there must not fault the hosted service either. Suggested acceptance line:
test(worker): Kill the two surviving mutants in the duplicate report (#…
feat(worker): Crawl each storage config as a work item, with scope ro…