Skip to content

Two cleanup paths never wired up: unbounded DownloadProcessingJobs; orphaned downloads never terminalized #640

@kevinheneveld

Description

@kevinheneveld

Two cleanup paths exist in the codebase but are never invoked, so state accumulates unbounded on long-running instances.

3a — DownloadProcessingJobs grows unbounded. IDownloadProcessingJobService.CleanupOldJobsAsync(retentionDays) is implemented (EfDownloadProcessingJobRepository.CleanupOldJobsAsync) but has no caller. The table accumulates completed/failed rows indefinitely, and every queue-snapshot reconciliation that queries it pays for the bloat.
PR incoming: a small hosted DownloadProcessingJobCleanupService that calls CleanupOldJobsAsync on a schedule (shortly after startup, then daily), with tests. Already implemented and running locally.

3b — orphaned downloads are never terminalized. The orphan branch in DownloadQueueService.GetQueueSnapshotAsync only logs ("keeping records for resilient monitoring/import handling") and never moves long-dead rows to a terminal state, so they linger in the queue-matching pool indefinitely.

Design care if we act on 3b:

  • preserve the existing empty-client guard (a temporarily-unreachable client returning 0 items must not mass-terminalize);
  • only terminalize an allowlist of "should be in the client but isn't" states (Queued/Paused) — never Ready/ImportPending/ImportBlocked, which are legitimately absent and would destroy pending imports;
  • key off continuous absence (missing-since), not StartedAt, so a single partial snapshot can't false-terminalize.

Open question for maintainers: should orphans ever auto-terminalize, or stay user-driven? Given the deliberate log-only choice, an opt-in setting (default off) seems safest.

Related: #631 (downloads getting Import Blocked) for 3b context. Surfaced during a scale-testing discussion in Discord.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No fields configured for Bug.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions