Skip to content

Report duplicate content (report-only; never merges or deletes) #36

Closed

Description

@bitbiter-dev

Decision (supersedes the original text)

Report-only. Deduplication stays per storage config: the same bytes under two configs are two assets, which is what makes multi-user work (CLAUDE.md). This issue never merges, hides or deletes anything. It tells the user where duplicate content lives so they can clean up the NAS.

What counts as a duplicate

  1. Within one config: the same ContentHash at two paths that both exist. Only one row exists, because the partial unique index allows one active asset per hash. The second copy is seen during the crawl, and Re-link moved/renamed files and refresh LastSeenOnNas on every crawl #35 identifies it ("different path, stored path still exists").
  2. Across one user's own configs: active rows with the same ContentHash in two configs owned by the same user.

Duplicates across different users are not reported. They are expected and private.

Scope for v1.0

  • Log both kinds at Information level, structured with ContentHash, the config id(s) and the paths, once per crawl.
  • An API listing or UI for duplicates is out of scope; open a follow-up if wanted.
  • A within-config duplicate can first be reported on the crawl after a move. If the old file is gone and two new copies are processed at the same moment, both are treated as the move and the row ends up at one of them. On the next crawl the other copy sees that path still exists and is reported. Expected, not a bug.

Acceptance

  • Both kinds are detected and logged with the fields above. Each pair is logged once per crawl.
  • Nothing is written to the database.
  • scripts/mutation-changed.sh at 100%.

Blocked by #35, which produces the within-config signal.

Activity

  1. changed the title [-]4.4 Implement periodic deduplication scan[/-] [+]Implement periodic deduplication scan[/+] on May 12, 2026
  2. modified the milestones: Epic 4: NAS Reconciliation & Data Health, on May 12, 2026
  3. added
    ready-for-agentFully specified, ready for an AFK agent
    and removed
    needs-triageMaintainer needs to evaluate this issue
    on Oct 16, 2026
  4. changed the title [-]Implement periodic deduplication scan[/-] [+]Report duplicate content (report-only; never merges or deletes)[/+] on Oct 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestready-for-agentFully specified, ready for an AFK agentv1.0Required for the v1.0 release

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions