Repository navigation
feat(worker): Report duplicate content, never merge it (#36) - #291
Merged
Merged
Conversation
Within a config, a copy of already-ingested content is now logged at Information with its content hash, storage config and both paths. Across one user's configs, SharedContentReporter runs after each crawl cycle over the configs just crawled and logs every pair of active rows that share a content hash. Duplicates across different users are not reported. A failed report is logged and never stops the crawl loop. Nothing is written to the database. Two new identical copies racing in the same crawl are first reported on the next crawl, the same delay the issue accepts for moves. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…bug (#36) A library with a mirrored config produced one Information line per duplicate on every crawl cycle. Each copy and pair is now logged at Debug, and the end-of-cycle report logs one Information summary per user: copies within one storage config, counted by DuplicateCopyTally while the crawl runs, and pairs across storage configs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #36. Follow-up for a per-user duplicates view: #290.
What
Report-only, as decided on #36: nothing is merged, hidden, deleted or written to the database.
KnownFileReconcileralready classified a second copy asDuplicateCopy. It now logs the copy atDebugwith its content hash, its config and both paths, and counts it inDuplicateCopyTally, a singleton that is safe to use from the concurrent consumers.Worker.CrawlAllAsyncrunsSharedContentReporterover the configs it has just crawled. One read-only query (IMediaAssetRepository.FindSharedContentAsync) finds active rows of the same user that share aContentHash, and every pair is logged atDebug.Informationline per user per cycle with the number of copies within one config and the number of pairs across configs. A user with nothing duplicated gets no line.Decisions (agreed in session)
Debugwith oneInformationsummary per user. Per-pairInformationlines would repeat thousands of lines every cycle for a mirrored config. A real surface for duplicates is spec: Per-user duplicates view — track duplicate content and show it to its owner #290.Query
Npgsql output, checked through
ToQueryString: one statement,StorageConfigId = ANY(@ids), with anEXISTSthat groups by(UserId, ContentHash)and hasHAVING count(*) > 1. The partial unique index allows one active row per hash per config, so a count above 1 means at least two configs.Verification
dotnet test: all green (Worker 549, Core 165, API 521, Infrastructure 61, Architecture 25, Testing 30).scripts/mutation-changed.sh: 100.00% (99 killed, 0 survived) overKnownFileReconciler,ClassificationMiddleware,DuplicateCopyTally,SharedContentReporter,Worker,MediaAssetRepositoryandWorkerServiceCollectionExtensions. No new Stryker directives or exclusions./code-review(medium): no correctness bugs. Its one finding, log volume, led to the Debug + summary change.🤖 Generated with Claude Code