feat(worker): Report duplicate content, never merge it (#36)

CLAUDE.md

@@ -169,6 +169,7 @@ The API reads files only through two narrow stores in `Anichron.Infrastructure/S
 - FFmpeg transcodes video with runtime GPU detection: QuickSync (`h264_qsv`) → NVENC (`h264_nvenc`) → AMF (`h264_amf`) → software (`libx264`)
 - Burst detection (**planned, not yet implemented**): will group rapid-fire sequences and assign a `PrimaryAssetId` cover. The `Burst` entity and schema exist; no detection code is wired into the Worker yet.
 - Reconciliation (**planned, Epic 4**): a periodic scan soft-deletes assets whose file is gone (interactions survive) and restores them when it reappears (#33, #34, #37). Moves are already handled at ingest (#35): when the hash matches an active row, `KnownFileReconciler` refreshes `LastSeenOnNas` if the path is unchanged (`Unchanged`), moves the row to the new path if its stored file is gone (`Moved`), and leaves it alone if the stored file still exists (`DuplicateCopy`, reported by #36). A Live Photo's MOV row gets the same treatment by its own hash. A move onto a path another row already holds, soft-deleted rows included (`PathTaken`), is logged as a warning and skipped; if a concurrent consumer takes the path after classification, the save fails and the item is retried on the next crawl. A file **edited in place** (hash unknown, but an active row holds the path: `ContentChanged`) is processed like new content and `ContentChangedWriter` updates that row: new hash, capture date, media type, `Metadata` and proxy rows (updated by type), while `Id`, interactions, `BurstId` and `PairedAssetId` are kept. A soft-deleted row at the path still blocks it (restore, #34). An edited Live Photo **MOV** keeps its stored row unchanged for now
+- Duplicate content is **reported, never merged** (#36). `SharedContentReporter` runs after each crawl cycle over the configs just crawled: it logs one `Information` summary per user (copies within one config, counted by `DuplicateCopyTally` during the crawl, and pairs of that user's active rows sharing a `ContentHash` across configs), with each copy and pair at `Debug`. Duplicates across different users are expected and never reported. Two new identical copies racing in one crawl are first reported on the next crawl
 - Failure ownership: `ProxyStagingWriter` is the single place a proxy reaches its final name (architecture rule `proxies-reach-final-name-only-in-staging-writer`) — it writes under a per-attempt `.<attempt id>.tmp` suffix (so concurrent attempts on the same content never share a temporary file), renames into place once complete, and registers the proxy on the `IngestionContext` at that moment; the proxy middlewares only supply the bytes. If a step throws, `IngestionPipelineRunner` deletes exactly those registered proxies plus its own temporary files, and removes the directory only if it is left empty — never recursively, since a concurrent attempt on the same content shares it until one of them persists. An attempt that loses a uniqueness race (`LostIngestionRace`: the active content-hash index or the one-proxy-per-asset-and-type index) first releases its paired MOV's registrations, because the winner owns those content-derived files. A clean shutdown is treated differently: the in-flight temporary file is dropped, but proxies already completed in that run are kept, and the consumer loop re-throws the cancellation instead of recording an item failure and draining the queue. Each attempt has its own `IngestionContext.AttemptId`, which names temporary files and scopes compensation; `IngestionContext.AssetId` is the row that owns the proxies, set by classification (the stored row for `ContentChanged`, the attempt id for new content). A hard kill, the storage-config cascade delete, and the old content directory of a file edited in place are out of reach in-process and belong to the orphan sweeper (#174); the old directory is deliberately not deleted after commit, because a copy of the old content elsewhere can be ingested into it in the same crawl.
 
 ### Flashback Interaction Rules

src/Anichron.Core.Tests.Unit/Data/Repository/MediaAssetRepositoryTests.cs

@@ -177,4 +177,96 @@ public async Task LoadMetadataAsync_LoadsTheAssetsMetadataRow()
         tracked.Metadata.Should().NotBeNull();
         tracked.Metadata.Width.Should().Be(640);
     }
+
+    // ==========================================================================
+    // FindSharedContentAsync
+    // ==========================================================================
+
+    private static readonly Guid Mira = Guid.Parse("aaaaaaaa-0000-0000-0000-000000000001");
+    private static readonly Guid Alice = Guid.Parse("aaaaaaaa-0000-0000-0000-000000000002");
+
+    private static UserStorageConfig ConfigOf(Guid userId)
+        => new() { Id = Guid.NewGuid(), UserId = userId, RootPath = $"/data/originals/{Guid.NewGuid():N}", IsActive = true };
+
+    private static async Task<IReadOnlyList<ContentLocation>> FindSharedContentAsync(
+        UserStorageConfig[] configs, MediaAsset[] assets, IReadOnlyCollection<Guid> crawled)
+    {
+        var ct = TestContext.Current.CancellationToken;
+        await using var seed = CreateDb(out var name);
+        seed.StorageConfigs.AddRange(configs);
+        seed.MediaAssets.AddRange(assets);
+        await seed.SaveChangesAsync(ct);
+        await using var db = AnichronDbContexts.InMemory(name);
+
+        return await new EfMediaAssetRepository(db).FindSharedContentAsync(crawled, ct);
+    }
+
+    [Fact]
+    public async Task FindSharedContentAsync_SameHashInTwoConfigsOfOneUser_ReturnsBothLocations()
+    {
+        var (home, phone) = (ConfigOf(Mira), ConfigOf(Mira));
+
+        var result = await FindSharedContentAsync(
+            [home, phone],
+            [AssetAt(home.Id, "cafe", "2023/a.jpg"), AssetAt(phone.Id, "cafe", "dcim/a.jpg"), AssetAt(home.Id, "beef", "b.jpg")],
+            [home.Id, phone.Id]);
+
+        result.Should().BeEquivalentTo(
+        [
+            new ContentLocation(Mira, "cafe", home.Id, "2023/a.jpg"),
+            new ContentLocation(Mira, "cafe", phone.Id, "dcim/a.jpg"),
+        ]);
+    }
+
+    [Fact]
+    public async Task FindSharedContentAsync_SameHashUnderTwoUsers_IsLeftOut()
+    {
+        var (mine, theirs) = (ConfigOf(Mira), ConfigOf(Alice));
+
+        var result = await FindSharedContentAsync(
+            [mine, theirs],
+            [AssetAt(mine.Id, "cafe", "a.jpg"), AssetAt(theirs.Id, "cafe", "a.jpg")],
+            [mine.Id, theirs.Id]);
+
+        result.Should().BeEmpty();
+    }
+
+    [Fact]
+    public async Task FindSharedContentAsync_ConfigNotCrawled_IsLeftOut()
+    {
+        var (home, phone) = (ConfigOf(Mira), ConfigOf(Mira));
+
+        var result = await FindSharedContentAsync(
+            [home, phone],
+            [AssetAt(home.Id, "cafe", "a.jpg"), AssetAt(phone.Id, "cafe", "a.jpg")],
+            [home.Id]);
+
+        result.Should().BeEmpty();
+    }
+
+    [Fact]
+    public async Task FindSharedContentAsync_SoftDeletedCopy_IsLeftOut()
+    {
+        var (home, phone) = (ConfigOf(Mira), ConfigOf(Mira));
+
+        var result = await FindSharedContentAsync(
+            [home, phone],
+            [AssetAt(home.Id, "cafe", "a.jpg"), AssetAt(phone.Id, "cafe", "a.jpg", softDeleted: true)],
+            [home.Id, phone.Id]);
+
+        result.Should().BeEmpty();
+    }
+
+    [Fact]
+    public void FindSharedContent_TranslatesToOneStatementScopedToTheCrawledConfigs()
+    {
+        using var db = AnichronDbContexts.NpgsqlModel();
+
+        var sql = EfMediaAssetRepository.SharedContentQuery(db, [Guid.NewGuid()]).ToQueryString();
+
+        sql.Should().Contain("GROUP BY")
+            .And.Contain("HAVING count(*)::int > 1")
+            .And.Contain("\"StorageConfigId\" = ANY (@")
+            .And.Contain("\"IsSoftDeleted\"");
+    }
 }

src/Anichron.Core/Data/Repository/MediaAssetRepository.cs

@@ -3,6 +3,8 @@
 
 namespace Anichron.Core.Data.Repository;
 
+public sealed record ContentLocation(Guid UserId, string ContentHash, Guid StorageConfigId, string FilePath);
+
 public interface IMediaAssetRepository
 {
     Task<MediaAsset?> FindByHashAsync(string contentHash, Guid storageConfigId, CancellationToken ct);
@@ -18,6 +20,11 @@ Task<IReadOnlyList<MediaAsset>> FindByHashesOrPathsAsync(
         IReadOnlyCollection<string> filePaths,
         CancellationToken ct);
 
+    /// <summary>Active rows whose content is also stored in another of the same user's configs.</summary>
+    /// <param name="storageConfigIds">Only rows in these configs are compared.</param>
+    /// <param name="ct">Cancels the query.</param>
+    Task<IReadOnlyList<ContentLocation>> FindSharedContentAsync(IReadOnlyCollection<Guid> storageConfigIds, CancellationToken ct);
+
     Task LoadMetadataAsync(MediaAsset asset, CancellationToken ct);
 
     void Add(MediaAsset asset);
@@ -43,6 +50,23 @@ public async Task<IReadOnlyList<MediaAsset>> FindByHashesOrPathsAsync(
                 && ((!a.IsSoftDeleted && contentHashes.Contains(a.ContentHash)) || filePaths.Contains(a.FilePath)))
             .ToListAsync(ct);
 
+    public async Task<IReadOnlyList<ContentLocation>> FindSharedContentAsync(
+        IReadOnlyCollection<Guid> storageConfigIds, CancellationToken ct)
+        => await SharedContentQuery(db, storageConfigIds).ToListAsync(ct);
+
+    internal static IQueryable<ContentLocation> SharedContentQuery(
+        AnichronDbContext db, IReadOnlyCollection<Guid> storageConfigIds)
+    {
+        var inScope = db.MediaAssets.Where(a => storageConfigIds.Contains(a.StorageConfigId));
+        var shared = inScope
+            .GroupBy(a => new { a.StorageConfig.UserId, a.ContentHash })
+            .Where(g => g.Count() > 1)
+            .Select(g => g.Key);
+        return inScope
+            .Where(a => shared.Any(k => k.UserId == a.StorageConfig.UserId && k.ContentHash == a.ContentHash))
+            .Select(a => new ContentLocation(a.StorageConfig.UserId, a.ContentHash, a.StorageConfigId, a.FilePath));
+    }
+
     public Task LoadMetadataAsync(MediaAsset asset, CancellationToken ct)
         => db.Entry(asset).Reference(a => a.Metadata).LoadAsync(ct);
 

src/Anichron.Worker.Tests.Unit/Crawling/SharedContentReporterTests.cs

@@ -0,0 +1,152 @@
+using Anichron.Core.Data.Repository;
+using Anichron.Core.Domain;
+using Anichron.Worker.Crawling;
+using Anichron.Worker.Ingestion.Classification;
+using Microsoft.Extensions.Logging;
+using Microsoft.Extensions.Logging.Testing;
+
+namespace Anichron.Worker.Tests.Unit.Crawling;
+
+public sealed class SharedContentReporterTests
+{
+    private static readonly Guid Mira = Guid.Parse("aaaaaaaa-0000-0000-0000-000000000001");
+    private static readonly Guid Alice = Guid.Parse("aaaaaaaa-0000-0000-0000-000000000002");
+    private static readonly Guid Home = Guid.Parse("11111111-0000-0000-0000-000000000000");
+    private static readonly Guid Phone = Guid.Parse("22222222-0000-0000-0000-000000000000");
+    private static readonly Guid Backup = Guid.Parse("33333333-0000-0000-0000-000000000000");
+
+    private static CancellationToken Ct => TestContext.Current.CancellationToken;
+
+    private static UserStorageConfig Config(Guid id, Guid userId) => new() { Id = id, UserId = userId, RootPath = "/unused" };
+
+    private sealed class TestFixture
+    {
+        public IMediaAssetRepository Repository { get; } = Substitute.For<IMediaAssetRepository>();
+        public DuplicateCopyTally Tally { get; } = new();
+        public FakeLogger<SharedContentReporter> Logger { get; } = new();
+
+        public async Task<IReadOnlyList<FakeLogRecord>> ReportAsync(
+            UserStorageConfig[] crawled, params ContentLocation[] locations)
+        {
+            Repository.FindSharedContentAsync(
+                    Arg.Is<IReadOnlyCollection<Guid>>(ids => ids.SequenceEqual(crawled.Select(c => c.Id))),
+                    Arg.Any<CancellationToken>())
+                .Returns(locations);
+
+            await new SharedContentReporter(Repository, Tally, Logger).ReportAsync(crawled, Ct);
+
+            return Logger.Collector.GetSnapshot();
+        }
+    }
+
+    private static IEnumerable<FakeLogRecord> Pairs(IEnumerable<FakeLogRecord> records)
+        => records.Where(r => r.Level == LogLevel.Debug);
+
+    private static IEnumerable<string> Summaries(IEnumerable<FakeLogRecord> records)
+        => records.Where(r => r.Level == LogLevel.Information).Select(r => r.Message);
+
+    // ==========================================================================
+    // Pairs across configs
+    // ==========================================================================
+
+    [Fact]
+    public async Task ReportAsync_ContentInTwoConfigs_LogsThePairOnceAtDebugAsync()
+    {
+        var records = await new TestFixture().ReportAsync(
+            [Config(Home, Mira), Config(Phone, Mira)],
+            new ContentLocation(Mira, "cafe", Phone, "dcim/a.jpg"),
+            new ContentLocation(Mira, "cafe", Home, "2023/a.jpg"));
+
+        Pairs(records).Should().ContainSingle().Which.Message.Should().Be(
+            "Duplicate content cafe for user aaaaaaaa-0000-0000-0000-000000000001: "
+            + "2023/a.jpg in storage config 11111111-0000-0000-0000-000000000000 and "
+            + "dcim/a.jpg in storage config 22222222-0000-0000-0000-000000000000.");
+    }
+
+    [Fact]
+    public async Task ReportAsync_ContentInThreeConfigs_LogsEachPairOnceAsync()
+    {
+        var records = await new TestFixture().ReportAsync(
+            [Config(Home, Mira), Config(Phone, Mira), Config(Backup, Mira)],
+            new ContentLocation(Mira, "cafe", Backup, "c.jpg"),
+            new ContentLocation(Mira, "cafe", Home, "a.jpg"),
+            new ContentLocation(Mira, "cafe", Phone, "b.jpg"));
+
+        Pairs(records).Select(r => (r.GetStructuredStateValue("FilePath"), r.GetStructuredStateValue("OtherFilePath")))
+            .Should().Equal(("a.jpg", "b.jpg"), ("a.jpg", "c.jpg"), ("b.jpg", "c.jpg"));
+    }
+
+    [Fact]
+    public async Task ReportAsync_TwoHashesAndTwoUsers_PairsOnlyWithinEachUsersHashAsync()
+    {
+        var records = await new TestFixture().ReportAsync(
+            [Config(Home, Mira), Config(Phone, Mira), Config(Backup, Alice)],
+            new ContentLocation(Mira, "cafe", Home, "a.jpg"),
+            new ContentLocation(Mira, "beef", Home, "x.jpg"),
+            new ContentLocation(Alice, "cafe", Backup, "theirs.jpg"),
+            new ContentLocation(Mira, "cafe", Phone, "b.jpg"),
+            new ContentLocation(Alice, "cafe", Phone, "theirs-too.jpg"),
+            new ContentLocation(Mira, "beef", Phone, "y.jpg"));
+
+        Pairs(records).Select(r => (r.GetStructuredStateValue("UserId"), r.GetStructuredStateValue("ContentHash"),
+                r.GetStructuredStateValue("FilePath"), r.GetStructuredStateValue("OtherFilePath")))
+            .Should().BeEquivalentTo(
+            [
+                (Mira.ToString(), "cafe", "a.jpg", "b.jpg"),
+                (Mira.ToString(), "beef", "x.jpg", "y.jpg"),
+                (Alice.ToString(), "cafe", "theirs-too.jpg", "theirs.jpg"),
+            ]);
+    }
+
+    // ==========================================================================
+    // Summary per user
+    // ==========================================================================
+
+    [Fact]
+    public async Task ReportAsync_CopiesWithinAndAcrossConfigs_SummarisesEachUserOnceAtInformationAsync()
+    {
+        var fx = new TestFixture();
+        fx.Tally.Count(Home);
+        fx.Tally.Count(Home);
+        fx.Tally.Count(Phone);
+        fx.Tally.Count(Backup);
+
+        var records = await fx.ReportAsync(
+            [Config(Home, Mira), Config(Phone, Mira), Config(Backup, Alice)],
+            new ContentLocation(Mira, "cafe", Home, "a.jpg"),
+            new ContentLocation(Mira, "cafe", Phone, "b.jpg"));
+
+        Summaries(records).Should().Equal(
+            "Duplicate content for user aaaaaaaa-0000-0000-0000-000000000001: "
+                + "copies within one storage config: 3, pairs across storage configs: 1.",
+            "Duplicate content for user aaaaaaaa-0000-0000-0000-000000000002: "
+                + "copies within one storage config: 1, pairs across storage configs: 0.");
+    }
+
+    [Fact]
+    public async Task ReportAsync_OnlyAPairAcrossConfigs_StillSummarisesTheUserAsync()
+    {
+        var records = await new TestFixture().ReportAsync(
+            [Config(Home, Mira), Config(Phone, Mira)],
+            new ContentLocation(Mira, "cafe", Home, "a.jpg"),
+            new ContentLocation(Mira, "cafe", Phone, "b.jpg"));
+
+        Summaries(records).Should().ContainSingle().Which.Should().EndWith(
+            "copies within one storage config: 0, pairs across storage configs: 1.");
+    }
+
+    [Fact]
+    public async Task ReportAsync_Reported_StartsTheNextCycleFromZeroAsync()
+    {
+        var fx = new TestFixture();
+        fx.Tally.Count(Home);
+
+        await fx.ReportAsync([Config(Home, Mira)]);
+
+        fx.Tally.Take(Home).Should().Be(0);
+    }
+
+    [Fact]
+    public async Task ReportAsync_NothingDuplicated_LogsNothingAsync()
+        => (await new TestFixture().ReportAsync([Config(Home, Mira), Config(Backup, Alice)])).Should().BeEmpty();
+}