Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -153,11 +153,11 @@ The API reads files only through two narrow stores in `Anichron.Infrastructure/S
- Generates proxy types: `Thumbnail`, `FullPreview`, `WebVideo` (the `ProxyType` enum, persisted as text), and a BlurHash string for images, stored in `Metadata.Blurhash`
- Image proxies are auto-oriented once after decoding: upright pixels, orientation tag 1. `Metadata.Orientation` applies only when serving originals — never rotate a proxy by it
- Image proxy sizes are edge-based, so portrait and landscape get the same treatment: preview long edge `PreviewLongEdge` (1920), thumbnail short edge `ThumbnailShortEdge` (300), blurhash sample long edge `BlurhashSampleLongEdge` (64). Nothing is ever upscaled
- Each `IImageProxyGenerator`, and the `BlurhashEncoder`, declares a `ProxyBound`; `ImageProxyMiddleware` decodes once to `Rgb24`, shrinks the shared image to each bound largest-first, and hands every step **its own copy** at that size. The encoder's string goes on the `IngestionContext` and `NewContentWriter` writes it to `Metadata`. Generators may crop/recolour/resize their copy freely; only the middleware resizes the shared image
- Each `IImageProxyGenerator`, and the `BlurhashEncoder`, declares a `ProxyBound`; `ImageProxyMiddleware` decodes once to `Rgb24`, shrinks the shared image to each bound largest-first, and hands every step **its own copy** at that size. The encoder's string goes on the `IngestionContext` and the content writer (`ProcessedContentWriter.DescribeContent`) writes it to `Metadata`. Generators may crop/recolour/resize their copy freely; only the middleware resizes the shared image
- FFmpeg transcodes video with runtime GPU detection: QuickSync (`h264_qsv`) → NVENC (`h264_nvenc`) → AMF (`h264_amf`) → software (`libx264`)
- Burst detection (**planned, not yet implemented**): will group rapid-fire sequences and assign a `PrimaryAssetId` cover. The `Burst` entity and schema exist; no detection code is wired into the Worker yet.
- Reconciliation (**planned, Epic 4**): a periodic scan soft-deletes assets whose file is gone (interactions survive) and restores them when it reappears (#33, #34, #37). Moves are already handled at ingest (#35): when the hash matches an active row, `KnownFileReconciler` refreshes `LastSeenOnNas` if the path is unchanged (`Unchanged`), moves the row to the new path if its stored file is gone (`Moved`), and leaves it alone if the stored file still exists (`DuplicateCopy`, reported by #36). A Live Photo's MOV row gets the same treatment by its own hash. A move onto a path another row already holds, soft-deleted rows included (`PathTaken`), is logged as a warning and skipped; if a concurrent consumer takes the path after classification, the save fails and the item is retried on the next crawl
- Failure ownership: `ProxyStagingWriter` is the single place a proxy reaches its final name (architecture rule `proxies-reach-final-name-only-in-staging-writer`) — it writes under a per-attempt `.<attempt id>.tmp` suffix (so concurrent attempts on the same content never share a temporary file), renames into place once complete, and registers the proxy on the `IngestionContext` at that moment; the proxy middlewares only supply the bytes. If a step throws, `IngestionPipelineRunner` deletes exactly those registered proxies plus its own temporary files, and removes the directory only if it is left empty — never recursively, since a concurrent attempt on the same content shares it until one of them persists. An attempt that loses a uniqueness race (`LostIngestionRace`: the active content-hash index or the one-proxy-per-asset-and-type index) first releases its paired MOV's registrations, because the winner owns those content-derived files. A clean shutdown is treated differently: the in-flight temporary file is dropped, but proxies already completed in that run are kept, and the consumer loop re-throws the cancellation instead of recording an item failure and draining the queue. A hard kill or the storage-config cascade delete is out of reach in-process and belongs to the orphan sweeper (#174).
- Reconciliation (**planned, Epic 4**): a periodic scan soft-deletes assets whose file is gone (interactions survive) and restores them when it reappears (#33, #34, #37). Moves are already handled at ingest (#35): when the hash matches an active row, `KnownFileReconciler` refreshes `LastSeenOnNas` if the path is unchanged (`Unchanged`), moves the row to the new path if its stored file is gone (`Moved`), and leaves it alone if the stored file still exists (`DuplicateCopy`, reported by #36). A Live Photo's MOV row gets the same treatment by its own hash. A move onto a path another row already holds, soft-deleted rows included (`PathTaken`), is logged as a warning and skipped; if a concurrent consumer takes the path after classification, the save fails and the item is retried on the next crawl. A file **edited in place** (hash unknown, but an active row holds the path: `ContentChanged`) is processed like new content and `ContentChangedWriter` updates that row: new hash, capture date, media type, `Metadata` and proxy rows (updated by type), while `Id`, interactions, `BurstId` and `PairedAssetId` are kept. A soft-deleted row at the path still blocks it (restore, #34). An edited Live Photo **MOV** keeps its stored row unchanged for now
- Failure ownership: `ProxyStagingWriter` is the single place a proxy reaches its final name (architecture rule `proxies-reach-final-name-only-in-staging-writer`) — it writes under a per-attempt `.<attempt id>.tmp` suffix (so concurrent attempts on the same content never share a temporary file), renames into place once complete, and registers the proxy on the `IngestionContext` at that moment; the proxy middlewares only supply the bytes. If a step throws, `IngestionPipelineRunner` deletes exactly those registered proxies plus its own temporary files, and removes the directory only if it is left empty — never recursively, since a concurrent attempt on the same content shares it until one of them persists. An attempt that loses a uniqueness race (`LostIngestionRace`: the active content-hash index or the one-proxy-per-asset-and-type index) first releases its paired MOV's registrations, because the winner owns those content-derived files. A clean shutdown is treated differently: the in-flight temporary file is dropped, but proxies already completed in that run are kept, and the consumer loop re-throws the cancellation instead of recording an item failure and draining the queue. Each attempt has its own `IngestionContext.AttemptId`, which names temporary files and scopes compensation; `IngestionContext.AssetId` is the row that owns the proxies, set by classification (the stored row for `ContentChanged`, the attempt id for new content). A hard kill, the storage-config cascade delete, and the old content directory of a file edited in place are out of reach in-process and belong to the orphan sweeper (#174); the old directory is deliberately not deleted after commit, because a copy of the old content elsewhere can be ingested into it in the same crawl.

### Flashback Interaction Rules

Expand All @@ -181,9 +181,9 @@ Each user has a private set of `UserStorageConfig` entries pointing to NAS roots

### Worker Processing Model

Processing uses a bounded-concurrency `Channel<T>` pipeline. The crawler produces file paths; N consumer tasks process them concurrently. Default concurrency: `Worker:MaxConcurrentFiles = 4`. Processing is always idempotent — files already in the DB (matched by `(StorageConfigId, ContentHash)`) skip extraction and proxy generation; only the row's `FilePath`/`LastSeenOnNas` are updated (see Reconciliation above).
Processing uses a bounded-concurrency `Channel<T>` pipeline. The crawler produces file paths; N consumer tasks process them concurrently. Default concurrency: `Worker:MaxConcurrentFiles = 4`. Processing is always idempotent — files already in the DB (matched by `(StorageConfigId, ContentHash)`) skip extraction and proxy generation; only the row's `FilePath`/`LastSeenOnNas` are updated, and a file whose bytes changed at a stored path updates that row in place (see Reconciliation above).

**Classify once, then one handler per outcome (#254).** Right after hashing, `ClassificationMiddleware` asks `IngestionClassifier`, which loads every relevant row in one query (`FindByHashesOrPathsAsync`: active rows by either half's hash, any row at either half's path), and gives each half a `FileOutcome` (`New`, `Unchanged`, `Moved`, `DuplicateCopy`, `PathTaken`) plus a `PairingAction` for the item (`None`, `AttachMov`, `BackfillMovProxy`). No other step asks the database whether content is known, except `IngestionSaver` checking whether a concurrent consumer's row landed after a lost insert race. A `New` primary continues through EXIF, proxies and `PersistenceMiddleware`, which hands the write to the `IOutcomeWriter` for that outcome (`NewContentWriter` inserts). Every other outcome stops the item after `LivePhotoPairer` and `KnownFileReconciler` have updated the stored rows. All saves go through `IngestionSaver`, and unique violations are recognised by index name with `IsUniqueViolation` (`Anichron.Core/Data/UniqueViolations.cs`). A new outcome, such as content changed at a known path (#250), is a classifier branch, a `FileOutcome` value and its handler or writer; no other middleware changes. When a Live Photo's HEIC is already stored without its MOV (the halves arrived in different crawls), `LivePhotoPairer` attaches the MOV to the stored row and commits before the halves are reconciled; `PairedMovResolver` reuses the MOV row the classifier found rather than inserting a duplicate. The paired MOV is its own asset and owns its `WebVideo` proxy, written into the MOV's own content-hash directory by `PairedMovProxyWriter` only when that MOV has none yet — on first ingest, on late pairing, and on any later crawl of an already-paired HEIC whose MOV still lacks it (self-heal).
**Classify once, then one handler per outcome (#254).** Right after hashing, `ClassificationMiddleware` asks `IngestionClassifier`, which loads every relevant row in one query (`FindByHashesOrPathsAsync`: active rows by either half's hash, any row at either half's path), and gives each half a `FileOutcome` (`New`, `ContentChanged`, `Unchanged`, `Moved`, `DuplicateCopy`, `PathTaken`) plus a `PairingAction` for the item (`None`, `AttachMov`, `BackfillMovProxy`). No other step asks the database whether content is known, except `IngestionSaver` checking whether a concurrent consumer's row landed after a lost insert race. A `New` or `ContentChanged` primary continues through EXIF, proxies and `PersistenceMiddleware`, which hands the write to the `IOutcomeWriter` for that outcome (`NewContentWriter` inserts, `ContentChangedWriter` updates the stored row in place; both share `ProcessedContentWriter`). Every other outcome stops the item after `LivePhotoPairer` and `KnownFileReconciler` have updated the stored rows. All saves go through `IngestionSaver`, and unique violations are recognised by index name with `IsUniqueViolation` (`Anichron.Core/Data/UniqueViolations.cs`). A new outcome, as content changed at a known path (#250) was, is a classifier branch, a `FileOutcome` value and its handler or writer; no other middleware changes. When a Live Photo's HEIC is already stored without its MOV (the halves arrived in different crawls), `LivePhotoPairer` attaches the MOV to the stored row and commits before the halves are reconciled; `PairedMovResolver` reuses the MOV row the classifier found rather than inserting a duplicate. The paired MOV is its own asset and owns its `WebVideo` proxy, written into the MOV's own content-hash directory by `PairedMovProxyWriter` only when that MOV has none yet — on first ingest, on late pairing, and on any later crawl of an already-paired HEIC whose MOV still lacks it (self-heal).

## Infrastructure

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -147,4 +147,34 @@ public void AddProxyFiles_TracksEveryRowForInsertEvenWithItsIdPreset()

proxies.Select(proxy => db.Entry(proxy).State).Should().Equal(EntityState.Added, EntityState.Added);
}

[Fact]
public void RemoveProxyFile_TracksTheRowForDelete()
{
using var db = AnichronDbContexts.InMemory();
var proxy = new ProxyFile { Id = Guid.NewGuid(), AssetId = Guid.NewGuid(), ProxyType = ProxyType.WebVideo };
db.ProxyFiles.Attach(proxy);

new EfMediaAssetRepository(db).RemoveProxyFile(proxy);

db.Entry(proxy).State.Should().Be(EntityState.Deleted);
}

[Fact]
public async Task LoadMetadataAsync_LoadsTheAssetsMetadataRow()
{
var ct = TestContext.Current.CancellationToken;
var asset = AssetAt(ConfigA, "cafe", "a.jpg");
asset.Metadata = new Metadata { AssetId = asset.Id, Width = 640, Height = 480 };
await using var seed = CreateDb(out var name);
seed.MediaAssets.Add(asset);
await seed.SaveChangesAsync(ct);
await using var db = AnichronDbContexts.InMemory(name);
var tracked = await db.MediaAssets.SingleAsync(a => a.Id == asset.Id, ct);

await new EfMediaAssetRepository(db).LoadMetadataAsync(tracked, ct);

tracked.Metadata.Should().NotBeNull();
tracked.Metadata.Width.Should().Be(640);
}
}
9 changes: 9 additions & 0 deletions src/Anichron.Core/Data/Repository/MediaAssetRepository.cs
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,11 @@ Task<IReadOnlyList<MediaAsset>> FindByHashesOrPathsAsync(
IReadOnlyCollection<string> filePaths,
CancellationToken ct);

Task LoadMetadataAsync(MediaAsset asset, CancellationToken ct);

void Add(MediaAsset asset);
void AddProxyFiles(IEnumerable<ProxyFile> proxyFiles);
void RemoveProxyFile(ProxyFile proxyFile);
}

public sealed class EfMediaAssetRepository(AnichronDbContext db) : IMediaAssetRepository
Expand All @@ -40,9 +43,15 @@ public async Task<IReadOnlyList<MediaAsset>> FindByHashesOrPathsAsync(
&& ((!a.IsSoftDeleted && contentHashes.Contains(a.ContentHash)) || filePaths.Contains(a.FilePath)))
.ToListAsync(ct);

public Task LoadMetadataAsync(MediaAsset asset, CancellationToken ct)
=> db.Entry(asset).Reference(a => a.Metadata).LoadAsync(ct);

public void Add(MediaAsset asset)
=> db.MediaAssets.Add(asset);

public void AddProxyFiles(IEnumerable<ProxyFile> proxyFiles)
=> db.ProxyFiles.AddRange(proxyFiles);

public void RemoveProxyFile(ProxyFile proxyFile)
=> db.ProxyFiles.Remove(proxyFile);
}
Original file line number Diff line number Diff line change
Expand Up @@ -218,7 +218,6 @@ public void AddWorkerHostedServices_RegistersFourHostedServices()
{ typeof(IngestionClassifier), typeof(IngestionClassifier), ServiceLifetime.Scoped },
{ typeof(LivePhotoPairer), typeof(LivePhotoPairer), ServiceLifetime.Scoped },
{ typeof(KnownFileReconciler), typeof(KnownFileReconciler), ServiceLifetime.Scoped },
{ typeof(IOutcomeWriter), typeof(NewContentWriter), ServiceLifetime.Scoped },
{ typeof(IProcessLauncher), typeof(SystemProcessLauncher), ServiceLifetime.Singleton },
{ typeof(IVideoProcessor), typeof(FfmpegVideoProcessor), ServiceLifetime.Singleton },
{ typeof(IImageProcessor), typeof(ImageSharpProcessor), ServiceLifetime.Singleton },
Expand All @@ -236,6 +235,18 @@ public void AddIngestionServices_RegistersTheImplementationWithItsLifetime(Type
.Which.Should().Match<ServiceDescriptor>(d => d.ImplementationType == implementation && d.Lifetime == lifetime);
}

[Fact]
public void AddIngestionServices_RegistersAScopedWriterForEveryOutcomeThatNeedsProcessing()
{
var services = new ServiceCollection();

services.AddIngestionServices();

var writers = services.Where(d => d.ServiceType == typeof(IOutcomeWriter)).ToList();
writers.Select(d => d.ImplementationType).Should().BeEquivalentTo([typeof(NewContentWriter), typeof(ContentChangedWriter)]);
writers.Should().OnlyContain(d => d.Lifetime == ServiceLifetime.Scoped);
}

[Fact]
public void AddWorkerCoreServices_RegistersTheSystemTimeProvider()
{
Expand Down
Loading
Loading