Add per-file partial-credit progress to the runtime metrics#868
Merged
Conversation
Fold in-flight files' partial progress into the byte-weighted progress.ratio and eta.seconds gauges so a long scan or re-encode advances the aggregate continuously instead of only at completion. - Metrics: per-file FileSink carrying byte weight and an atomic permyriad fraction, a lock-free in-flight registry, and a ThreadLocal current-file sink captured at each tool call site. - MediaTool.ExecuteStreamStdOut streams stdout for progress while capturing stderr for failure logging. - Instrument the long ops only: ffprobe full-file scan (pts over duration), ffmpeg re-encode (-progress pipe:1), HandBrake re-encode (--json Working.Progress). Fast remux/copy/setts stay at zero until completion. - Tests for the fold math and the three tool progress parsers. - Convert all README and HISTORY inline URIs to reference-style links, grouped shields/internal/external and alphabetized. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This pull request enhances PlexCleaner’s runtime metrics (System.Diagnostics.Metrics) to report smoother weighted progress by awarding partial credit to in-flight files based on long-running tool progress, improving progress.ratio and eta.seconds stability during scans and re-encodes.
Changes:
- Adds a per-file in-flight “sink” (bytes + atomic fraction) and folds partial in-flight credit into
Metrics.ComputeProgress. - Streams tool progress output for ffmpeg (
-progress pipe:1) and HandBrake (--json) and parses progress lines to update the current file’s fraction. - Updates docs (README/HISTORY) to describe partial-credit progress and standardizes links to reference-style; adds/updates unit tests for progress parsing and progress folding.
Reviewed changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| README.md | Documents partial-credit runtime progress; converts many links to reference-style and reorganizes link refs. |
| HISTORY.md | Updates v3.22 notes to mention partial-credit progress and dotnet-monitor compatibility; converts links to reference-style. |
| PlexCleaner/Metrics.cs | Introduces per-file FileSink, in-flight registry, ThreadLocal current sink, and partial-credit folding in ComputeProgress. |
| PlexCleaner/MediaTool.cs | Adds ExecuteStreamStdOut and an error-logging overload to support stdout progress streaming with stderr capture. |
| PlexCleaner/ProcessDriver.cs | Passes per-file byte size into Metrics.FileStarted to seed in-flight weighting. |
| PlexCleaner/ProcessFile.cs | Reports ffprobe packet-scan progress as a fraction of media duration. |
| PlexCleaner/Process.cs | Seeds per-file duration (microseconds) for ffmpeg progress fraction computation. |
| PlexCleaner/FfMpegBuilder.cs | Adds -progress pipe:1 builder helper. |
| PlexCleaner/FfMpegTool.cs | Parses ffmpeg -progress lines and reports per-file progress fraction during encode. |
| PlexCleaner/HandBrakeBuilder.cs | Adds --json global option helper. |
| PlexCleaner/HandBrakeTool.cs | Enables --json, parses HandBrake progress, and reports per-file progress fraction while streaming stdout. |
| PlexCleanerTests/MetricsTests.cs | Updates tests for new FileStarted(size) and adds coverage for partial-credit folding and clamping. |
| PlexCleanerTests/ToolProgressParsingTests.cs | Adds unit tests for ffmpeg and HandBrake progress line parsing. |
Packets arrive in demux order with interleaved audio/video and reordered B-frame timestamps, so raw per-packet PtsTime is non-monotonic. Report the running max pts over duration so the in-flight scan fraction advances without backward jitter. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Complete one file and start a second that stays in flight, instead of starting two files on one thread, so the test matches the ThreadLocal current-file semantics, and clear the in-flight sink at the end. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
DurationUs is written on the worker thread and read per progress line on the tool pipe thread, so back it with a Volatile.Read getter and a SetDurationUs writer, matching the permyriad fraction pattern. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Runtime metrics v2 (issue #848 follow-on). v1 credited a file's whole byte weight only at completion, so
progress.ratiostaircased andeta.secondsjumped. v2 folds partial credit for in-flight files: each in-flight file carries a fraction in [0,1] driven by its currently-running long tool, andComputeProgressaddsSum(inflight_bytes * fraction)to the completed total. Metrics-only, no state file, no per-file detail exposed.Changes
FileSink(byte weight + atomic permyriad fraction), a lock-free in-flight registry, and aThreadLocalcurrent-file sink captured at each tool call site so a CliWrap pipe-thread closure can report progress without touching the ThreadLocal. Completion still credits the whole size (no double counting); the sink is removed and the ThreadLocal cleared in the driverfinally.MediaTool.ExecuteStreamStdOut: streams stdout line-by-line for progress while capturing stderr for failure logging.-progress pipe:1), HandBrake re-encode (--jsonWorking.Progress). Fast remux/copy/setts stay at 0 until completion.No version bump (stays 3.22). No new NuGet package (Metrics + ConcurrentDictionary + ThreadLocal are BCL).
Verification
TreatWarningsAsErrors/AnalysisMode=Alldotnet format style --verify-no-changes --severity=infoclean, CSharpier cleanFollow-up (not in this PR)
Develop Docker build, then regression testing with
dotnet-counters monitor --counters PlexCleaner.Processto confirmprogress.ratioclimbs smoothly andeta.secondsdecays smoothly during real scans/re-encodes.🤖 Generated with Claude Code