Skip to content

Add per-file partial-credit progress to the runtime metrics#868

Merged
ptr727 merged 4 commits into
developfrom
feature/metrics-v2-partial-progress
Jul 19, 2026
Merged

Add per-file partial-credit progress to the runtime metrics#868
ptr727 merged 4 commits into
developfrom
feature/metrics-v2-partial-progress

Conversation

@ptr727

@ptr727 ptr727 commented Jul 19, 2026

Copy link
Copy Markdown
Owner

Summary

Runtime metrics v2 (issue #848 follow-on). v1 credited a file's whole byte weight only at completion, so progress.ratio staircased and eta.seconds jumped. v2 folds partial credit for in-flight files: each in-flight file carries a fraction in [0,1] driven by its currently-running long tool, and ComputeProgress adds Sum(inflight_bytes * fraction) to the completed total. Metrics-only, no state file, no per-file detail exposed.

Changes

  • Metrics: per-file FileSink (byte weight + atomic permyriad fraction), a lock-free in-flight registry, and a ThreadLocal current-file sink captured at each tool call site so a CliWrap pipe-thread closure can report progress without touching the ThreadLocal. Completion still credits the whole size (no double counting); the sink is removed and the ThreadLocal cleared in the driver finally.
  • MediaTool.ExecuteStreamStdOut: streams stdout line-by-line for progress while capturing stderr for failure logging.
  • Instrumented long ops only: ffprobe full-file scan (pts/duration), ffmpeg re-encode (-progress pipe:1), HandBrake re-encode (--json Working.Progress). Fast remux/copy/setts stay at 0 until completion.
  • Docs: README Runtime Metrics section notes partial-credit progress + dotnet-counters/dotnet-monitor recipes; HISTORY 3.22 entry. All README/HISTORY inline URIs converted to reference-style links, grouped shields/internal/external and alphabetized.

No version bump (stays 3.22). No new NuGet package (Metrics + ConcurrentDictionary + ThreadLocal are BCL).

Verification

  • Build analyzer-clean, 0 warnings under TreatWarningsAsErrors/AnalysisMode=All
  • dotnet format style --verify-no-changes --severity=info clean, CSharpier clean
  • 241/241 tests pass, incl. new fold-math and tool progress-parser tests
  • markdownlint + cspell clean

Follow-up (not in this PR)

Develop Docker build, then regression testing with dotnet-counters monitor --counters PlexCleaner.Process to confirm progress.ratio climbs smoothly and eta.seconds decays smoothly during real scans/re-encodes.

🤖 Generated with Claude Code

Fold in-flight files' partial progress into the byte-weighted
progress.ratio and eta.seconds gauges so a long scan or re-encode
advances the aggregate continuously instead of only at completion.

- Metrics: per-file FileSink carrying byte weight and an atomic
  permyriad fraction, a lock-free in-flight registry, and a
  ThreadLocal current-file sink captured at each tool call site.
- MediaTool.ExecuteStreamStdOut streams stdout for progress while
  capturing stderr for failure logging.
- Instrument the long ops only: ffprobe full-file scan (pts over
  duration), ffmpeg re-encode (-progress pipe:1), HandBrake
  re-encode (--json Working.Progress). Fast remux/copy/setts stay
  at zero until completion.
- Tests for the fold math and the three tool progress parsers.
- Convert all README and HISTORY inline URIs to reference-style
  links, grouped shields/internal/external and alphabetized.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings July 19, 2026 03:36

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This pull request enhances PlexCleaner’s runtime metrics (System.Diagnostics.Metrics) to report smoother weighted progress by awarding partial credit to in-flight files based on long-running tool progress, improving progress.ratio and eta.seconds stability during scans and re-encodes.

Changes:

  • Adds a per-file in-flight “sink” (bytes + atomic fraction) and folds partial in-flight credit into Metrics.ComputeProgress.
  • Streams tool progress output for ffmpeg (-progress pipe:1) and HandBrake (--json) and parses progress lines to update the current file’s fraction.
  • Updates docs (README/HISTORY) to describe partial-credit progress and standardizes links to reference-style; adds/updates unit tests for progress parsing and progress folding.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
README.md Documents partial-credit runtime progress; converts many links to reference-style and reorganizes link refs.
HISTORY.md Updates v3.22 notes to mention partial-credit progress and dotnet-monitor compatibility; converts links to reference-style.
PlexCleaner/Metrics.cs Introduces per-file FileSink, in-flight registry, ThreadLocal current sink, and partial-credit folding in ComputeProgress.
PlexCleaner/MediaTool.cs Adds ExecuteStreamStdOut and an error-logging overload to support stdout progress streaming with stderr capture.
PlexCleaner/ProcessDriver.cs Passes per-file byte size into Metrics.FileStarted to seed in-flight weighting.
PlexCleaner/ProcessFile.cs Reports ffprobe packet-scan progress as a fraction of media duration.
PlexCleaner/Process.cs Seeds per-file duration (microseconds) for ffmpeg progress fraction computation.
PlexCleaner/FfMpegBuilder.cs Adds -progress pipe:1 builder helper.
PlexCleaner/FfMpegTool.cs Parses ffmpeg -progress lines and reports per-file progress fraction during encode.
PlexCleaner/HandBrakeBuilder.cs Adds --json global option helper.
PlexCleaner/HandBrakeTool.cs Enables --json, parses HandBrake progress, and reports per-file progress fraction while streaming stdout.
PlexCleanerTests/MetricsTests.cs Updates tests for new FileStarted(size) and adds coverage for partial-credit folding and clamping.
PlexCleanerTests/ToolProgressParsingTests.cs Adds unit tests for ffmpeg and HandBrake progress line parsing.

Comment thread PlexCleaner/ProcessFile.cs
Packets arrive in demux order with interleaved audio/video and
reordered B-frame timestamps, so raw per-packet PtsTime is
non-monotonic. Report the running max pts over duration so the
in-flight scan fraction advances without backward jitter.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings July 19, 2026 03:43

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.

Comment thread PlexCleanerTests/MetricsTests.cs Outdated
Comment thread PlexCleaner/Metrics.cs
Complete one file and start a second that stays in flight, instead
of starting two files on one thread, so the test matches the
ThreadLocal current-file semantics, and clear the in-flight sink at
the end.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings July 19, 2026 03:51

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 1 comment.

Comment thread PlexCleaner/Metrics.cs
DurationUs is written on the worker thread and read per progress
line on the tool pipe thread, so back it with a Volatile.Read
getter and a SetDurationUs writer, matching the permyriad fraction
pattern.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings July 19, 2026 03:58

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated no new comments.

@ptr727
ptr727 merged commit 33c05ca into develop Jul 19, 2026
15 checks passed
@ptr727
ptr727 deleted the feature/metrics-v2-partial-progress branch July 19, 2026 04:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants