feat(metrics): track thread pool backlog

[yordis]

• Operators need visibility into work queued outside the node's named internal queues.
• Thread-pool pressure should appear in the same telemetry surface operators already use for queue health.

[coderabbitai]

Warning

Rate limit exceeded

@yordis has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 4 minutes and 3 seconds before requesting another review.

You’ve run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: e8972272-ec7f-44c4-9c37-e4616bc57486

📥 Commits

Reviewing files that changed from the base of the PR and between 3860089 and d460a51.

📒 Files selected for processing (5)

• src/EventStore.ClusterNode/metricsconfig.json
• src/EventStore.Core.XUnit.Tests/Metrics/QueueStatsCollectorTests.cs
• src/EventStore.Core/Bus/QueueStatsCollector.cs
• src/EventStore.Core/ClusterVNode.cs
• src/EventStore.Core/Metrics/ThreadPoolBacklogMonitor.cs

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch yordis/feat-thread-pool-backlog-monitor

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[cursor]

PR Summary

Medium Risk
Adds a new background sampler that queues work onto the .NET thread pool and reports backlog as queue metrics; incorrect scheduling or disposal could add overhead or leak timers/work items during node lifetime/shutdown.

Overview
Adds a new ThreadPoolBacklogMonitor that periodically samples ThreadPool.PendingWorkItemCount and reports it via QueueStatsCollector/QueueMonitor, so thread-pool pressure shows up alongside existing internal queue health metrics.

Updates ClusterVNode to start and dispose this monitor with node lifecycle, extends QueueStatsCollector with ReportQueueLength (and updates GetStatistics to include current length in peak tracking), adds unit tests for the new peak behavior, and labels the new ThreadPoolBacklog queue in metricsconfig.json.

^{Reviewed by Cursor Bugbot for commit d460a51. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit d460a51. Configure here.}

[cursor]

Redundant ReportQueueLength call after ProcessingStarted

Low Severity

ProcessingStarted<ThreadPoolBacklogSample>(length) already calls ReportQueueLength(length) internally (via the overload in QueueStatsCollector). The explicit _queueStats.ReportQueueLength(length) call immediately after is redundant since it applies Math.Max with the same value, producing no effect. This could mislead future maintainers into thinking the two calls serve distinct purposes.

 

^{Reviewed by Cursor Bugbot for commit d460a51. Configure here.}

[cursor]

Timer disposal race can throw during shutdown

Low Severity

A TOCTOU race exists between the Volatile.Read(ref _stopped) check and the _timer.Change(...) call in Execute(). If Dispose() is called concurrently (during shutdown), Stop() sets _stopped and then _timer.Dispose() runs between the check and the Change call, causing an ObjectDisposedException on the thread pool — which in .NET 6+ crashes the process by default.

Additional Locations (1)

• src/EventStore.Core/Metrics/ThreadPoolBacklogMonitor.cs#L47-L52

 

^{Reviewed by Cursor Bugbot for commit d460a51. Configure here.}