feat(metrics): expose queued work backlog

[yordis]

• Operators need an item-count backlog signal alongside queueing duration so saturation is visible before latency spikes.
• Unmatched queues should stay silent when metrics configuration says they are not collected.

[cursor]

PR Summary

Medium Risk
Touches hot-path queue processing (QueuedHandlerThreadPool and ThreadBasedScheduler) to record queue length, which could affect performance or metric cardinality if misconfigured. Functional risk is moderate since changes are additive and gated by existing queue label matching/config flags.

Overview
Adds a new queue length/backlog metric (eventstore_queue_length_items) to complement queue busy time and queueing-duration metrics.

Implements IQueueLengthTracker/QueueLengthTracker and extends QueueTracker/QueueTrackers so only matched/enabled queues emit length observations, with tests covering matched vs unmatched behavior.

Wires queue-length recording into QueuedHandlerThreadPool (on enqueue/dequeue) and ThreadBasedScheduler (after scheduling/executing) and bootstraps the new eventstore-queue-length observable metric in MetricsBootstrapper.

^{Reviewed by Cursor Bugbot for commit 508d76c. Bugbot is set up for automated code reviews on this repo. Configure here.}

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: b5303e88-0729-412a-bc84-f085383f85ba

📥 Commits

Reviewing files that changed from the base of the PR and between df16068 and 508d76c.

📒 Files selected for processing (8)

• docs/diagnostics/metrics.md
• src/EventStore.Core.XUnit.Tests/Metrics/QueueTrackersTests.cs
• src/EventStore.Core/Bus/QueuedHandlerThreadPool.cs
• src/EventStore.Core/Metrics/QueueLengthTracker.cs
• src/EventStore.Core/Metrics/QueueTracker.cs
• src/EventStore.Core/Metrics/QueueTrackers.cs
• src/EventStore.Core/MetricsBootstrapper.cs
• src/EventStore.Core/Services/TimerService/ThreadBasedScheduler.cs

✅ Files skipped from review due to trivial changes (2)

• docs/diagnostics/metrics.md
• src/EventStore.Core/Services/TimerService/ThreadBasedScheduler.cs

🚧 Files skipped from review as they are similar to previous changes (6)

• src/EventStore.Core/Bus/QueuedHandlerThreadPool.cs
• src/EventStore.Core/Metrics/QueueTracker.cs
• src/EventStore.Core/MetricsBootstrapper.cs
• src/EventStore.Core/Metrics/QueueLengthTracker.cs
• src/EventStore.Core.XUnit.Tests/Metrics/QueueTrackersTests.cs
• src/EventStore.Core/Metrics/QueueTrackers.cs

---

Walkthrough

This PR adds queue-length metric tracking to EventStore by introducing a new IQueueLengthTracker abstraction, integrating it through tracker factories and bootstrap, and recording queue sizes at runtime from queued handler and scheduler code paths.

Changes

Queue Length Metrics Implementation

Layer / File(s)	Summary
Queue length metric abstraction and implementation src/EventStore.Core/Metrics/QueueLengthTracker.cs	IQueueLengthTracker interface with SetQueueLength(int length) method. QueueLengthTracker implementation registers an observable measurement on the provided metric using a queue-name tag, and stores/reports length via Volatile read/write. Nested NoOp implementation for disabled-metric scenarios.
QueueTracker dependency and recording method src/EventStore.Core/Metrics/QueueTracker.cs	QueueTracker constructor accepts IQueueLengthTracker parameter and stores it as a private field. New public RecordQueueLength(int length) method forwards calls to the tracker's SetQueueLength.
QueueTrackers factory and private tracker wiring src/EventStore.Core/Metrics/QueueTrackers.cs	QueueTrackers constructor accepts lengthTrackerFactory function. Internal SharedTrackers record gains an Enabled flag, and PrivateTrackers record gains an IQueueLengthTracker field. GetTrackerForQueue resolves private trackers based on shared metric enablement and passes both private busy and private length trackers into each QueueTracker.
MetricsBootstrapper metric creation and factory initialization src/EventStore.Core/MetricsBootstrapper.cs	Creates eventstore-queue-length ObservableUpDownMetric<int>. Initializes separate factories for queue length (using QueueLengthTracker) and queueing-duration max (using DurationMaxTracker). Updates QueueTrackers constructor invocation to pass both new and existing tracker factories.
Runtime queue length recording in schedulers src/EventStore.Core/Bus/QueuedHandlerThreadPool.cs, src/EventStore.Core/Services/TimerService/ThreadBasedScheduler.cs	QueuedHandlerThreadPool records queue length in dequeue-processing loop and after message enqueue. ThreadBasedScheduler records combined pending and scheduled task count after scheduler phases.
Test coverage and documentation src/EventStore.Core.XUnit.Tests/Metrics/QueueTrackersTests.cs, docs/diagnostics/metrics.md	New test methods validate queue-length recording for matched/unmapped queues. FakeTracker implements IQueueLengthTracker with nullable Length property. Metrics documentation references the new eventstore_queue_length_items{queue=<QUEUE_GROUP>} gauge.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Poem

A rabbit hops through queues so neat,
Counting items with nimble feet,
From thread pools to the scheduler's song,
Each queue's length now hums along,
Metrics dance — the numbers meet! 🐰📊

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The PR title 'feat(metrics): expose queued work backlog' accurately and concisely describes the main change—adding queue length metrics to surface work backlog.
Description check	✅ Passed	The PR description is directly related to the changeset, explaining the motivation (need for backlog signal alongside duration) and behavior requirement (unmatched queues stay silent).
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch yordis/feat-queue-length-metrics

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}