Skip to content

chore: standardize count metric descriptions on "Number of"#3314

Open
chet wants to merge 1 commit into
NVIDIA:mainfrom
chet:gh-issue-3307
Open

chore: standardize count metric descriptions on "Number of"#3314
chet wants to merge 1 commit into
NVIDIA:mainfrom
chet:gh-issue-3307

Conversation

@chet

@chet chet commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Every count-style metric description now opens with Number of -- the preferred prefix @polarweasel called out in the #3266 review -- instead of the previous mix of "The amount of", "The number of", "The total number of", "Count of", etc.

Primary callouts are:

  • The vault client counters in forge_vault.rs (carbide-api.vault.requests_*) claimed to count "tls connections" -- a copy-paste from the bmc-proxy metrics. They describe Vault requests now.
  • The state-controller templates lose their mangled grammar: "The amount of types that object handling tasks that have been ..." becomes "Number of object handling tasks that have been ...", and the state entered/exited counters now read "Number of times objects of type X have entered a certain state".
  • Small grammar fixes ride along inside touched strings only -- "machine validation runs", "pairs that have been identified", "machines that report software components", and lowercase "ips"/"tls"/"tcp" become "IPs"/"TLS"/"TCP".
  • The golden /metrics fixtures assert exposition output verbatim, so their # HELP lines get the same rewrite -- 67 lines across 11 fixture files.
  • Time and duration descriptions ("The amount of time ...", "Total time ...") are not counts and stay untouched, as does the multi-kind carbide_site_explorer_update_explored_endpoints_count row.

Tests updated!

Verified by running test_integration locally -- the regenerated core_metrics.md comes out identical to what's committed here.

This supports #3307

@chet chet requested review from a team and polarweasel as code owners July 9, 2026 16:41
@chet

chet commented Jul 9, 2026

Copy link
Copy Markdown
Contributor Author

@coderabbitai full_review, thanks!

@coderabbitai

coderabbitai Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

@chet, will do — kicking off a full review of all changes in this PR now.

(=^-ω-^=) 🥕

✅ Action performed

Full review finished.

@coderabbitai

coderabbitai Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

Walkthrough

This PR standardizes human-readable metric and event descriptions across observability code, Prometheus fixtures, and documentation. Metric names, types, labels, callbacks, observed values, and recording behavior remain unchanged, except for one measured-boot fixture cleanup.

Changes

Metric description standardization

Layer / File(s) Summary
API-core metrics and fixtures
crates/agent/..., crates/api-core/...
Updates API-core, validation, measured-boot, firmware, secrets, and related fixture descriptions.
Transport and network metrics
crates/bmc-proxy/..., crates/dns/..., crates/network-segment-controller/..., crates/preingestion-manager/..., crates/secrets/..., crates/mqtt-common/...
Standardizes transport, DNS, network, preingestion, MQTT, and Vault descriptions.
Service and controller metrics
crates/dpa-manager/..., crates/dsx-exchange-consumer/..., crates/fmds/..., crates/sqlx-query-tracing/..., crates/ssh-console/..., crates/state-controller/...
Rewords service and controller metric metadata without changing instrumentation behavior.
Health and infrastructure metrics
crates/health-metrics/..., crates/health/..., crates/ib-fabric/..., crates/machine-controller/..., crates/nvlink-manager/..., crates/site-explorer/...
Revises health, infrastructure, and machine metric wording; one tenant-label construction is only reformatted.
Observability documentation
docs/observability/core_metrics.md, crates/dsx-exchange-consumer/README.md
Aligns documented metric descriptions and names with the updated instrumentation metadata.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 42.86% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly summarizes the main change: standardizing count metric descriptions to start with 'Number of'.
Description check ✅ Passed The description accurately matches the changeset, including metric wording updates, fixture rewrites, and the Vault/state-controller specifics.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 5

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
crates/api-core/src/listener.rs (1)

299-310: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Widen the failure wording.

Line 309 is too narrow: this counter is incremented for both TCP accept errors and TLS handshake failures, so “TCP connections” misstates what operators will see in Prometheus. Consider wording it as “Number of connection attempts that failed” or “Number of TCP/TLS connections that failed.”

Suggested wording
-        .with_description("Number of TCP connections that failed")
+        .with_description("Number of connection attempts that failed")
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/api-core/src/listener.rs` around lines 299 - 310, The description for
the failure counter in listener.rs is too narrow for what
connection_failed_counter actually measures. Update the with_description text in
the metric setup near
connection_total_counter/connection_succeeded_counter/connection_failed_counter
so it reflects both TCP accept errors and TLS handshake failures, using wording
like “Number of connection attempts that failed” or “Number of TCP/TLS
connections that failed.”
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/api-core/src/machine_update_manager/dpu_nic_firmware_metrics.rs`:
- Line 50: Update the DPU firmware metrics fixture so its HELP text matches the
current metric descriptions in dpu_nic_firmware_metrics.rs. The fixture
test_dpu_nic_firmware_metrics.txt still contains the old wording for all three
metrics, so sync each HELP line with the corresponding definitions in the
DpuNicFirmwareMetrics setup, especially the metric descriptions built with
with_description().

In `@crates/api-core/src/machine_update_manager/host_firmware.rs`:
- Line 261: The host firmware metrics fixture is out of sync with the current
metric descriptions for carbide_pending_host_firmware_update_count and
carbide_active_host_firmware_update_count. Update the expected # HELP text in
the host firmware metrics test fixture so it matches the descriptions emitted by
the related host firmware metrics definitions in
machine_update_manager::host_firmware, keeping both metric names aligned with
the current wording.

In `@crates/api-core/src/machine_validation/metrics.rs`:
- Line 50: The machine-validation metrics HELP text in the metrics definition no
longer matches the fixture used by the direct parsed-metrics comparison test.
Update the description in the machine-validation metrics setup so it matches the
expected text in the test metrics fixture, and verify the referenced metric in
the metrics collector test still uses the same HELP string as the definition.

In `@crates/api-core/src/measured_boot/metrics_collector/metrics.rs`:
- Line 85: The measured-boot metrics HELP text has changed in
MetricsCollector::with_description, but the golden fixture still expects the old
wording. Update the HELP lines in the measured-boot test data so they match the
current description strings from metrics.rs, keeping the fixture aligned with
the metrics emitted by the collector.

In `@crates/bmc-proxy/src/bmc_proxy.rs`:
- Around line 249-253: The failure counter description in bmc_proxy::BmcProxy is
misleading because it is used for both TCP accept errors and TLS handshake
failures. Update the metric help text in the connection_failed_counter setup to
reflect both paths, or split the counting in the relevant accept/handshake logic
into separate counters if you want distinct TCP and TLS failure metrics.

---

Outside diff comments:
In `@crates/api-core/src/listener.rs`:
- Around line 299-310: The description for the failure counter in listener.rs is
too narrow for what connection_failed_counter actually measures. Update the
with_description text in the metric setup near
connection_total_counter/connection_succeeded_counter/connection_failed_counter
so it reflects both TCP accept errors and TLS handshake failures, using wording
like “Number of connection attempts that failed” or “Number of TCP/TLS
connections that failed.”
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 54696b23-4273-4797-aed7-a4c52803d906

📥 Commits

Reviewing files that changed from the base of the PR and between b907061 and 9e5c97e.

📒 Files selected for processing (34)
  • crates/agent/src/instrumentation.rs
  • crates/api-core/src/handlers/machine_scout.rs
  • crates/api-core/src/listener.rs
  • crates/api-core/src/logging/service_health_metrics.rs
  • crates/api-core/src/machine_update_manager/dpu_nic_firmware_metrics.rs
  • crates/api-core/src/machine_update_manager/host_firmware.rs
  • crates/api-core/src/machine_update_manager/metrics.rs
  • crates/api-core/src/machine_validation/metrics.rs
  • crates/api-core/src/measured_boot/metrics_collector/metrics.rs
  • crates/api-core/src/secrets/metrics.rs
  • crates/bmc-proxy/src/bmc_proxy.rs
  • crates/dhcp/src/metrics.rs
  • crates/dns/src/lib.rs
  • crates/dpa-manager/src/metrics.rs
  • crates/dsx-exchange-consumer/src/metrics.rs
  • crates/fmds/src/http_request_metrics.rs
  • crates/health-metrics/src/lib.rs
  • crates/health/src/collectors/runtime.rs
  • crates/health/src/lib.rs
  • crates/health/src/metrics.rs
  • crates/ib-fabric/src/metrics.rs
  • crates/machine-controller/src/metrics.rs
  • crates/mqtt-common/src/metrics.rs
  • crates/network-segment-controller/src/metrics.rs
  • crates/preingestion-manager/src/metrics.rs
  • crates/secrets/src/forge_vault.rs
  • crates/site-explorer/src/metrics.rs
  • crates/sqlx-query-tracing/src/lib.rs
  • crates/ssh-console/src/bmc/client_pool.rs
  • crates/ssh-console/src/ssh_server.rs
  • crates/state-controller/src/controller/periodic_enqueuer.rs
  • crates/state-controller/src/controller/processor.rs
  • crates/state-controller/src/metrics.rs
  • docs/observability/core_metrics.md

Comment thread crates/api-core/src/machine_update_manager/host_firmware.rs
Comment thread crates/api-core/src/machine_validation/metrics.rs
Comment thread crates/api-core/src/measured_boot/metrics_collector/metrics.rs
Comment thread crates/bmc-proxy/src/bmc_proxy.rs Outdated

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

♻️ Duplicate comments (1)
crates/bmc-proxy/src/bmc_proxy.rs (1)

241-253: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Broaden the failure metric description.

This is the same mismatch previously called out: connection_failed_counter still records TCP accept errors, TLS certificate reload failures, and TLS handshake failures, so “TCP connections that failed” under-describes the metric.

♻️ Proposed fix
-            .with_description("Number of TCP connections that failed")
+            .with_description("Number of connections that failed")
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/bmc-proxy/src/bmc_proxy.rs` around lines 241 - 253, The
`connection_failed_counter` metric description is too narrow for what it counts.
Update the description in `BmcProxy::...` where `connection_failed_counter` is
built so it covers all failure cases the counter records, including TCP accept
errors, TLS certificate reload failures, and TLS handshake failures. Keep the
metric name the same and just broaden the wording in the `with_description` call
to match its actual usage.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/api-core/src/listener.rs`:
- Line 309: The metric description in the listener counter is too narrow because
it covers both TCP accept failures and TLS handshake failures. Update the
description used in the listener setup around with_description so it reflects
all connection failures, not just TCP, and keep it aligned with the counter’s
actual scope in the listener logic.

In `@crates/api-core/src/logging/service_health_metrics.rs`:
- Line 74: The HELP description in service_health_metrics was changed, so update
the matching expected strings in the service-health metrics fixture as well.
Keep the descriptions in ServiceHealthMetrics aligned with the exact text
asserted by test_service_health_metrics.txt, including the other affected metric
descriptions at the referenced spots, so the exact metrics comparison continues
to pass.

In `@crates/ib-fabric/src/metrics.rs`:
- Line 351: The description for
carbide_ib_monitor_allow_insecure_fabric_configuration_count duplicates the
insecure_fabric_configuration wording, so update the metric metadata in
metrics.rs to clearly describe the allow_insecure_fabric_configuration boolean
tracked by the builder around the gauge definition. Use the existing metric
registration code for the two gauges to give each a distinct, accurate
description that reflects its own configuration state and avoids contract
ambiguity.

In `@docs/observability/core_metrics.md`:
- Line 55: The metric descriptions generated from
`crates/ib-fabric/src/metrics.rs` and `crates/nvlink-manager/src/metrics.rs`
still use internal field-style wording. Update the description strings in the
relevant metric definitions so the docs read naturally, following the
`*_status_updates_count` entries and phrasing them like “Number of machines
whose InfiniBand status observation was updated” and “Number of machines whose
NVLink status observation was updated.”

---

Duplicate comments:
In `@crates/bmc-proxy/src/bmc_proxy.rs`:
- Around line 241-253: The `connection_failed_counter` metric description is too
narrow for what it counts. Update the description in `BmcProxy::...` where
`connection_failed_counter` is built so it covers all failure cases the counter
records, including TCP accept errors, TLS certificate reload failures, and TLS
handshake failures. Keep the metric name the same and just broaden the wording
in the `with_description` call to match its actual usage.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: e7305efe-607f-46e6-8ab2-edc05d5a2033

📥 Commits

Reviewing files that changed from the base of the PR and between b907061 and 9e5c97e.

📒 Files selected for processing (34)
  • crates/agent/src/instrumentation.rs
  • crates/api-core/src/handlers/machine_scout.rs
  • crates/api-core/src/listener.rs
  • crates/api-core/src/logging/service_health_metrics.rs
  • crates/api-core/src/machine_update_manager/dpu_nic_firmware_metrics.rs
  • crates/api-core/src/machine_update_manager/host_firmware.rs
  • crates/api-core/src/machine_update_manager/metrics.rs
  • crates/api-core/src/machine_validation/metrics.rs
  • crates/api-core/src/measured_boot/metrics_collector/metrics.rs
  • crates/api-core/src/secrets/metrics.rs
  • crates/bmc-proxy/src/bmc_proxy.rs
  • crates/dhcp/src/metrics.rs
  • crates/dns/src/lib.rs
  • crates/dpa-manager/src/metrics.rs
  • crates/dsx-exchange-consumer/src/metrics.rs
  • crates/fmds/src/http_request_metrics.rs
  • crates/health-metrics/src/lib.rs
  • crates/health/src/collectors/runtime.rs
  • crates/health/src/lib.rs
  • crates/health/src/metrics.rs
  • crates/ib-fabric/src/metrics.rs
  • crates/machine-controller/src/metrics.rs
  • crates/mqtt-common/src/metrics.rs
  • crates/network-segment-controller/src/metrics.rs
  • crates/preingestion-manager/src/metrics.rs
  • crates/secrets/src/forge_vault.rs
  • crates/site-explorer/src/metrics.rs
  • crates/sqlx-query-tracing/src/lib.rs
  • crates/ssh-console/src/bmc/client_pool.rs
  • crates/ssh-console/src/ssh_server.rs
  • crates/state-controller/src/controller/periodic_enqueuer.rs
  • crates/state-controller/src/controller/processor.rs
  • crates/state-controller/src/metrics.rs
  • docs/observability/core_metrics.md

Comment thread crates/api-core/src/listener.rs Outdated
Comment thread crates/api-core/src/logging/service_health_metrics.rs
Comment thread crates/ib-fabric/src/metrics.rs Outdated
Comment thread docs/observability/core_metrics.md Outdated
@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown

🔍 Container Scan Summary

Service Total Critical High Medium Low Other
boot-artifacts-aarch64 3 0 0 3 0 0
boot-artifacts-x86_64 3 0 0 3 0 0
forge-admin-cli-x86_64 271 13 34 91 7 126
machine-validation-runner 804 40 237 296 36 195
machine_validation 804 40 237 296 36 195
machine_validation-aarch64 804 40 237 296 36 195
nvmetal-carbide 804 40 237 296 36 195
TOTAL 3493 173 982 1281 151 906

Per-CVE detail lives in the per-service grype-* artifacts (JSON + SARIF). Severity counts only — no CVE IDs published here.

@chet chet force-pushed the gh-issue-3307 branch from 9e5c97e to 1280c1a Compare July 9, 2026 18:38
@chet

chet commented Jul 9, 2026

Copy link
Copy Markdown
Contributor Author

@coderabbitai full_review, thanks!

@coderabbitai

coderabbitai Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

@chet, you're welcome — kicking off a full review of all changes in this PR now.

(=^・ω・^=)

✅ Action performed

Full review finished.

@chet chet force-pushed the gh-issue-3307 branch from 1280c1a to 0e3f721 Compare July 9, 2026 23:32
@chet

chet commented Jul 9, 2026

Copy link
Copy Markdown
Contributor Author

@coderabbitai full_review, thanks! (Rebased over the merged instrumentation sweeps -- the only conflicts were with the Kea-hook PR, which had already deleted the counter constructions and fixture lines this branch re-worded; both sides wanted "Number of", so the resolution was taking main's framework events, which already comply.)

@coderabbitai

coderabbitai Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

@chet, understood — thanks for the context on the rebase resolution. Taking main's framework events (already using "Number of" phrasing) as the resolution for the Kea-hook conflict sounds correct, since both branches converged on the same wording. Kicking off a full review of all changes now.

(=^-ω-^=)

✅ Action performed

Full review finished.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/dsx-exchange-consumer/src/metrics.rs`:
- Line 39: Update the DSX metrics documentation table in README.md to match the
standardized descriptions defined by the metric declarations in metrics.rs,
including the metadata cache metric and the other affected entries. Ensure every
listed description matches the `/metrics` HELP output exactly.

In `@crates/ib-fabric/src/metrics.rs`:
- Line 376: Update the metric descriptions near the affected entries to use
complete wording: replace “incl non Forge” with “including non-Forge” and “incl
non Forge managed” with “including non-Forge-managed”.

In `@crates/machine-controller/src/metrics.rs`:
- Around line 437-443: Update the descriptions for the u64_histogram instruments
machine_reboot_attempts_in_booting_with_discovery_image and
machine_reboot_attempts_in_failed_during_discovery to describe recorded
reboot-attempt values/samples rather than counting machines; preserve the
existing state and failure-context details.

In `@crates/site-explorer/src/metrics.rs`:
- Line 526: Update the description for the
carbide_endpoint_exploration_machines_explored_overall_count metric to mention
both emitted label dimensions, expectation and machine_type, using wording such
as “Number of machines explored by expectation and machine type.”
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 6250076a-4897-41f2-a790-04495375f278

📥 Commits

Reviewing files that changed from the base of the PR and between 8651f88 and 0e3f721.

📒 Files selected for processing (44)
  • crates/agent/src/instrumentation.rs
  • crates/api-core/src/handlers/machine_scout.rs
  • crates/api-core/src/listener.rs
  • crates/api-core/src/logging/service_health_metrics.rs
  • crates/api-core/src/machine_update_manager/dpu_nic_firmware_metrics.rs
  • crates/api-core/src/machine_update_manager/host_firmware.rs
  • crates/api-core/src/machine_update_manager/metrics.rs
  • crates/api-core/src/machine_validation/fixtures/test_metrics_collector.txt
  • crates/api-core/src/machine_validation/metrics.rs
  • crates/api-core/src/measured_boot/metrics_collector/metrics.rs
  • crates/api-core/src/measured_boot/tests/test_data/test_metrics_collector.txt
  • crates/api-core/src/secrets/metrics.rs
  • crates/api-core/src/tests/metrics_fixtures/test_dpu_nic_firmware_metrics.txt
  • crates/api-core/src/tests/metrics_fixtures/test_host_firmware_metrics.txt
  • crates/api-core/src/tests/metrics_fixtures/test_machine_update_manager_metrics.txt
  • crates/api-core/src/tests/metrics_fixtures/test_network_segment_metrics_admin.txt
  • crates/api-core/src/tests/metrics_fixtures/test_network_segment_metrics_tenant.txt
  • crates/api-core/src/tests/metrics_fixtures/test_network_segment_metrics_tor.txt
  • crates/api-core/src/tests/metrics_fixtures/test_service_health_metrics.txt
  • crates/bmc-proxy/src/bmc_proxy.rs
  • crates/dns/src/lib.rs
  • crates/dpa-manager/src/metrics.rs
  • crates/dsx-exchange-consumer/src/metrics.rs
  • crates/fmds/src/http_request_metrics.rs
  • crates/health-metrics/src/lib.rs
  • crates/health/src/collectors/runtime.rs
  • crates/health/src/lib.rs
  • crates/health/src/metrics.rs
  • crates/ib-fabric/src/metrics.rs
  • crates/machine-controller/src/metrics.rs
  • crates/mqtt-common/src/metrics.rs
  • crates/network-segment-controller/src/metrics.rs
  • crates/nvlink-manager/src/metrics.rs
  • crates/preingestion-manager/src/fixtures/test_metrics_collector.txt
  • crates/preingestion-manager/src/metrics.rs
  • crates/secrets/src/forge_vault.rs
  • crates/site-explorer/src/metrics.rs
  • crates/sqlx-query-tracing/src/lib.rs
  • crates/ssh-console/src/bmc/client_pool.rs
  • crates/ssh-console/src/ssh_server.rs
  • crates/state-controller/src/controller/periodic_enqueuer.rs
  • crates/state-controller/src/controller/processor.rs
  • crates/state-controller/src/metrics.rs
  • docs/observability/core_metrics.md

Comment thread crates/dsx-exchange-consumer/src/metrics.rs
Comment thread crates/ib-fabric/src/metrics.rs Outdated
Comment thread crates/machine-controller/src/metrics.rs Outdated
Comment thread crates/site-explorer/src/metrics.rs Outdated
@chet chet force-pushed the gh-issue-3307 branch 2 times, most recently from 5c79d2b to afc6fb4 Compare July 10, 2026 01:20
@chet

chet commented Jul 10, 2026

Copy link
Copy Markdown
Contributor Author

@coderabbitai full_review, thanks!

@coderabbitai

coderabbitai Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

@chet, you're welcome — kicking off another full review of all changes in this PR now.

(=^・ω・^=)

✅ Action performed

Full review finished.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 6

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
crates/api-core/src/measured_boot/metrics_collector/metrics.rs (1)

163-186: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Correct the “per a given” wording in metric descriptions.

Use “for a given measured boot bundle state” and “for a given measured boot machine state”; the current phrasing is grammatically incorrect in exported HELP text.

As per path instructions, Markdown and user-facing text should be reviewed for grammar and clarity.

Proposed wording
- .with_description("Number of machines per a given measured boot bundle state.")
+ .with_description("Number of machines for a given measured boot bundle state.")

- .with_description("Number of machines per a given measured boot machine state.")
+ .with_description("Number of machines for a given measured boot machine state.")
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/api-core/src/measured_boot/metrics_collector/metrics.rs` around lines
163 - 186, Update the descriptions in the observable gauges for the bundle-state
and machine-state metrics to replace “per a given” with “for a given,” using
“Number of machines for a given measured boot bundle state.” and “Number of
machines for a given measured boot machine state.”

Source: Path instructions

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/bmc-proxy/src/bmc_proxy.rs`:
- Around line 241-242: The description for connection_total_counter inaccurately
says “TLS connections” although the counter increments before listener.accept()
and includes TCP accept failures. Update its with_description text to “Number of
inbound connection attempts,” or move the increment to the successful accept
path if the metric should count only accepted connections.

In `@crates/ib-fabric/src/metrics.rs`:
- Around line 167-169: Update the descriptions for the gauge near
carbide_ib_monitor_machines_by_port_state_count and the adjacent partition
metric to use consistent grammar: write “Number of machines whose total and
active port counts match the attribute values” and lowercase “machines” in the
partition description.

In `@crates/machine-controller/src/metrics.rs`:
- Around line 445-450: Update the description in the
carbide_hosts_with_bios_password_set observable gauge registration to use
sentence case, changing “Number of Hosts...” to “Number of hosts...”, and
synchronize the corresponding generated documentation row.

In `@crates/site-explorer/src/metrics.rs`:
- Around line 738-740: Update the metric HELP text associated with the blocker
description to capitalize the acronym consistently, replacing “host+dpu” with
“host+DPU” and “dpu(s)” with “DPU(s)”.

In `@crates/state-controller/src/metrics.rs`:
- Around line 344-347: Update the description in the object-state observable
gauge within the metrics registration code to use present-tense wording,
replacing “had been longer in a state than allowed per SLA” with language
indicating objects currently exceeding their allowed state SLA.
- Around line 374-376: Update the metric description in the builder for the
callback observing handling_errors_per_type and its sum to describe the count of
state-handling errors, not the number of objects; retain the object_type context
while aligning the wording with the observed value.

---

Outside diff comments:
In `@crates/api-core/src/measured_boot/metrics_collector/metrics.rs`:
- Around line 163-186: Update the descriptions in the observable gauges for the
bundle-state and machine-state metrics to replace “per a given” with “for a
given,” using “Number of machines for a given measured boot bundle state.” and
“Number of machines for a given measured boot machine state.”
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 73ac0ab2-122d-4fa5-af43-07e6133659ad

📥 Commits

Reviewing files that changed from the base of the PR and between 699951d and afc6fb4.

📒 Files selected for processing (45)
  • crates/agent/src/instrumentation.rs
  • crates/api-core/src/handlers/machine_scout.rs
  • crates/api-core/src/listener.rs
  • crates/api-core/src/logging/service_health_metrics.rs
  • crates/api-core/src/machine_update_manager/dpu_nic_firmware_metrics.rs
  • crates/api-core/src/machine_update_manager/host_firmware.rs
  • crates/api-core/src/machine_update_manager/metrics.rs
  • crates/api-core/src/machine_validation/fixtures/test_metrics_collector.txt
  • crates/api-core/src/machine_validation/metrics.rs
  • crates/api-core/src/measured_boot/metrics_collector/metrics.rs
  • crates/api-core/src/measured_boot/tests/test_data/test_metrics_collector.txt
  • crates/api-core/src/secrets/metrics.rs
  • crates/api-core/src/tests/metrics_fixtures/test_dpu_nic_firmware_metrics.txt
  • crates/api-core/src/tests/metrics_fixtures/test_host_firmware_metrics.txt
  • crates/api-core/src/tests/metrics_fixtures/test_machine_update_manager_metrics.txt
  • crates/api-core/src/tests/metrics_fixtures/test_network_segment_metrics_admin.txt
  • crates/api-core/src/tests/metrics_fixtures/test_network_segment_metrics_tenant.txt
  • crates/api-core/src/tests/metrics_fixtures/test_network_segment_metrics_tor.txt
  • crates/api-core/src/tests/metrics_fixtures/test_service_health_metrics.txt
  • crates/bmc-proxy/src/bmc_proxy.rs
  • crates/dns/src/lib.rs
  • crates/dpa-manager/src/metrics.rs
  • crates/dsx-exchange-consumer/README.md
  • crates/dsx-exchange-consumer/src/metrics.rs
  • crates/fmds/src/http_request_metrics.rs
  • crates/health-metrics/src/lib.rs
  • crates/health/src/collectors/runtime.rs
  • crates/health/src/lib.rs
  • crates/health/src/metrics.rs
  • crates/ib-fabric/src/metrics.rs
  • crates/machine-controller/src/metrics.rs
  • crates/mqtt-common/src/metrics.rs
  • crates/network-segment-controller/src/metrics.rs
  • crates/nvlink-manager/src/metrics.rs
  • crates/preingestion-manager/src/fixtures/test_metrics_collector.txt
  • crates/preingestion-manager/src/metrics.rs
  • crates/secrets/src/forge_vault.rs
  • crates/site-explorer/src/metrics.rs
  • crates/sqlx-query-tracing/src/lib.rs
  • crates/ssh-console/src/bmc/client_pool.rs
  • crates/ssh-console/src/ssh_server.rs
  • crates/state-controller/src/controller/periodic_enqueuer.rs
  • crates/state-controller/src/controller/processor.rs
  • crates/state-controller/src/metrics.rs
  • docs/observability/core_metrics.md

Comment thread crates/bmc-proxy/src/bmc_proxy.rs Outdated
Comment thread crates/ib-fabric/src/metrics.rs Outdated
Comment thread crates/machine-controller/src/metrics.rs Outdated
Comment thread crates/site-explorer/src/metrics.rs Outdated
Comment thread crates/state-controller/src/metrics.rs
Comment thread crates/state-controller/src/metrics.rs
@chet

chet commented Jul 10, 2026

Copy link
Copy Markdown
Contributor Author

@coderabbitai full_review, thanks!

@coderabbitai

coderabbitai Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

@chet, you're welcome — kicking off another full review of all changes in this PR now.

(=^・ω・^=)

✅ Action performed

Full review finished.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/site-explorer/src/metrics.rs`:
- Line 661: The metric descriptions for the machine-pair and power-shelf gauges
use awkward past-tense wording and unnecessary capitalization. Update the
strings passed to the relevant with_description calls near the machine-pair
metric and the corresponding power-shelf metric to concise present-tense wording
that directly describes the observed count, using lowercase common nouns.
- Around line 759-761: Update the metric description associated with the
migration signal callback to describe the dimension as “by signal type” rather
than “by kind,” while preserving the existing signal categories and `signal`
label.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: d0e9abf8-242a-478d-bd68-9b5f86bb81a5

📥 Commits

Reviewing files that changed from the base of the PR and between 2fa5b3b and 914022f.

📒 Files selected for processing (45)
  • crates/agent/src/instrumentation.rs
  • crates/api-core/src/handlers/machine_scout.rs
  • crates/api-core/src/listener.rs
  • crates/api-core/src/logging/service_health_metrics.rs
  • crates/api-core/src/machine_update_manager/dpu_nic_firmware_metrics.rs
  • crates/api-core/src/machine_update_manager/host_firmware.rs
  • crates/api-core/src/machine_update_manager/metrics.rs
  • crates/api-core/src/machine_validation/fixtures/test_metrics_collector.txt
  • crates/api-core/src/machine_validation/metrics.rs
  • crates/api-core/src/measured_boot/metrics_collector/metrics.rs
  • crates/api-core/src/measured_boot/tests/test_data/test_metrics_collector.txt
  • crates/api-core/src/secrets/metrics.rs
  • crates/api-core/src/tests/metrics_fixtures/test_dpu_nic_firmware_metrics.txt
  • crates/api-core/src/tests/metrics_fixtures/test_host_firmware_metrics.txt
  • crates/api-core/src/tests/metrics_fixtures/test_machine_update_manager_metrics.txt
  • crates/api-core/src/tests/metrics_fixtures/test_network_segment_metrics_admin.txt
  • crates/api-core/src/tests/metrics_fixtures/test_network_segment_metrics_tenant.txt
  • crates/api-core/src/tests/metrics_fixtures/test_network_segment_metrics_tor.txt
  • crates/api-core/src/tests/metrics_fixtures/test_service_health_metrics.txt
  • crates/bmc-proxy/src/bmc_proxy.rs
  • crates/dns/src/lib.rs
  • crates/dpa-manager/src/metrics.rs
  • crates/dsx-exchange-consumer/README.md
  • crates/dsx-exchange-consumer/src/metrics.rs
  • crates/fmds/src/http_request_metrics.rs
  • crates/health-metrics/src/lib.rs
  • crates/health/src/collectors/runtime.rs
  • crates/health/src/lib.rs
  • crates/health/src/metrics.rs
  • crates/ib-fabric/src/metrics.rs
  • crates/machine-controller/src/metrics.rs
  • crates/mqtt-common/src/metrics.rs
  • crates/network-segment-controller/src/metrics.rs
  • crates/nvlink-manager/src/metrics.rs
  • crates/preingestion-manager/src/fixtures/test_metrics_collector.txt
  • crates/preingestion-manager/src/metrics.rs
  • crates/secrets/src/forge_vault.rs
  • crates/site-explorer/src/metrics.rs
  • crates/sqlx-query-tracing/src/lib.rs
  • crates/ssh-console/src/bmc/client_pool.rs
  • crates/ssh-console/src/ssh_server.rs
  • crates/state-controller/src/controller/periodic_enqueuer.rs
  • crates/state-controller/src/controller/processor.rs
  • crates/state-controller/src/metrics.rs
  • docs/observability/core_metrics.md

Comment thread crates/site-explorer/src/metrics.rs Outdated
Comment thread crates/site-explorer/src/metrics.rs Outdated
@chet

chet commented Jul 10, 2026

Copy link
Copy Markdown
Contributor Author

@coderabbitai full_review, thanks!

@coderabbitai

coderabbitai Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

@chet, you're welcome — kicking off another full review of all changes in this PR now.

✅ Action performed

Full review finished.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 5

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/api-core/src/measured_boot/metrics_collector/metrics.rs`:
- Line 164: The metric HELP descriptions in the measured boot metrics
definitions are grammatically awkward; update the bundle-state description to
“Number of machines in each measured boot bundle state” and the machine-state
description to “Number of machines in each measured boot machine state,” then
update the corresponding golden fixture lines.

In `@crates/ib-fabric/src/metrics.rs`:
- Around line 397-400: Update the description passed to the
carbide_ib_monitor_ufm_ports_by_state_count gauge registration to explicitly
state that the metric reports port counts grouped by port_state, while retaining
the note about UFM and non-Forge-managed ports.

In `@crates/machine-controller/src/metrics.rs`:
- Around line 126-127: Update the description passed to the
carbide_gpus_total_count gauge in the metrics registration code to describe the
total GPUs accumulated from object_metrics.num_gpus, not GPU availability;
reserve availability/allocatable wording for the separate gpus_usable metric.

In `@crates/network-segment-controller/src/metrics.rs`:
- Line 64: Update the descriptions of all three metrics in the metrics
definitions, including the entries near the currently referenced line and the
additional occurrences, to describe per-network-segment values rather than
site-wide totals. Use consistent wording such as “Number of available IPs per
network segment” for each metric’s with_description call.

In `@crates/ssh-console/src/bmc/client_pool.rs`:
- Around line 219-221: Update the description of the _failed_machines observable
gauge in the client pool metric initialization to accurately state that it
counts host BMCs in the ConnectionError state, using wording such as “Number of
host BMCs with connection errors” rather than implying multiple errors.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 21d20f37-07a9-4f4e-aec9-4066b3d0590f

📥 Commits

Reviewing files that changed from the base of the PR and between 2fa5b3b and c60c6e3.

📒 Files selected for processing (45)
  • crates/agent/src/instrumentation.rs
  • crates/api-core/src/handlers/machine_scout.rs
  • crates/api-core/src/listener.rs
  • crates/api-core/src/logging/service_health_metrics.rs
  • crates/api-core/src/machine_update_manager/dpu_nic_firmware_metrics.rs
  • crates/api-core/src/machine_update_manager/host_firmware.rs
  • crates/api-core/src/machine_update_manager/metrics.rs
  • crates/api-core/src/machine_validation/fixtures/test_metrics_collector.txt
  • crates/api-core/src/machine_validation/metrics.rs
  • crates/api-core/src/measured_boot/metrics_collector/metrics.rs
  • crates/api-core/src/measured_boot/tests/test_data/test_metrics_collector.txt
  • crates/api-core/src/secrets/metrics.rs
  • crates/api-core/src/tests/metrics_fixtures/test_dpu_nic_firmware_metrics.txt
  • crates/api-core/src/tests/metrics_fixtures/test_host_firmware_metrics.txt
  • crates/api-core/src/tests/metrics_fixtures/test_machine_update_manager_metrics.txt
  • crates/api-core/src/tests/metrics_fixtures/test_network_segment_metrics_admin.txt
  • crates/api-core/src/tests/metrics_fixtures/test_network_segment_metrics_tenant.txt
  • crates/api-core/src/tests/metrics_fixtures/test_network_segment_metrics_tor.txt
  • crates/api-core/src/tests/metrics_fixtures/test_service_health_metrics.txt
  • crates/bmc-proxy/src/bmc_proxy.rs
  • crates/dns/src/lib.rs
  • crates/dpa-manager/src/metrics.rs
  • crates/dsx-exchange-consumer/README.md
  • crates/dsx-exchange-consumer/src/metrics.rs
  • crates/fmds/src/http_request_metrics.rs
  • crates/health-metrics/src/lib.rs
  • crates/health/src/collectors/runtime.rs
  • crates/health/src/lib.rs
  • crates/health/src/metrics.rs
  • crates/ib-fabric/src/metrics.rs
  • crates/machine-controller/src/metrics.rs
  • crates/mqtt-common/src/metrics.rs
  • crates/network-segment-controller/src/metrics.rs
  • crates/nvlink-manager/src/metrics.rs
  • crates/preingestion-manager/src/fixtures/test_metrics_collector.txt
  • crates/preingestion-manager/src/metrics.rs
  • crates/secrets/src/forge_vault.rs
  • crates/site-explorer/src/metrics.rs
  • crates/sqlx-query-tracing/src/lib.rs
  • crates/ssh-console/src/bmc/client_pool.rs
  • crates/ssh-console/src/ssh_server.rs
  • crates/state-controller/src/controller/periodic_enqueuer.rs
  • crates/state-controller/src/controller/processor.rs
  • crates/state-controller/src/metrics.rs
  • docs/observability/core_metrics.md

Comment thread crates/api-core/src/measured_boot/metrics_collector/metrics.rs Outdated
Comment thread crates/ib-fabric/src/metrics.rs
Comment thread crates/machine-controller/src/metrics.rs Outdated
Comment thread crates/network-segment-controller/src/metrics.rs Outdated
Comment thread crates/ssh-console/src/bmc/client_pool.rs Outdated
@chet

chet commented Jul 10, 2026

Copy link
Copy Markdown
Contributor Author

CI fix: the three failing tests were the network-segment exposition fixtures -- they byte-pin the whole scrape, so the state-controller SLA/handling-errors template wording (and the per-segment IP descriptions) from the review rounds had to land in the fixture files too. All three suites pass locally now, plus the firmware/validation fixture suites re-verified. That failing job was never the flake I first suspected -- my miss for syncing the catalogue but not these fixtures.

Every count-style metric description now opens with `Number of` -- the preferred prefix @polarweasel called out in the NVIDIA#3266 review -- instead of the previous mix of "The amount of", "The number of", "The total number of", "Count of", etc.

Primary callouts are:

- The vault client counters in `forge_vault.rs` (`carbide-api.vault.requests_*`) claimed to count "tls connections" -- a copy-paste from the bmc-proxy metrics. They describe Vault requests now.
- The `state-controller` templates lose their mangled grammar: "The amount of types that object handling tasks that have been ..." becomes "Number of object handling tasks that have been ...", and the state entered/exited counters now read "Number of times objects of type X have entered a certain state".
- Small grammar fixes ride along inside touched strings only -- "machine validation runs", "pairs that have been identified", "machines that report software components", and lowercase "ips"/"tls"/"tcp" become "IPs"/"TLS"/"TCP".
- The golden `/metrics` fixtures assert exposition output verbatim, so their `# HELP` lines get the same rewrite -- 67 lines across 11 fixture files.
- Time and duration descriptions ("The amount of time ...", "Total time ...") are not counts and stay untouched, as does the multi-kind `carbide_site_explorer_update_explored_endpoints_count` row.

Tests updated!

Verified by running `test_integration` locally -- the regenerated `core_metrics.md` comes out identical to what's committed here.

This supports NVIDIA#3307

Signed-off-by: Chet Nichols III <chetn@nvidia.com>
@chet

chet commented Jul 10, 2026

Copy link
Copy Markdown
Contributor Author

CI: the repo-clean check was failing because several catalogue rows had drifted from the code's descriptions during the review-round amends -- this push replaces the hand-synced rows with the regeneration's own output (local test_integration green, regen-stable by construction). The check should pass this cycle.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants