Skip to content

feat(health): allow excluding noisy LogServices in periodic log collection#3349

Open
cdraman wants to merge 1 commit into
NVIDIA:mainfrom
cdraman:HAAS-169
Open

feat(health): allow excluding noisy LogServices in periodic log collection#3349
cdraman wants to merge 1 commit into
NVIDIA:mainfrom
cdraman:HAAS-169

Conversation

@cdraman

@cdraman cdraman commented Jul 10, 2026

Copy link
Copy Markdown

When forge-hw-health runs in periodic log collection mode, it enumerates
every Redfish LogService on the BMC — including the bmcweb Journal, which
is the BMC web server's own HTTP access log.

This creates a self-reinforcing noise loop: every Redfish poll we make is
itself recorded as a Journal entry. The next poll collects those entries,
which generates more entries. Volume grows as a function of our own polling
rate, not real hardware events, drowning out meaningful signal like GPU XID
faults and CPER records.

Add exclude_services to collectors.logs.periodic — a list of substrings
matched against each LogService @odata.id. Matching services are skipped
at discovery time before any entries are fetched.

Related issues

Closes #3344

Type of Change

  • Add - New feature or capability
  • Change - Changes in existing functionality
  • Fix - Bug fixes
  • Remove - Removed features or deprecated functionality
  • Internal - Internal changes (refactoring, tests, docs, etc.)

Breaking Changes

  • This PR contains breaking changes

Testing

  • Unit tests added/updated
  • Integration tests added/updated
  • Manual testing performed
  • No testing required (docs, internal refactor, etc.)

Additional Notes

@cdraman cdraman requested a review from a team as a code owner July 10, 2026 00:44
@coderabbitai

coderabbitai Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 77005d70-e90e-4bf7-b0a7-928ec823f247

📥 Commits

Reviewing files that changed from the base of the PR and between 790262c and a985af8.

📒 Files selected for processing (5)
  • crates/health/example/config.bmc-mock-periodic.toml
  • crates/health/example/config.example.toml
  • crates/health/src/collectors/logs/periodic.rs
  • crates/health/src/config.rs
  • crates/health/src/discovery/spawn.rs
✅ Files skipped from review due to trivial changes (2)
  • crates/health/src/discovery/spawn.rs
  • crates/health/example/config.example.toml
🚧 Files skipped from review as they are similar to previous changes (3)
  • crates/health/example/config.bmc-mock-periodic.toml
  • crates/health/src/config.rs
  • crates/health/src/collectors/logs/periodic.rs

Summary by CodeRabbit

  • New Features
    • Added an option for periodic log collection to exclude selected Redfish LogService entries using configurable substring patterns matched against each service’s @odata.id.
    • Log discovery now reports both the total discovered services and how many were excluded.
  • Documentation
    • Updated example configuration(s) to document and demonstrate the new exclude_services setting (including skipping “Journal” log noise).
  • Tests
    • Added unit tests covering empty exclusions, substring matching, multi-pattern exclusion, and case-sensitive behavior.

Walkthrough

Periodic log collection now accepts configurable, case-sensitive substring filters for Redfish LogService OData IDs. Matching services are skipped during discovery, exclusion counts are logged, defaults preserve existing behavior, and example configurations and unit tests are added.

Changes

Periodic LogService Exclusion

Layer / File(s) Summary
Configuration contract and examples
crates/health/src/config.rs, crates/health/example/config.example.toml, crates/health/example/config.bmc-mock-periodic.toml
Periodic configuration adds exclude_services with an empty default; examples document and configure Journal exclusion.
Collector configuration wiring
crates/health/src/discovery/spawn.rs, crates/health/src/collectors/logs/periodic.rs
Periodic startup passes exclusion patterns into LogsCollector, which stores them for discovery.
Service discovery filtering and validation
crates/health/src/collectors/logs/periodic.rs
Manager, chassis, and system LogService discovery filters matching OData IDs, de-duplicates retained services, reports exclusion counts, and tests matching behavior.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant PeriodicLogConfig
  participant spawn_generic_redfish_collectors
  participant LogsCollector
  participant RedfishBMC
  PeriodicLogConfig->>spawn_generic_redfish_collectors: provide exclude_services
  spawn_generic_redfish_collectors->>LogsCollector: create LogsCollectorConfig
  LogsCollector->>RedfishBMC: discover LogServices
  LogsCollector->>LogsCollector: filter matching OData IDs
  LogsCollector-->>LogsCollector: log total and excluded service counts
Loading
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 55.56% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title accurately summarizes the new ability to exclude noisy LogServices during periodic log collection.
Description check ✅ Passed The description is directly related to the code changes and clearly explains the periodic LogService exclusion feature.
Linked Issues check ✅ Passed The changes satisfy the linked issue requirements: new exclude_services config, substring matching, exclusion before fetch, reporting, and preserved defaults.
Out of Scope Changes check ✅ Passed The PR appears scoped to the requested log-service filtering feature and related examples/tests, with no obvious unrelated changes.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown

🔍 Container Scan Summary

Service Total Critical High Medium Low Other
boot-artifacts-aarch64 3 0 0 3 0 0
boot-artifacts-x86_64 3 0 0 3 0 0
forge-admin-cli-x86_64 271 13 34 91 7 126
machine-validation-runner 800 37 234 291 43 195
machine_validation 800 37 234 291 43 195
machine_validation-aarch64 800 37 234 291 43 195
nvmetal-carbide 800 37 234 291 43 195
TOTAL 3477 161 970 1261 179 906

Per-CVE detail lives in the per-service grype-* artifacts (JSON + SARIF). Severity counts only — no CVE IDs published here.

@cdraman

cdraman commented Jul 10, 2026

Copy link
Copy Markdown
Author

@coderabbitai full review

@coderabbitai

coderabbitai Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor
✅ Action performed

Full review finished.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
crates/health/src/collectors/logs/periodic.rs (1)

427-481: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Prefer check_values over ad-hoc #[test] functions for this matcher.

service_is_excluded is a pure validator with two inputs (pattern list, service id) and a boolean output — exactly the case the style guide calls out for table-driven tests. As per coding guidelines, "Use check_cases / check_values directly when a macro would obscure a table with several inputs or several expected fields per row in table-driven tests." A single check_values table covering all five current scenarios (empty list, substring match, non-match, multi-pattern, case sensitivity) would replace the five separate #[test] functions with one enumerable table and make it trivial to add future cases.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/health/src/collectors/logs/periodic.rs` around lines 427 - 481,
Replace the five ad-hoc tests in the tests module with one table-driven
check_values test covering empty exclusions, substring matches, non-matches,
multiple patterns, and case sensitivity. Keep each case’s pattern list, service
ID, and expected boolean explicit, and use the project’s existing check_values
convention while continuing to exercise service_is_excluded.

Sources: Coding guidelines, Path instructions

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/health/src/collectors/logs/periodic.rs`:
- Around line 417-425: Update service_is_excluded to ignore empty
exclude_services patterns before performing substring matching, ensuring only
non-empty patterns are checked against service_id.

---

Nitpick comments:
In `@crates/health/src/collectors/logs/periodic.rs`:
- Around line 427-481: Replace the five ad-hoc tests in the tests module with
one table-driven check_values test covering empty exclusions, substring matches,
non-matches, multiple patterns, and case sensitivity. Keep each case’s pattern
list, service ID, and expected boolean explicit, and use the project’s existing
check_values convention while continuing to exercise service_is_excluded.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 515f362c-71e1-4934-99a0-26ac9837da9e

📥 Commits

Reviewing files that changed from the base of the PR and between 699951d and 790262c.

📒 Files selected for processing (5)
  • crates/health/example/config.bmc-mock-periodic.toml
  • crates/health/example/config.example.toml
  • crates/health/src/collectors/logs/periodic.rs
  • crates/health/src/config.rs
  • crates/health/src/discovery/spawn.rs

Comment thread crates/health/src/collectors/logs/periodic.rs
…ction

The periodic log collector enumerates and drains every Redfish LogService
under Managers/Chassis/Systems. On real BMCs this includes the bmcweb
"Journal" HTTP-access log, which is high-volume noise and self-referential:
our own Redfish polling generates Journal entries that the next poll
re-collects, so volume grows as a function of our own polling and can
drown the real signal (e.g. GPU XID events).

Add a `collectors.logs.periodic.exclude_services` option: a list of
substrings matched against each LogService odata id; matching services
are skipped during discovery. Empty (default) preserves current behavior.
Periodic mode only — SSE never enumerates LogServices.

- config: new `exclude_services: Vec<String>` on PeriodicLogConfig
  (#[serde(default)], empty default; backward compatible)
- periodic: apply the filter in discover_log_services() and report an
  `excluded_services` count; extract the predicate into a testable
  free function with unit tests
- spawn: thread the field into the collector config
- examples: document the option in config.example.toml and add a
  runnable config.bmc-mock-periodic.toml demonstrating exclude_services

Signed-off-by: Dasa Chandramouli <dchandramoul@nvidia.com>
@yoks

yoks commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Lets make exclude Journal by default, seems like this behavior is not desired in vast majority of cases

# Typically used to drop the bmcweb "Journal" HTTP-access log (high-volume noise,
# and self-referential since polling generates Journal entries). Empty (default)
# collects from every discovered LogService. Periodic mode only.
exclude_services = []

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to also define this config attrubute on L198.

Auto mode accepts both sse and periodic config variables for fallback behavior

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: Allow excluding noisy LogServices in periodic log collection in health

3 participants