feat(nico): add BFX03-01 hardware serial inventory + STG05-01 failure-domain observability

[abegnoche]

Summary

Implements two M7 NICo bare-metal observability checks. Both are provider-neutral validations grounded in the public NICo (infra-controller) REST API, following the established NICo-validation pattern (#457/#458/#466). Neither is feasible on AWS: EC2 virtualizes hardware (no real chassis/baseboard/NIC serials) and exposes no physical rack/enclosure topology. Both ship unreleased (released_tests.json untouched); exercise with ISVTEST_INCLUDE_UNRELEASED=1.

Issue	Test ID	Check	Source
#320 BFX03-01	BFX03-01	HardwareSerialCheck	machine.metadata.{dmiData,gpus,networkInterfaces,infinibandInterfaces} + machineCapabilities (via includeMetadata=true)
#361 STG05-01	STG05-01	FailureDomainObservabilityCheck	machine.labels — RackIdentifier or rack

Changes

BFX03-01 (#320) — hardware serial-number inventory

• query_serial_numbers.py: reduces each machine's discovered hardware metadata to a per-component stable-identifier record — chassis/baseboard DMI serials, GPU serials, NIC MAC/GUID, CPU model. Chassis falls back to the provider-visible machine serialNumber.
• HardwareSerialCheck (hardware.py): asserts a stable identifier is queryable for every present component (chassis, baseboard, CPU, GPU, NIC). Absent components (e.g. GPUs on a CPU-only node) are reported as skipped subtests, never failures. Per BFX03-01, obfuscated-but-stable IDs are accepted; CPUs have no per-socket DMI serial, so the stable CPU model descriptor is the identifier.

STG05-01 (#361) — failure-domain topology observability

• query_topology.py: reads each host's rack/failure-domain identifier from machine labels (RackIdentifier, then rack) and emits per-host records (host_id, failure_domain).
• FailureDomainObservabilityCheck (topology.py): asserts every host maps to a named failure domain and the site meets a configurable diversity floor (min_failure_domains, default 1 = observability only).

Orchestration

• NICo bare_metal.yaml: mark all standalone query steps continue_on_failure: true so unrelated probe failures (e.g. IB tenant isolation when the org has no tenant) do not skip later independent steps such as serial inventory and topology observability. Matches the pattern already used in NICo observability.yaml.

Refactor (no outcome change)

• Trim topology JSON contract to consumed fields only (hosts[]; no derived observed / failure_domains aggregates).
• Reuse common.inventory.first_string for rack-label extraction; limit keys to documented RackIdentifier / rack.
• Read min_* config knobs via BaseValidation._parse_positive_int.

Wiring / docs

• NICo bare_metal.yaml: query_serial_numbers + query_topology steps.
• suites/bare_metal.yaml: hardware_serials (BFX03-01) + failure_domain_observability (STG05-01) groups.
• suites/README.md step tables + docs/test-plan.yaml label unions updated.

Test plan

• make lint && make test (1220 + 97 pass)
• make demo-test
• uv run pytest isvtest/tests/test_hardware.py isvtest/tests/test_topology.py isvctl/tests/providers/nico/test_nico_provider.py (new validation + script contract tests)
• End-to-end via the orchestrator against a local stub (ISVTEST_INCLUDE_UNRELEASED=1): both checks PASS on healthy inventory; a present GPU with no serial and an unlabeled host both flow through to actionable failures.

Closes #320
Closes #361

  

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[coderabbitai]

📝 Walkthrough

Walkthrough

Added NICo bare-metal checks for stable hardware identifiers and physical failure domains. The changes include provider scripts, validation classes, suite configuration, execution settings, documentation, and unit/integration coverage.

Changes

Bare-metal observability

Layer / File(s)	Summary
Provider-agnostic validation checks isvtest/src/isvtest/validations/hardware.py, isvtest/src/isvtest/validations/topology.py, isvtest/tests/test_hardware.py, isvtest/tests/test_topology.py	Adds HardwareSerialCheck and FailureDomainObservabilityCheck, including threshold, missing-data, unmapped-host, and step-failure handling with corresponding tests.
NICo inventory and topology queries isvctl/configs/providers/nico/scripts/hardware_inventory/query_serial_numbers.py, isvctl/configs/providers/nico/scripts/topology/query_topology.py	Adds CLI queries that emit structured hardware identifier and host failure-domain results, including skipped and error responses.
Bare-metal suite wiring and execution isvctl/configs/providers/nico/config/bare_metal.yaml, isvctl/configs/suites/bare_metal.yaml, isvctl/configs/suites/README.md, docs/test-plan.yaml	Registers both checks, documents expected outputs and labels, and enables continued execution for existing NICo steps.
NICo provider integration coverage isvctl/tests/providers/nico/test_nico_provider.py	Adds mocked API coverage for identifier mapping, GPU absence, chassis fallback, rack extraction, empty sites, and validation results.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant BareMetalSuite
  participant query_serial_numbers.py
  participant query_topology.py
  participant NICoAPI
  participant HardwareSerialCheck
  participant FailureDomainObservabilityCheck
  BareMetalSuite->>query_serial_numbers.py: Run hardware inventory query
  query_serial_numbers.py->>NICoAPI: Fetch site machines with metadata
  NICoAPI-->>query_serial_numbers.py: Return machine hardware metadata
  query_serial_numbers.py-->>HardwareSerialCheck: Provide component identifiers
  BareMetalSuite->>query_topology.py: Run topology query
  query_topology.py->>NICoAPI: Fetch site machines
  NICoAPI-->>query_topology.py: Return machine labels
  query_topology.py-->>FailureDomainObservabilityCheck: Provide host failure domains

Loading

Suggested reviewers: mresvanis, huaweic-nv, jkenyon-nvidia

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	The PR adds the BFX03-01 serial inventory and STG05-01 topology checks, plus scripts, wiring, and tests for both.
Out of Scope Changes check	✅ Passed	The changes stay focused on the two linked NICo validation features and their docs, configs, and tests.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely captures the two main additions: BFX03-01 hardware serial inventory and STG05-01 failure-domain observability.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch cursor/nico-hardware-inventory-observability-45cf

---

_{Comment @coderabbitai help to get the list of available commands.}

[abegnoche]

/ok to test 0aaf6d1

[github-actions]

🔐 TruffleHog Secret Scan

✅ No secrets or credentials found!

Your code has been scanned for 700+ types of secrets and credentials. All clear! 🎉

🔗 View scan details

_{🕐 Last updated: 2026-07-13 21:40:38 UTC | Commit: 0aaf6d1}

[coderabbitai]

🧹 Nitpick comments (1)

isvctl/configs/providers/nico/scripts/topology/query_topology.py (1)

82-82: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Annotate RACK_LABEL_KEYS.

Proposed fix

-RACK_LABEL_KEYS = ("RackIdentifier", "rack")
+RACK_LABEL_KEYS: tuple[str, ...] = ("RackIdentifier", "rack")

As per coding guidelines, “All Python files must use type annotations following PEP 585.”

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@isvctl/configs/providers/nico/scripts/topology/query_topology.py` at line 82,
Annotate the RACK_LABEL_KEYS constant with an appropriate PEP 585 type,
preserving its tuple of string values and existing ordering.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@isvctl/configs/providers/nico/scripts/topology/query_topology.py`:
- Line 82: Annotate the RACK_LABEL_KEYS constant with an appropriate PEP 585
type, preserving its tuple of string values and existing ordering.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: a1c8e815-677f-4f11-92be-a8163328bf44

📥 Commits

Reviewing files that changed from the base of the PR and between 129af16 and 0aaf6d1.

📒 Files selected for processing (11)

• docs/test-plan.yaml
• isvctl/configs/providers/nico/config/bare_metal.yaml
• isvctl/configs/providers/nico/scripts/hardware_inventory/query_serial_numbers.py
• isvctl/configs/providers/nico/scripts/topology/query_topology.py
• isvctl/configs/suites/README.md
• isvctl/configs/suites/bare_metal.yaml
• isvctl/tests/providers/nico/test_nico_provider.py
• isvtest/src/isvtest/validations/hardware.py
• isvtest/src/isvtest/validations/topology.py
• isvtest/tests/test_hardware.py
• isvtest/tests/test_topology.py