Skip to content

RMS firmware update error reporting #3422

Description

@tjcanro

Is this a new feature, an enhancement, or a change to existing functionality?

Enhancement

How would you describe the priority of this feature request

High

Please provide a clear description of problem this feature solves

This enhancement solves the firmware update debugging gap between NICo, component-manager, RMS, and operator-facing status output.

The supported firmware update path can accept and track update requests, but the user-facing status, component state, and logs do not consistently surface enough detail to diagnose failures. In testing, firmware update status could report Result=success while the firmware update state was failed, queued, or in-progress, and the Error field remained empty. This makes it unclear whether “success” means the update succeeded or only that the request/status API call succeeded.

Component-level state later showed RMS-reported failures, but the exposed reason was usually generic, such as RMS reported failure for . That does not identify the failing firmware component, firmware type, target version, failure phase, retryability, RMS job ID, or underlying RMS error. For rack-scoped updates, rack show could report aggregate failure counts such as firmware upgrade failed: 16/17 devices failed, but without a per-component breakdown. For switch-scoped updates, the update request could return success while the follow-up status path reported no firmware job tracked for this switch, creating ambiguity about whether the job was scheduled, persisted, completed, or queried through the wrong path.

This leaves operators with a failed firmware update but no clear failure source, no obvious correlation path between CLI output, NICo state, and RMS logs, and no reliable post-update firmware version verification path.

Feature Description

Improve RMS/NICo firmware update error reporting so firmware update failures are actionable, consistent, and traceable from the operator-facing CLI down to RMS job/log details.

The current component-manager firmware update flow should preserve enough error context for operators to understand what failed, where it failed, and what to inspect next. Failed firmware update status should populate meaningful Error information instead of -. The output should clearly distinguish request/API success from firmware job state and firmware job result, so operators do not misread Result=success as a successful firmware update when the job state is actually failed.

Rack-scoped firmware updates should expose a per-component status breakdown, including component ID, component type, current state, target version, current version if known, firmware type, failure reason, last updated timestamp, and RMS job or correlation ID. Host, switch, and power shelf state should preserve RMS root-cause details instead of reducing failures to RMS reported failure for .

Switch-scoped firmware update tracking should be made consistent. If a switch update request is accepted, the system should return or expose the firmware job ID and ensure follow-up status queries use the same tracked job record. If switch-scoped firmware updates are not fully supported in a given release or path, the request/status output should say so clearly.

Post-update verification should also be supported. Operators need a reliable way to determine the targeted firmware version, observed/current firmware version, which components succeeded, which failed, and which remain unknown. This could be surfaced through firmware update status, component-manager get-firmware-versions, component state commands such as mh show, ms show, sw show, and ps show, or a dedicated per-component firmware inventory view.

Describe your ideal solution

No response

Describe any alternatives you have considered

No response

Additional context

No response

Code of Conduct

  • I agree to follow NVIDIA Infra Controller's Code of Conduct
  • I have searched the open feature requests and have found no duplicates for this feature request

Metadata

Metadata

Assignees

Labels

featureFeature (deprecated - use issue type, but it's needed for reporting now)rack lifecycleIssues that relate to managing the lifecycle of a full rack (compute, switches and powershelves)

Fields

No fields configured for Enhancement.

Projects

Status
Triage

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions