Skip to content

feat: NV_ERR_NO_MEMORY on GPU panel (slow journal scan) - #48

Closed
MiaAI-Lab wants to merge 1 commit into
mainfrom
feat/nv-err-no-memory
Closed

feat: NV_ERR_NO_MEMORY on GPU panel (slow journal scan)#48
MiaAI-Lab wants to merge 1 commit into
mainfrom
feat/nv-err-no-memory

Conversation

@MiaAI-Lab

Copy link
Copy Markdown
Owner

Why this replaces #40

#40 put an NV_ERR_NO_MEMORY counter on CpuPanel and ran journalctl -k | grep -c on the unified-memory poll (~2s). CpuPanel is gone, and a full kernel-journal scan every 2s is not a metric.

What this does

  • Count NVRM NV_ERR_NO_MEMORY lines in the kernel journal since boot
  • Attach it to GPU metrics and show it on the GPU panel when the count is > 0
  • Scan at most once per POLL_INTERVAL_NVERR (default 60s), cached — not on the 2s GPU/memory loop
  • Local Docker uses host nsenter (same as nvidia-smi); remotes use SSH
  • No per-browser localStorage reset; the number is journal-since-boot
  • Parser unit tests only (no --experimental-test-module-mocks)

Test plan

  • node --test server/collectors/__tests__/SystemCollector.nvErr.test.js
  • npm run typecheck
  • On a Spark/host with no NVRM errors: GPU panel unchanged (row hidden)
  • After NV_ERR_NO_MEMORY in journalctl -k: row appears on GPU panel, not a CPU panel

Kernel journal scan is cached off the 2s GPU loop. Count is since boot; the row is hidden at zero. No CPU panel.

Co-authored-by: Cursor <cursoragent@cursor.com>
@vincenzopalazzo

Copy link
Copy Markdown
Collaborator

ACK — NV_ERR_NO_MEMORY on GPU panel

Production Readiness: GO

Minimal. Cache + POLL_INTERVAL_NVERR is the load-bearing choice vs #42. Take this version. Merge.

— Vincent

@vincenzopalazzo

Copy link
Copy Markdown
Collaborator

Already landed via #78 (Batch A ACK merge onto main). Closing this one as superseded — thanks for the contribution!

— Vincent

@vincenzopalazzo

Copy link
Copy Markdown
Collaborator

Correction: #78 is ready but not yet merged (needs MiaAI-Lab write access). Once #78 is rebase-merged, please close this PR as superseded. Thanks again!

— Vincent

@vincenzopalazzo

Copy link
Copy Markdown
Collaborator

Already merged via #78. Thanks for the contribution!

— Vincent

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants