Skip to content

[fix] keep one-row metric data when command output is short - #4308

Open
orangeCatDeveloper wants to merge 1 commit into
apache:masterfrom
orangeCatDeveloper:fix/issue-1560-empty-command-output
Open

[fix] keep one-row metric data when command output is short#4308
orangeCatDeveloper wants to merge 1 commit into
apache:masterfrom
orangeCatDeveloper:fix/issue-1560-empty-command-output

Conversation

@orangeCatDeveloper

@orangeCatDeveloper orangeCatDeveloper commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

What's changed?

Fixes #1560.

Scenario: an SSH/script monitor uses oneRow parsing — each output line fills one field. The reporter's script ends with an existence check:

echo <cpu>; echo <memory>; echo <disk>
df -hT | grep -o "centos-hermitlv"   # mount present -> prints its name; gone -> prints nothing, exits 1

When the mount disappears, the script outputs 3 lines instead of 4 — and the user expects the last field to show empty so they can alert on it.

What actually happened — two code paths, both losing the data:

  1. Partial output (multi-field template, the reporter's case): parseResponseDataByOne rejected the whole row when output lines were fewer than alias fields, discarding the cpu/memory/disk values that were collected. The metric page went blank and every cycle logged ssh response data not enough. Ironically, the loop right below that length check already pads missing columns with NULL_VALUE — the check just never let short output reach it.
  2. Empty output (single-field template): any blank stdout was hard-failed with response data is null, treating "grep found nothing" (a meaningful answer) the same as "the command blew up".

Fix:

  • Partial output: remove the length check; missing trailing fields become NULL_VALUE, collected values survive.
  • Empty output: if there is no stderr and the exit status is <= 1 (the grep-no-match signature), emit a row of NULL_VALUE and succeed. Real failures still fail — and the message now carries the captured stderr or exit code instead of the fixed response data is null.
  • To tell those apart, both collectors now capture stderr and the exit status, which they previously discarded (ssh dumped remote stderr into the collector's own System.err; script never read it — it is drained on a daemon thread to avoid the classic stderr-pipe deadlock).
  • The two collectors carried identical copy-pasted one-row parsing; it is merged into OneRowResponseSupport. The script collector shares the same defect and is fixed together; no other collector has one-row command semantics.
  • Bonus: SshProtocol gains an optional charset (default UTF-8) for non-UTF-8 remote output, matching what ScriptProtocol already had.

Known tradeoff (pinned by an explicit test): a command that exits 1 silently with no output and no stderr is indistinguishable from a grep no-match and is accepted as empty data. Requiring exit 0 instead would break the reported use case itself, since grep exits 1 on no match.

With NULL_VALUE flowing through, the alert calculator sees the field as null, so rules like !exists(nfs_mount) can fire on "the mount is gone" — exactly the issue's ask.

Before / after

Verified against a real sshd container, using the reporter's script shape (echo 52; echo 35.8033; echo 5%; grep -o 'missing-mount' /proc/mounts) with 5 alias fields (cpu, memory, disk, nfs_mount, responseTime):

Before — no metric row is stored; the collector logs the same error as the issue's screenshot, every collect cycle:

ERROR SshCollectImpl Line:249 - ssh response data not enough: 52
35.8033
5%

After — the row is stored with the gap visible as null, and no errors are logged:

"origin": "52",        // cpu
"origin": "35.8033",   // memory
"origin": "5%",        // disk
"origin": null,        // nfs_mount  <- alertable via !exists(nfs_mount)
"origin": "14",        // responseTime

Checklist

  • I have read the Contributing Guide
  • I have written the necessary doc or comment.
  • I have added the necessary unit tests and all cases have passed.

Add or update API

  • I have added the necessary e2e tests and all cases have passed.

A trailing existence check like 'grep -o mount-name' emits nothing when
the target is gone, so the row was dropped (partial output) or the
whole collection failed (empty output) and the metric vanished exactly
when it mattered (apache#1560). Pad missing trailing fields with null, accept
blank output as empty data when stderr is empty and exit status <= 1,
and carry stderr/exit code into failure messages. Ssh and script
collectors shared the copy-pasted logic, now merged into
OneRowResponseSupport; ssh also gains an optional output charset.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Use the grep -o command to match fields. If the fields are not matched, the collection system will report an error.

1 participant