Skip to content

[fix] strip NUL bytes before tdengine history insert - #4310

Open
orangeCatDeveloper wants to merge 1 commit into
apache:masterfrom
orangeCatDeveloper:fix/1481-tdengine-control-chars
Open

[fix] strip NUL bytes before tdengine history insert#4310
orangeCatDeveloper wants to merge 1 commit into
apache:masterfrom
orangeCatDeveloper:fix/1481-tdengine-control-chars

Conversation

@orangeCatDeveloper

@orangeCatDeveloper orangeCatDeveloper commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

What's changed?

Fixes #1481: Windows SNMP interface descriptions carry NUL padding bytes (\^@). The TDengine history storage builds its INSERT by hand, so the NUL reached the SQL text, TDengine rejected it with error 216, and the whole batch was lost — including the healthy rows.

Fix in TdEngineDataStorage.formatStringValue, the single choke point all string values, labels json and the instance tag pass through:

  • strip \u0000 before anything else (only NUL — tabs/newlines are legitimate data)
  • truncate the logical value before SQL-escaping: the old escape-then-truncate order could cut an escape pair like \' in half, leaving a dangling backslash that escapes the closing quote and kills the batch the same way

Only TDengine is affected: the other history backends use typed clients / JSON / line protocol instead of hand-built SQL.

Verified against a real TDengine 3.3.5.0 with a metric whose string value ends in \u0000 (one poisoned row, one healthy row per batch):

Before — every batch rejected, both rows lost:

ERROR TdEngineDataStorage - TDengine ERROR (216): sql: INSERT INTO `rows1481_rows_null_80_v2`
USING `rows1481_rows_super_v2` TAGS ('null:80') VALUES
(1786324976558, '{}', 1.0, 'Software Loopback Interface 1', 10.0)
(1786324976559, '{}', 2.0, 'normal-item', 20.0),
desc: syntax error near ''Software Loopback Interface 1' (invalid data or symbol)

taos> select * from hertzbeat.rows1481_rows_super_v2 limit 5;
Query OK, 0 row(s) in set

After — no errors, both rows stored, NUL stripped:

taos> select * from hertzbeat.rows1481_rows_super_v2 limit 5;
           ts            | seq |             name               | value |
==========================================================================
 2026-08-10 01:25:43.746 | 1.0 | Software Loopback Interface 1  | 10.0  |
 2026-08-10 01:25:43.747 | 2.0 | normal-item                    | 20.0  |
Query OK, 4 row(s) in set

Checklist

  • I have read the Contributing Guide
  • I have written the necessary doc or comment.
  • I have added the necessary unit tests and all cases have passed.

Add or update API

  • I have added the necessary e2e tests and all cases have passed.

Windows snmp interface descriptions carry NUL padding that made the
hand-built INSERT fail with error 216 and lose the whole batch (apache#1481).
Remove NUL from string values before escaping, and truncate the logical
value before escaping so the cut cannot split an escape sequence.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] <using tdengine 3.1.1.12 error>

1 participant