Preserve integer precision in ClickHouse query parameters#283
Open
damilolaedwards wants to merge 1 commit into
Open
Preserve integer precision in ClickHouse query parameters#283damilolaedwards wants to merge 1 commit into
damilolaedwards wants to merge 1 commit into
Conversation
Operation request bodies were decoded without UseNumber, so JSON numbers became float64. The ClickHouse parameter formatters then fell through to fmt.Sprint, which renders float64 with %g: any integer of 1e6 or more came out in scientific notation (7.5e+06), which ClickHouse cannot parse as an integer type, and integers above 2^53 lost precision before they were ever formatted. This broke parameterized queries filtering by block, slot, timestamp, or gas. Decode numbers as json.Number so values keep their exact form, teach optionalIntArg to read json.Number, and format json.Number verbatim (with a plain-decimal fallback for float64). Add tests covering large integers, arrays, precision above 2^53, and non-integer values.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Operation request bodies are decoded with a plain
json.Decoder, so every JSON number in a query'sparametersmap becomes afloat64. The ClickHouse parameter formatters (formatClickHouseParamValue,formatClickHouseArrayLiteral) have no numeric case and fall through tofmt.Sprint, which renders afloat64with%g.The result:
7500000becomes7.5e+06. ClickHouse cannot parse that as aUInt64/Int64, so the query fails.This breaks parameterized queries filtering by block number, slot, timestamp, or gas, which are the normal ranges on any real network. Array parameters have the same gap.
Fix
Decode numbers as
json.Number(UseNumber) so values keep their exact form, teachoptionalIntArgto readjson.Number(parseInt64Argalready did), and formatjson.Numberverbatim in both ClickHouse formatters, with a plain-decimal fallback forfloat64.Tests
Added tests covering large integers (block/slot/timestamp), array parameters, precision above 2^53, and non-integer values rendering without an exponent. The new tests fail on the old code and pass on the new code, and the full
pkg/serversuite passes (theUseNumberchange does not affect other operations).