From dff782db1f86bb6ed45fe1bf559e4d459201c2fe Mon Sep 17 00:00:00 2001 From: Robert Yokota Date: Sun, 14 Jun 2026 19:37:35 -0700 Subject: [PATCH] Fix UB in number tokenizer: use pointer arithmetic instead of dereferencing end iterator MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Summary The number-literal scanner in `Tokenizer.cpp` passed `&*byteStart` and `&*byteEnd` to `std::regex_search`, where `byteEnd` is `path_.cend()`. Dereferencing a past-the-end iterator (`*path_.cend()`) is **undefined behavior**, even when only its address is taken. Most standard libraries (libstdc++, libc++) happen to tolerate this, but MSVC's checked iterators (`_ITERATOR_DEBUG_LEVEL`) correctly reject it, causing an assertion failure / test crash on Windows MSVC 2022 builds. ## Fix Form the regex range from `path_.data()` pointers instead of dereferencing iterators: ```cpp const char* byteStart = path_.data() + byte_offsets_[position_]; const char* byteEnd = path_.data() + path_.size(); std::cmatch match; if (std::regex_search(byteStart, byteEnd, match, numregex) && ...) ``` `data() + size()` is a valid one-past-the-end **pointer** (legal to form, never dereferenced), which is exactly what `std::regex_search` expects for its `[first, last)` range. `std::cmatch` already operates on `const char*` ranges, so the surrounding match handling is unchanged. No behavioral change on conforming platforms — this only removes the UB that MSVC flags. ## Testing - All existing generated tests pass. Reported by an external contributor building with MSVC 2022 on Windows. Co-Authored-By: Claude Opus 4.8 (1M context) --- src/jsonata/Tokenizer.cpp | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/src/jsonata/Tokenizer.cpp b/src/jsonata/Tokenizer.cpp index 066e2ad..3aec3e9 100644 --- a/src/jsonata/Tokenizer.cpp +++ b/src/jsonata/Tokenizer.cpp @@ -372,11 +372,13 @@ std::unique_ptr Tokenizer::next(bool prefix) { { static const std::regex numregex( "^-?(0|([1-9][0-9]*))(\\.[0-9]+)?([Ee][-+]?[0-9]+)?"); - // Use byte offsets to get iterators into the original string without copying - auto byteStart = path_.cbegin() + static_cast(byte_offsets_[position_]); - auto byteEnd = path_.cend(); + // Use byte offsets to get pointers into the original string without copying. + // Use pointer arithmetic on data() rather than dereferencing iterators: + // &*path_.cend() dereferences a past-the-end iterator + const char* byteStart = path_.data() + byte_offsets_[position_]; + const char* byteEnd = path_.data() + path_.size(); std::cmatch match; - if (std::regex_search(&*byteStart, &*byteEnd, match, numregex) && + if (std::regex_search(byteStart, byteEnd, match, numregex) && match.position() == 0) { std::string numStr = match.str(0); double num = std::stod(numStr);