GH-51037: [C++] Replace RapidJSON with simdjson in JSON parser by Reranko05 · Pull Request #51038 · apache/arrow

Reranko05 · 2026-08-29T20:11:07Z

Rationale for this change

This PR continues the simdjson migration by replacing the RapidJSON-based parsing implementation used by the JSON parser.

The existing parser uses RapidJSON's SAX/handler interface to parse JSON values and populate Arrow builders. This change replaces that implementation with simdjson's ondemand API while retaining the existing builder and type-inference logic.

Changes

Replace the RapidJSON parser and handler interface with simdjson's ondemand API.
Parse JSON documents using simdjson::ondemand::parser::iterate_many.
Use ResolveSimdjsonResult() consistently when handling simdjson results.
Preserve support for nested objects and arrays.
Preserve explicit-schema and inferred-field behavior.
Preserve unexpected-field handling for Error, Ignore, and InferType.
Continue storing numeric values as raw JSON tokens.
Trim trailing whitespace from numeric raw tokens to preserve existing behavior.
Preserve JSON parse error propagation through Status::Invalid.
Remove the parser's RapidJSON-specific dependencies.

Fixes: #51037

GitHub Issue: [C++] Replace RapidJSON with simdjson in JSON parser #51037

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Reranko05 · 2026-09-07T03:41:30Z

@pitrou @kou Should we use padded_string_view with a reusable buffer here as well, similar to the chunker?

Reranko05 · 2026-09-07T04:36:11Z

@github-actions crossbow submit -g cpp

github-actions · 2026-09-07T04:39:08Z

Revision: a4dea4b

Submitted crossbow builds: ursacomputing/crossbow @ actions-9fde492db2

Task	Status
example-cpp-minimal-build-static
example-cpp-minimal-build-static-system-dependency
example-cpp-tutorial
test-build-cpp-fuzz
test-conda-cpp
test-conda-cpp-valgrind
test-debian-13-cpp-amd64
test-debian-13-cpp-i386
test-debian-experimental-cpp-gcc-15
test-fedora-42-cpp
test-ubuntu-22.04-cpp
test-ubuntu-22.04-cpp-bundled
test-ubuntu-22.04-cpp-emscripten
test-ubuntu-22.04-cpp-no-threading
test-ubuntu-24.04-cpp
test-ubuntu-24.04-cpp-bundled-offline
test-ubuntu-24.04-cpp-gcc-13-bundled
test-ubuntu-24.04-cpp-gcc-14
test-ubuntu-24.04-cpp-minimal-with-formats
test-ubuntu-24.04-cpp-thread-sanitizer

pitrou · 2026-09-07T11:06:37Z

@pitrou @kou Should we use padded_string_view with a reusable buffer here as well, similar to the chunker?

We should check the Buffer capacity first to see if it has enough padding already.

Reranko05 · 2026-09-08T14:21:58Z

@github-actions crossbow submit -g cpp

github-actions · 2026-09-08T14:28:19Z

Revision: 713496d

Submitted crossbow builds: ursacomputing/crossbow @ actions-884a0df4e2

Task	Status
example-cpp-minimal-build-static
example-cpp-minimal-build-static-system-dependency
example-cpp-tutorial
test-build-cpp-fuzz
test-conda-cpp
test-conda-cpp-valgrind
test-debian-13-cpp-amd64
test-debian-13-cpp-i386
test-debian-experimental-cpp-gcc-15
test-fedora-42-cpp
test-ubuntu-22.04-cpp
test-ubuntu-22.04-cpp-bundled
test-ubuntu-22.04-cpp-emscripten
test-ubuntu-22.04-cpp-no-threading
test-ubuntu-24.04-cpp
test-ubuntu-24.04-cpp-bundled-offline
test-ubuntu-24.04-cpp-gcc-13-bundled
test-ubuntu-24.04-cpp-gcc-14
test-ubuntu-24.04-cpp-minimal-with-formats
test-ubuntu-24.04-cpp-thread-sanitizer

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Reranko05 · 2026-09-08T15:47:53Z

@pitrou @rok I’ve rebased the parser PR and incorporated the latest feedback, including reusing the simdjson::ondemand::parser and using padded_string_view when the Arrow Buffer has sufficient capacity.

The native C++ build and tests pass, but Crossbow’s ubuntu-cpp-emscripten job now fails in arrow-dataset-file-json-test:

RuntimeError: Aborted(). Build with -sASSERTIONS for more info.
    at abort (/build/cpp/debug/arrow-dataset-file-json-test.js:491:11)
    at _abort (/build/cpp/debug/arrow-dataset-file-json-test.js:4483:7)
    at invoke_v (/build/cpp/debug/arrow-dataset-file-json-test.js:5907:29)
    at arrow-dataset-file-json-test.wasm.std::__terminate(void (*)()) (wasm://wasm/arrow-dataset-file-json-test.wasm-0f1ca1f2:wasm-function[41574]:0x277e6cb)
    at arrow-dataset-file-json-test.wasm.std::terminate() (wasm://wasm/arrow-dataset-file-json-test.wasm-0f1ca1f2:wasm-function[41572]:0x277e6a4)
    at arrow-dataset-file-json-test.wasm.simdjson::fallback::ondemand::document_stream::start() (wasm://wasm/arrow-dataset-file-json-test.wasm-0f1ca1f2:wasm-function[20092]:0x10168d0)
    at arrow-dataset-file-json-test.wasm.arrow::Status arrow::Status arrow::json::HandlerBase::DoParse<arrow::json::Handler<(arrow::json::UnexpectedFieldBehavior)2>>(arrow::json::Handler<(arrow::json::UnexpectedFieldBehavior)2>&, std::__2::shared_ptr<arrow::Buffer> const&)::'lambda'(arrow::json::Handler<(arrow::json::UnexpectedFieldBehavior)2> const&)::operator()<simdjson::padded_string>(arrow::json::Handler<(arrow::json::UnexpectedFieldBehavior)2> const&) const (wasm://wasm/arrow-dataset-file-json-test.wasm-0f1ca1f2:wasm-function[20180]:0x1030c99)
    at invoke_viii (/build/cpp/debug/arrow-dataset-file-json-test.js:5885:29)
    at arrow-dataset-file-json-test.wasm.arrow::Status arrow::json::HandlerBase::DoParse<arrow::json::Handler<(arrow::json::UnexpectedFieldBehavior)2>>(arrow::json::Handler<(arrow::json::UnexpectedFieldBehavior)2>&, std::__2::shared_ptr<arrow::Buffer> const&) (wasm://wasm/arrow-dataset-file-json-test.wasm-0f1ca1f2:wasm-function[20178]:0x102f33c)
    at arrow-dataset-file-json-test.wasm.arrow::json::Handler<(arrow::json::UnexpectedFieldBehavior)2>::Parse(std::__2::shared_ptr<arrow::Buffer> const&) (wasm://wasm/arrow-dataset-file-json-test.wasm-0f1ca1f2:wasm-function[20177]:0x102ef89)

Do you have any idea what I need to do to fix this?

pitrou · 2026-09-08T16:48:35Z

@Reranko05 No idea without taking a deeper look :-) But I'd like us to merge the chunker PR first and then come back to this one.

Reranko05 · 2026-09-08T16:50:07Z

@pitrou Okay, will wait until chunker is merged.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Reranko05 · 2026-09-16T09:04:31Z

@github-actions crossbow submit test-ubuntu-22.04-cpp-emscripten

github-actions · 2026-09-16T09:08:59Z

Revision: 277fa60

Submitted crossbow builds: ursacomputing/crossbow @ actions-35754b78ac

Task	Status
test-ubuntu-22.04-cpp-emscripten

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Reranko05 · 2026-09-16T09:42:50Z

@github-actions crossbow submit test-ubuntu-22.04-cpp-emscripten

github-actions · 2026-09-16T09:45:57Z

Revision: b51b310

Submitted crossbow builds: ursacomputing/crossbow @ actions-84aeb7fa19

Task	Status
test-ubuntu-22.04-cpp-emscripten

Reranko05 · 2026-09-16T11:18:07Z

@pitrou @rok I tried a couple of things to narrow down the Emscripten failure:

Kept the persistent simdjson::ondemand::parser and added a try/catch around the document stream iteration — the failure remained unchanged.
Changed it to a local simdjson::ondemand::parser inside DoParse() to rule out parser reuse — the failure is still the same.

In both cases the Emscripten job fails at:

[ RUN      ] TestJsonFormat.IsSupported
Aborted()
/build/cpp/debug/arrow-dataset-file-json-test.js:138
      throw ex;
      ^

RuntimeError: Aborted(). Build with -sASSERTIONS for more info.
    at abort (/build/cpp/debug/arrow-dataset-file-json-test.js:491:11)
    at _abort (/build/cpp/debug/arrow-dataset-file-json-test.js:4483:7)
    at invoke_v (/build/cpp/debug/arrow-dataset-file-json-test.js:5907:29)
    at arrow-dataset-file-json-test.wasm.std::__terminate(void (*)()) (wasm://wasm/arrow-dataset-file-json-test.wasm-0f1ebc22:wasm-function[41577]:0x2787242)
    at arrow-dataset-file-json-test.wasm.std::terminate() (wasm://wasm/arrow-dataset-file-json-test.wasm-0f1ebc22:wasm-function[41575]:0x278721b)
    at arrow-dataset-file-json-test.wasm.simdjson::fallback::ondemand::document_stream::start() (wasm://wasm/arrow-dataset-file-json-test.wasm-0f1ebc22:wasm-function[18910]:0xf7aedf)
    at arrow-dataset-file-json-test.wasm.arrow::Status arrow::Status arrow::json::HandlerBase::DoParse<arrow::json::Handler<(arrow::json::UnexpectedFieldBehavior)2>>(arrow::json::Handler<(arrow::json::UnexpectedFieldBehavior)2>&, std::__2::shared_ptr<arrow::Buffer> const&)::'lambda'(arrow::json::Handler<(arrow::json::UnexpectedFieldBehavior)2> const&)::operator()<simdjson::padded_string>(arrow::json::Handler<(arrow::json::UnexpectedFieldBehavior)2> const&) const (wasm://wasm/arrow-dataset-file-json-test.wasm-0f1ebc22:wasm-function[20153]:0x1029453)
    at invoke_viii (/build/cpp/debug/arrow-dataset-file-json-test.js:5885:29)
    at arrow-dataset-file-json-test.wasm.arrow::Status arrow::json::HandlerBase::DoParse<arrow::json::Handler<(arrow::json::UnexpectedFieldBehavior)2>>(arrow::json::Handler<(arrow::json::UnexpectedFieldBehavior)2>&, std::__2::shared_ptr<arrow::Buffer> const&) (wasm://wasm/arrow-dataset-file-json-test.wasm-0f1ebc22:wasm-function[20151]:0x102796a)
    at arrow-dataset-file-json-test.wasm.arrow::json::Handler<(arrow::json::UnexpectedFieldBehavior)2>::Parse(std::__2::shared_ptr<arrow::Buffer> const&) (wasm://wasm/arrow-dataset-file-json-test.wasm-0f1ebc22:wasm-function[20150]:0x10275b7)

Node.js v24.19.0

I am not sure what else could be causing this specifically in the Emscripten configuration. Any ideas on what I should look at next?

github-actions Bot added Component: C++ awaiting review Awaiting review labels Aug 29, 2026

Reranko05 added the CI: Extra: C++ Run extra C++ CI label Aug 29, 2026

Reranko05 force-pushed the gh-35460-parser branch 3 times, most recently from d02c90a to d9c6d74 Compare August 30, 2026 05:57

github-actions Bot added the Component: Python label Aug 30, 2026

Reranko05 marked this pull request as ready for review August 30, 2026 06:34

Reranko05 requested review from AlenkaF and raulcd as code owners August 30, 2026 06:34

Copilot AI lite review requested due to automatic review settings August 30, 2026 06:34

Reranko05 requested review from pitrou and rok as code owners August 30, 2026 06:34

Copilot AI reviewed Aug 30, 2026

Reranko05 marked this pull request as draft September 4, 2026 11:31

Reranko05 force-pushed the gh-35460-parser branch from a4dea4b to 48626ae Compare September 8, 2026 12:13

Reranko05 marked this pull request as ready for review September 8, 2026 14:41

Copilot AI review requested due to automatic review settings September 8, 2026 14:41

Copilot AI reviewed Sep 8, 2026

Reranko05 added 2 commits September 16, 2026 13:43

Replace header

95a3b62

replace parser logic

d697915

Reranko05 added 6 commits September 16, 2026 13:43

Lint Fix

475e7cd

Fix CI

8dcd806

Update Python test expectations

b939078

resolve namespace error

d5b27fd

make parser persistent

bc1b9d7

use padded_string_view

3eacd65

Copilot AI review requested due to automatic review settings September 16, 2026 08:14

Reranko05 force-pushed the gh-35460-parser branch from 713496d to 3eacd65 Compare September 16, 2026 08:14

Copilot AI reviewed Sep 16, 2026

Copilot AI review requested due to automatic review settings September 16, 2026 09:03

Copilot AI reviewed Sep 16, 2026

fix emscripten

b51b310

Reranko05 force-pushed the gh-35460-parser branch from 277fa60 to b51b310 Compare September 16, 2026 09:42

Copilot AI review requested due to automatic review settings September 16, 2026 09:42

Copilot AI reviewed Sep 16, 2026

Uh oh!

Conversation

Reranko05 commented Aug 29, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

Changes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Reranko05 commented Sep 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reranko05 commented Sep 7, 2026

Uh oh!

github-actions Bot commented Sep 7, 2026

Uh oh!

pitrou commented Sep 7, 2026

Uh oh!

Reranko05 commented Sep 8, 2026

Uh oh!

github-actions Bot commented Sep 8, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Reranko05 commented Sep 8, 2026

Uh oh!

pitrou commented Sep 8, 2026

Uh oh!

Reranko05 commented Sep 8, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Reranko05 commented Sep 16, 2026

Uh oh!

github-actions Bot commented Sep 16, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Reranko05 commented Sep 16, 2026

Uh oh!

github-actions Bot commented Sep 16, 2026

Uh oh!

Reranko05 commented Sep 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Reranko05 commented Aug 29, 2026 •

edited by github-actions Bot

Loading

Reranko05 commented Sep 7, 2026 •

edited

Loading