Skip to content

fix(arrow): bound array recursion depth (FB-3438) - #45

Merged
lorenzhs merged 4 commits into
release-24.0.0from
lorenz/recursion-depth-limits
Aug 27, 2026
Merged

lorenzhs merged 4 commits into
release-24.0.0from
lorenz/recursion-depth-limits

Conversation

@lorenzhs

@lorenzhs lorenzhs commented Aug 27, 2026

Copy link
Copy Markdown
Collaborator

Background

Firebolt validates Arrow columns before converting them into engine columns. Arrow array validation recursively walks nested child data without enforcing a nesting limit, so sufficiently deep attacker-controlled input could exhaust the native stack.

The existing JSON and Parquet guards used 1000 levels, but measurements found the shallowest stack overflow at roughly 1650 frames on a 2 MiB stack. That margin is too small for recursive walks whose frame sizes differ. We change the limit to 256 here, matching Firebolt's native type nesting limit.

Summary

  • Bound Array::Validate() and Array::ValidateFull() recursion at 256 levels.
  • Count list, struct, union, dictionary, run-end encoded, and extension-storage descent through the existing validation recursion.
  • Lower the fork's JSON and Parquet schema nesting limits from 1000 to 256 so all untrusted recursive input paths use the same conservative ceiling.

256 is far beyond realistic schemas while preserving about an order of magnitude of headroom below the shallowest measured overflow.

This closes the validation-before-depth-check gap found while hardening Firebolt's Arrow consumers.

Testing

Testing is in the PackDB companion PR.

@lorenzhs
lorenzhs requested a review from a team August 27, 2026 12:20
@lorenzhs
lorenzhs marked this pull request as ready for review August 27, 2026 12:21
@lorenzhs lorenzhs changed the title fix(arrow): bound array and IPC schema recursion depth (FB-3438) fix(arrow): bound array recursion depth (FB-3438) Aug 27, 2026
@lorenzhs
lorenzhs merged commit cc35e3d into release-24.0.0 Aug 27, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants