Skip to content

fix(parquet): bound schema nesting depth; prevent negative array offsets (FB-3438) - #44

Merged
lorenzhs merged 2 commits into
release-24.0.0from
lorenz/bound-schema-depth
Aug 26, 2026
Merged

lorenzhs merged 2 commits into
release-24.0.0from
lorenz/bound-schema-depth

Conversation

@lorenzhs

@lorenzhs lorenzhs commented Aug 26, 2026

Copy link
Copy Markdown
Collaborator

Two independent stack/bounds gaps on untrusted-input paths, both found while auditing Firebolt's Arrow column converter.

schema::Unflatten: unbounded recursion

A Parquet schema is a flat list of SchemaElements whose num_children fields describe the tree, so a file can nest as deep as it has elements. Unflatten rebuilds that tree with a recursive lambda and no depth guard, one native frame per level. A schema a few tens of thousands deep overflows the thread stack while the metadata is parsed, before any consumer sees a row — an unauthenticated remote crash on any read of an attacker-supplied file.

Threads a depth counter through NextNode and rejects past kMaxSchemaNestingDepth (1000), matching the JSON reader's kMaxNestingDepth. Far above anything a real writer emits; Parquet's 3-level LIST encoding spends 2 elements per array level.

Repro: a metadata-only file (no row groups) nesting 100000 groups. Pre-patch, SIGSEGV inside Unflatten. Post-patch, Parquet schema nesting depth exceeds the maximum of 1000.

ValidateLayout: negative array offset

ValidateLayout rejects a negative length but not a negative offset, and the two are not equivalent. Every buffer-size check in the function sizes buffers by length_plus_offset; a negative offset shrinks that, so a buffer far too small for the array satisfies all of them. ArrayData::GetValues() then returns buffers[i]->data() + offset — a pointer before the buffer — to every consumer that trusted Validate().

Nothing in the format permits it, but an ArrowArray crossing the C data interface carries whatever its producer wrote: ArrayImporter::AllocateArrayData copies c_struct_->offset in unchecked, and the importer's own buffer sizing is wrong in the same direction. Validate() is where an imported array is meant to become trustworthy, so the check belongs there. Two lines beside the existing length check; the recursion into child_data covers nested arrays.

A Parquet schema is a flat list of SchemaElements whose num_children fields
describe the tree, so a file can nest as deep as it has elements.
schema::Unflatten rebuilds that tree with a recursive lambda and no depth
guard, running one native frame per level. A schema a few tens of thousands
deep overflows the thread stack while the metadata is being parsed, before any
consumer sees a single row: an unauthenticated remote crash on any read of an
attacker-supplied Parquet file.

Reject schemas nesting deeper than kMaxSchemaNestingDepth (1000) by threading a
depth counter through NextNode. 1000 matches the JSON reader's kMaxNestingDepth
so every untrusted-nesting path in the library fails at the same number, and it
is far above any schema a real writer produces (Parquet's own 3-level LIST
encoding spends 2 elements per array level).

The file that motivated this is a metadata-only Parquet file (no row groups)
whose schema nests 100000 groups: pre-patch it SIGSEGVs inside Unflatten;
post-patch it raises "Parquet schema nesting depth exceeds the maximum of 1000".

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012wER7pGuTzevy6j4DS7hcL
@lorenzhs
lorenzhs requested a review from a team August 26, 2026 15:53
ValidateLayout checks a negative length but not a negative offset, and the two
are not equivalent. Every buffer-size check in the function sizes buffers by
length_plus_offset; a negative offset shrinks that, so a buffer far too small
for the array satisfies all of them. ArrayData::GetValues() then hands out
buffers[i]->data() + offset, a pointer before the start of the buffer, and
every consumer that trusts Validate() reads out of bounds.

Nothing in the columnar format permits a negative offset, but an ArrowArray
arriving over the C data interface carries whatever its producer wrote:
ArrayImporter::AllocateArrayData copies c_struct_->offset into ArrayData
unchecked, and the importer's own buffer-size arithmetic is wrong in the same
direction. Validate() is the boundary where an imported array is supposed to
become trustworthy, so the check belongs here.

Two lines next to the existing length check; recursion into child_data means
nested arrays are covered too.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012wER7pGuTzevy6j4DS7hcL
@lorenzhs lorenzhs changed the title fix(parquet): bound schema nesting depth to prevent stack overflow (FB-3438) fix(parquet): bound schema nesting depth; prevent negative array offsets (FB-3438) Aug 26, 2026
@lorenzhs
lorenzhs merged commit 7bee733 into release-24.0.0 Aug 26, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants