Skip to content

GH-38149: [C++][Parquet] Avoid rebuilding column paths during schema initialization - #51400

Draft
namtran1812 wants to merge 1 commit into
apache:mainfrom
namtran1812:optimize-parquet-projected-metadata
Draft

namtran1812 wants to merge 1 commit into
apache:mainfrom
namtran1812:optimize-parquet-projected-metadata

Conversation

@namtran1812

@namtran1812 namtran1812 commented Sep 20, 2026

Copy link
Copy Markdown

Rationale for this change

SchemaDescriptor::BuildTree currently constructs each leaf's dot-separated path through node->path()->ToDotString(). For wide schemas, this repeatedly walks parent links and creates temporary path and string objects for every leaf during schema initialization.

This change reduces that overhead. It is relevant to the wide-schema metadata costs discussed in #38149, although it does not eliminate full-schema metadata deserialization or fully resolve selective column reads.

What changes are included in this PR?

  • Build the dot-separated column path incrementally while traversing the schema tree instead of reconstructing it independently for every leaf.
  • Insert the constructed path directly into leaf_to_idx_.
  • Preserve existing nested-path and duplicate-path lookup semantics.
  • Add a regression test verifying that duplicate paths are still disambiguated by node identity.
  • Extend ReadFileMetadata benchmarks with 5K, 10K, 20K, and 50K-column single-row-group cases.

Local benchmark results (median wall time):

Columns Before After Improvement
1,000 0.907 ms 0.771 ms 15.1%
5,000 4.619 ms 3.936 ms 14.8%
10,000 9.369 ms 8.017 ms 14.4%
20,000 18.951 ms 16.213 ms 14.4%
50,000 47.894 ms 42.542 ms 11.2%

The 50K-column case showed higher variance than the smaller cases. Across these measurements, metadata-read time improved by approximately 11–15%.

Are these changes tested?

Yes.

  • parquet-schema-test: 44/44 tests passed.
  • Added TestSchemaDescriptor.ColumnIndexDuplicatePath to cover duplicate-path identity lookup.
  • git diff --check passes.
  • The updated metadata benchmark was built and run locally across 1K–50K-column schemas.

Are there any user-facing changes?

No. This is an internal Parquet schema initialization optimization with no intended public API or behavioral changes.

Was AI used for this PR?

AI was used to assist with implementation analysis, review, testing strategy, and drafting the PR description. The changes and benchmark/test results were reviewed and validated locally before submission.

PR code and description written by:

  • Human
  • AI

Reviewed before submission by:

  • Human
  • AI
  • Not reviewed

@github-actions

Copy link
Copy Markdown

Thanks for opening a pull request!

This pull request has been automatically converted to a draft because its title doesn't match Arrow's required format.

If this is not a minor PR, could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

After updating the title, you can mark the pull request as ready for review.

See also:

@github-actions
github-actions Bot marked this pull request as draft September 20, 2026 18:27
@namtran1812 namtran1812 changed the title [C++][Parquet] Avoid rebuilding column paths during schema initializa… GH-38149: [C++][Parquet] Avoid rebuilding column paths during schema initialization Sep 20, 2026
@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #38149 has been automatically assigned in GitHub to PR creator.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant