feat(parquet): provide direct access to dictionary pages by DarkWanderer · Pull Request #10420 · apache/arrow-rs

DarkWanderer · 2026-07-23T19:24:14Z

Which issue does this PR close?

Closes Enable access to column dictionaries in async reader #9010.
Supersedes Added capability to fetch dictionary values #9011 (closed as stale)

Rationale for this change

This change provides low-level API necessary for enabling dictionary-based pruning in DataFusion: apache/datafusion#23851

What changes are included in this PR?

parquet::file::metadata::dictionary::decode_dictionary_page: decodes
a BYTE_ARRAY dictionary page (Thrift header parse, decompress, PLAIN
decode) into a Utf8/Binary Arrow array.
ParquetMetaDataReader::read_column_dictionary (sync) and
read_column_dictionary_async (async) to fetch and decode a given row group/column's dictionary page from a ParquetMetaData, returning Ok(None) if the chunk has no dictionary page.
ParquetRecordBatchStreamBuilder::get_row_group_column_dictionary convenience method mirroring get_row_group_column_bloom_filter.

Are these changes tested?

Yes: a round-trip unit test for a dictionary-encoded string column, a
non-BYTE_ARRAY rejection test, and sync + async reader tests that
decode a real dictionary page written through ArrowWriter.

Are there any user-facing changes?

Yes, three new public APIs (see above). No changes to
existing API.

alamb · 2026-08-15T10:31:41Z

Thanks @DarkWanderer -- I think it would help to have some sort of example here that shows the "usecase" of these APIs -- so either a test or something in parquet-examples that shows how these new APIs are meant to be used

i see that apache/datafusion#23851 is sort of an example but I think we should distill the use down to something just arrow-rs specific so the API is clear

Does that makes sense?

DarkWanderer · 2026-08-15T16:46:02Z

Thank you for review @alamb

Yes, this makes sense. I have added a test in async_reader module - does this match what you had in mind? Alternatively, happy to add a separate example parallel to parquet/examples/read_with_rowgroup.rs.

Add sync and async APIs for decoding a BYTE_ARRAY column chunk dictionary page without materializing the full column, enabling exact row-group membership pruning for fully dictionary-encoded chunks. Reuse the regular page reader header, size validation, and decryption path so standalone dictionary decoding handles encrypted, truncated, and malformed pages safely. See apache#9010. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y1dzYmhkPzEC6H48UVneAJ

Co-authored-by: Ed Seidl <etseidl@users.noreply.github.com>

DarkWanderer · 2026-09-02T05:54:44Z

Thank you for spotting @etseidl , applied the suggestion

etseidl · 2026-09-02T23:25:54Z

Thanks @DarkWanderer, and sorry this is taking so long. I hope to find time to review this Friday.

# Conflicts: # parquet/src/arrow/async_reader/mod.rs # parquet/src/arrow/mod.rs

DarkWanderer · 2026-09-17T20:33:38Z

@etseidl let me know if I can provide additional info to help review

etseidl

Thanks @DarkWanderer, sorry for the delay reviewing this. I think this looks good, mod a few last nits. It would be nice to eventually extend this to all data types (or at least physical types), but I suspect strings are the major use case for this.

Wouldn't mind a second opinion from @alamb or @Jefffrey

Jefffrey · 2026-09-23T03:20:07Z

Thanks @DarkWanderer, sorry for the delay reviewing this. I think this looks good, mod a few last nits. It would be nice to eventually extend this to all data types (or at least physical types), but I suspect strings are the major use case for this.

Wouldn't mind a second opinion from @alamb or @Jefffrey

ill just chip in and say i probably wont have time to review this 😅

so long as it looks good to you and the new public APIs are reasonable 👍

DarkWanderer · 2026-09-23T12:09:55Z

I do not have access to press "merge" here, in case this is implicitly expected. Let me know if I can clarify anything else or make further tweaks

etseidl

Sorry for further delay @DarkWanderer, and thank you for your patience! Because I'm less familiar with the arrow bits and there isn't enough reviewer bandwidth, I had Codex look for some issues. Here's my translation of what Codex found, which I think are reasonable nits. There were other nits around correctness of the metadata, but that applies all over this crate so those can be deferred. I think if we address these, then I'll feel more confident merging this.

etseidl · 2026-09-25T16:57:51Z

+    parquet_meta_data: &ParquetMetaData,
+    row_group_idx: usize,
+    column_idx: usize,
+) -> Result<ArrayRef> {


This function will conditionally return binary or string arrays depending on the parquet logical type. If we want to later extend this to other physical types, we're now in the business of casting all of those to the appropriate arrow type, like we do in the array readers (e.g.

arrow-rs/parquet/src/arrow/array_reader/primitive_array.rs

Line 162 in 9e0ab28

fn consume_batch(&mut self) -> Result<ArrayRef> {

). This also doesn't handle decimals with a BYTE_ARRAY physical type.

Maybe for now we can simply return a binary array, and leave it to the user to either cast to an appropriate arrow type, or instead cast probes to the physical type. Perhaps a later effort could add helpers for either approach (potentially reusing the current parquet->arrow conversion logic).

etseidl · 2026-09-25T17:13:17Z

+    let crypto_context =
+        CryptoContext::for_column(file_decryptor, crypto_metadata, row_group_idx, column_idx)?
+            .for_dictionary_page();


This is using the passed in row_group_idx, which is an index into the current parquet_meta_data::row_groups. But the row groups may have been filtered at this point, so the 0th row group may have been the 2nd in the original file. I think it would be better to use parquet_meta_data.row_group(row_group_idx).ordinal() here, and error if there is no ordinal in the metadata (it should be present when modular encryption is used, for exactly this purpose).

Done incl test

etseidl · 2026-09-25T17:26:50Z

+    let is_utf8 = is_utf8(column_descriptor);
+    let mut decoder = ByteArrayDecoderPlain::new(buf, num_values, Some(num_values), is_utf8);
+    let mut offsets = OffsetBuffer::<i32>::with_capacity(num_values);
+    decoder.read(&mut offsets, usize::MAX)?;


Apparently read here can return when the input buffer is exhausted, but it still returns to_read, rather than the actual number of values read. You might want to do something like

Suggested change

decoder.read(&mut offsets, usize::MAX)?;

decoder.read(&mut offsets, usize::MAX)?;

if offsets.len() != num_values {

return Err(general_err!(

"did not read entire dictionary expected {num_values}, got {}",

offsets.len()

));

}

etseidl · 2026-09-25T17:33:14Z

+    if header.r#type != PageType::DICTIONARY_PAGE {
+        return Err(ParquetError::General(format!(
+            "Expected a dictionary page, found {:?}",
+            header.r#type
+        )));
+    }


We should also check header.dictionary_page_header.encoding to ensure it's PLAIN. I think there's been talk of using other encodings (although maybe just replacing RLE) so it would be nice to future-proof this.

Added (below)

etseidl

Thanks @DarkWanderer. I think this is ready now.

etseidl · 2026-09-25T19:01:41Z

+
+    #[cfg(feature = "encryption")]
+    #[test]
+    fn read_column_dictionary_uses_file_ordinal_after_filtering() {


❤️

Validated this fails when row_group_idx is used.

etseidl · 2026-09-25T19:15:06Z

I'll add this makes #10963 more useful now.

etseidl · 2026-09-25T19:19:49Z

Thanks again @DarkWanderer. cc @ranflarion who might find this useful.

alamb · 2026-09-25T20:21:43Z

EPIC

apache/arrow-rs#10420 is merged to arrow-rs main as b9b1d5005; the [patch.crates-io] block (repointed during the upstream/main merge) now tracks that branch instead of the old DarkWanderer/arrow-rs fork. Cargo.lock is re-pinned to the merged commit for every patched arrow-*/parquet crate. Review changed the final API before merging, so adapt to it: - ParquetRecordBatchStreamBuilder::get_row_group_column_dictionary was renamed to get_column_chunk_dictionary. - Dictionary pages now always decode as Binary, never Utf8, so DictionaryStatistics::insert drops its Utf8/StringArray arm. Also: - Fix a tag collision in ParquetOptions' proto message: our dictionary_filter_on_read (38) collided with upstream's newly added max_in_list_size (also 38). Move ours to the next free tag, 39, and regenerate proto-common/proto-models. - Adapt dictionary_filter.rs's test helper to reader.rs's ParquetFileReader, which upstream refactored to build its ParquetObjectReader internally rather than taking one as a field. - Migrate two more PruningPredicate::try_new call sites (deprecated since 55.0.0) to PruningPredicateBuilder, needed for a clean `-D warnings` clippy run. - Fix a FixedSizeListArray::value_offset deprecation (arrow-rs 60.0.0) in unnest.rs by switching to value_offset_at. - Add the row_groups_pruned_dictionary / bytes_processed metrics to the .slt golden files upstream's merge didn't touch (DataSourceExec lines outside the merge's conflict hunks). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

github-actions Bot added the parquet Changes to the parquet crate label Jul 23, 2026

DarkWanderer mentioned this pull request Jul 23, 2026

Prune parquet row groups using fully dictionary-encoded columns apache/datafusion#23851

Draft

DarkWanderer changed the title ~~feat(parquet): decode dictionary pages independent of the array reader~~ feat(parquet): provide direct access to dictionary pages Jul 23, 2026

DarkWanderer force-pushed the get-dictionary branch from 7c65729 to 8145bb8 Compare July 23, 2026 22:19

DarkWanderer marked this pull request as ready for review July 24, 2026 08:22

DarkWanderer force-pushed the get-dictionary branch from 09c54af to 4371070 Compare August 30, 2026 10:18

DarkWanderer and others added 3 commits August 30, 2026 12:24

test(parquet): demonstrate dictionary row group pruning

d1f6bb2

fix(parquet): nest or-patterns in is_utf8 to satisfy clippy

90eb024

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y1dzYmhkPzEC6H48UVneAJ

DarkWanderer force-pushed the get-dictionary branch from 4371070 to 90eb024 Compare August 30, 2026 10:25

Merge branch 'main' into get-dictionary

6f5094d

etseidl reviewed Aug 31, 2026

View reviewed changes

Comment thread parquet/src/file/metadata/mod.rs Outdated

Apply suggestion from @etseidl

b00bc69

Co-authored-by: Ed Seidl <etseidl@users.noreply.github.com>

etseidl mentioned this pull request Sep 2, 2026

[Parquet] Add writer option to skip bloom filters for column chunks whose data pages are all dictionary encoded #10963

Merged

DarkWanderer added 2 commits September 9, 2026 18:03

Merge remote-tracking branch 'upstream/main' into get-dictionary

22d060b

# Conflicts: # parquet/src/arrow/async_reader/mod.rs # parquet/src/arrow/mod.rs

Merge remote-tracking branch 'origin/get-dictionary' into get-dictionary

17914d1

Merge branch 'main' into get-dictionary

fe64de5

etseidl approved these changes Sep 17, 2026

View reviewed changes

Comment thread parquet/src/arrow/async_reader/mod.rs Outdated

Comment thread parquet/src/file/metadata/reader.rs Outdated

Comment thread parquet/src/arrow/async_reader/mod.rs

Address review notes

f3377f3

etseidl requested changes Sep 25, 2026

View reviewed changes

etseidl mentioned this pull request Sep 25, 2026

ByteArrayDecoderPlain::read in the parquet crate can silently fail #11221

Open

Fix dictionary page decoding and encrypted row group lookup

fecbc60

etseidl approved these changes Sep 25, 2026

View reviewed changes

etseidl merged commit b9b1d50 into apache:main Sep 25, 2026
29 checks passed

DarkWanderer deleted the get-dictionary branch September 25, 2026 19:24

Jefffrey added the enhancement Any new improvement worthy of a entry in the changelog label Sep 25, 2026

-    decoder.read(&mut offsets, usize::MAX)?;
+    decoder.read(&mut offsets, usize::MAX)?;
+    if offsets.len() != num_values {
+        return Err(general_err!(
+            "did not read entire dictionary expected {num_values}, got {}",
+            offsets.len()
+        ));
+    }

Conversation

DarkWanderer commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

alamb commented Aug 15, 2026

Uh oh!

DarkWanderer commented Aug 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

DarkWanderer commented Sep 2, 2026

Uh oh!

etseidl commented Sep 2, 2026

Uh oh!

DarkWanderer commented Sep 17, 2026

Uh oh!

etseidl left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Jefffrey commented Sep 23, 2026

Uh oh!

DarkWanderer commented Sep 23, 2026

Uh oh!

etseidl left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

etseidl Sep 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

etseidl left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

etseidl commented Sep 25, 2026

Uh oh!

Uh oh!

etseidl commented Sep 25, 2026

Uh oh!

alamb commented Sep 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

DarkWanderer commented Jul 23, 2026 •

edited

Loading

DarkWanderer commented Aug 15, 2026 •

edited

Loading

etseidl Sep 25, 2026 •

edited

Loading