[Parquet] Add writer option to skip bloom filters for column chunks whose data pages are all dictionary encoded by ranflarion · Pull Request #10963 · apache/arrow-rs

ranflarion · 2026-09-02T20:42:56Z

Which issue does this PR close?

Closes [Parquet] Option to skip bloom filters for column chunks whose data pages are all dictionary encoded #10962.

Rationale for this change

A column chunk whose data pages are all dictionary encoded carries its exact set of distinct values in the dictionary page, so a bloom filter for it adds nothing a reader cannot already get exactly, while every value is still hashed into the filter during the write and the filter is serialized after the chunk. parquet-java stopped writing these in PARQUET-2251 (apache/parquet-java#1033, 1.13.0), so files from Spark, Hive and Iceberg never have a bloom filter on a dictionary-only chunk, and there was no way to get the same output from this crate. Details in #10962.

What changes are included in this PR?

WriterProperties::bloom_filter_for_dictionary_encoded_chunks with WriterPropertiesBuilder::set_bloom_filter_for_dictionary_encoded_chunks and DEFAULT_BLOOM_FILTER_FOR_DICTIONARY_ENCODED_CHUNKS = true, so the default output is unchanged.
In GenericColumnWriter::close, when the option is false, the bloom filter is dropped unless encoding_stats records at least one DATA_PAGE/DATA_PAGE_V2 whose encoding is not PLAIN_DICTIONARY or RLE_DICTIONARY, the same test ParquetFileWriter.writeColumnChunk applies in parquet-java. flush_bloom_filter is still called so the encoder state is reset as before.

Are these changes tested?

Yes, test_bloom_filter_for_dictionary_encoded_chunks writes a small dictionary-friendly Int32 column across the dictionary on/off × option on/off matrix and asserts a filter is present in every case except dictionary on with the option off.

Are there any user-facing changes?

One new writer property, opt-in, documented on the setter. No breaking changes.

…a pages are all dictionary encoded

etseidl

This seems reasonable, assuming one has access to the dictionary (which we may be adding here soon #10420).

It would be nice if we could also skip populating the bloom filter while a dictionary is in use. On fallback to non-dictionary, we could then add all the dict keys to the filter.

etseidl · 2026-09-02T23:07:21Z

+    /// The dictionary page of such a chunk already lists every distinct value, so parquet-java
+    /// skips the bloom filter for it; set this to `false` to write files the same way.


I don't think we need the comparison to parquet-java here. I'd instead say that since the dictionary contains all distinct values, the bloom filter is redundant and users might wish to save the space by setting this to false.

alamb · 2026-09-08T20:53:39Z

This one looks like it has a merge conflict which I will fix. Otherwise is it ready to merge?

…ry-encoded-chunks # Conflicts: # parquet/src/column/writer/mod.rs

etseidl · 2026-09-08T20:56:06Z

This one looks like it has a merge conflict which I will fix. Otherwise is it ready to merge?

Yes, I believe it is. Thanks!

alamb · 2026-09-09T14:38:13Z

Thank you @ranflarion and @etseidl

@etseidl

…s dictionary encoded (#10966) # Which issue does this PR close? - Closes #10965. - Related to #10963, which skips the filter for chunks whose data pages are all dictionary encoded; the two are independent and merge in either order. # Rationale for this change While a column is dictionary encoded every row was still hashed into the bloom filter, although the interner already holds the distinct values. For a low-cardinality column that is N inserts where D carry the same information, and those are exactly the columns that stay dictionary encoded. Suggested by @etseidl in #10963. # What changes are included in this PR? - `ColumnValueEncoderImpl::write_slice` and the byte array `encode` insert into the filter only when no dictionary encoder is active. - `flush_dict_page` on both encoders inserts every interned value before handing the dictionary page over. It runs on fallback and at chunk close, so after a fallback the filter holds the dictionary's values plus every value written plain afterwards. - `DictEncoder::uniques` exposes the interned values for the primitive path. The set of values inserted is unchanged and folding decides from the final fill rate, so the serialized filter is byte-identical; only the write-side cost changes. Benchmark (`cargo bench -p parquet --bench arrow_writer -- '<batch>/bloom_filter'`, Apple M-series, criterion, main vs this branch): | batch | main | this PR | change | |---|---|---|---| | string_dictionary_low_cardinality_100 | 19.20 ms | 14.01 ms | -27.0% | | primitive_non_null | 58.90 ms | 57.98 ms | -1.6% | | string_dictionary | 48.85 ms | 49.10 ms | +0.5% (p = 0.24) | | string_non_null | 108.63 ms | 108.96 ms | +0.3% (p = 0.05) | # Are these changes tested? Yes. New round-trip tests for `StringArray` and `Int64Array` cover a chunk that stays dictionary encoded (asserted through the page encoding mask) and a chunk that falls back to plain after a small dictionary page limit, checking the filter for every written value and for absent ones. The existing bloom filter round-trip tests already sweep dictionary disabled, immediate fallback and dictionary enabled and pass unchanged. # Are there any user-facing changes? No API or output change. `DictEncoder::uniques` is new but the type is not exported from the crate.

ranflarion · 2026-09-09T16:15:35Z

Thank you @ranflarion and @etseidl

Thanks @etseidl @alamb

…hose data pages are all dictionary encoded (apache#10963) # Which issue does this PR close? - Closes apache#10962. # Rationale for this change A column chunk whose data pages are all dictionary encoded carries its exact set of distinct values in the dictionary page, so a bloom filter for it adds nothing a reader cannot already get exactly, while every value is still hashed into the filter during the write and the filter is serialized after the chunk. parquet-java stopped writing these in PARQUET-2251 (apache/parquet-java#1033, 1.13.0), so files from Spark, Hive and Iceberg never have a bloom filter on a dictionary-only chunk, and there was no way to get the same output from this crate. Details in apache#10962. # What changes are included in this PR? - `WriterProperties::bloom_filter_for_dictionary_encoded_chunks` with `WriterPropertiesBuilder::set_bloom_filter_for_dictionary_encoded_chunks` and `DEFAULT_BLOOM_FILTER_FOR_DICTIONARY_ENCODED_CHUNKS = true`, so the default output is unchanged. - In `GenericColumnWriter::close`, when the option is `false`, the bloom filter is dropped unless `encoding_stats` records at least one `DATA_PAGE`/`DATA_PAGE_V2` whose encoding is not `PLAIN_DICTIONARY` or `RLE_DICTIONARY`, the same test `ParquetFileWriter.writeColumnChunk` applies in parquet-java. `flush_bloom_filter` is still called so the encoder state is reset as before. # Are these changes tested? Yes, `test_bloom_filter_for_dictionary_encoded_chunks` writes a small dictionary-friendly Int32 column across the dictionary on/off × option on/off matrix and asserts a filter is present in every case except dictionary on with the option off. # Are there any user-facing changes? One new writer property, opt-in, documented on the setter. No breaking changes. --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

@etseidl

…s dictionary encoded (apache#10966) # Which issue does this PR close? - Closes apache#10965. - Related to apache#10963, which skips the filter for chunks whose data pages are all dictionary encoded; the two are independent and merge in either order. # Rationale for this change While a column is dictionary encoded every row was still hashed into the bloom filter, although the interner already holds the distinct values. For a low-cardinality column that is N inserts where D carry the same information, and those are exactly the columns that stay dictionary encoded. Suggested by @etseidl in apache#10963. # What changes are included in this PR? - `ColumnValueEncoderImpl::write_slice` and the byte array `encode` insert into the filter only when no dictionary encoder is active. - `flush_dict_page` on both encoders inserts every interned value before handing the dictionary page over. It runs on fallback and at chunk close, so after a fallback the filter holds the dictionary's values plus every value written plain afterwards. - `DictEncoder::uniques` exposes the interned values for the primitive path. The set of values inserted is unchanged and folding decides from the final fill rate, so the serialized filter is byte-identical; only the write-side cost changes. Benchmark (`cargo bench -p parquet --bench arrow_writer -- '<batch>/bloom_filter'`, Apple M-series, criterion, main vs this branch): | batch | main | this PR | change | |---|---|---|---| | string_dictionary_low_cardinality_100 | 19.20 ms | 14.01 ms | -27.0% | | primitive_non_null | 58.90 ms | 57.98 ms | -1.6% | | string_dictionary | 48.85 ms | 49.10 ms | +0.5% (p = 0.24) | | string_non_null | 108.63 ms | 108.96 ms | +0.3% (p = 0.05) | # Are these changes tested? Yes. New round-trip tests for `StringArray` and `Int64Array` cover a chunk that stays dictionary encoded (asserted through the page encoding mask) and a chunk that falls back to plain after a small dictionary page limit, checking the filter for every written value and for absent ones. The existing bloom filter round-trip tests already sweep dictionary disabled, immediate fallback and dictionary enabled and pass unchanged. # Are there any user-facing changes? No API or output change. `DictEncoder::uniques` is new but the type is not exported from the crate.

…hose data pages are all dictionary encoded (apache#10963) # Which issue does this PR close? - Closes apache#10962. # Rationale for this change A column chunk whose data pages are all dictionary encoded carries its exact set of distinct values in the dictionary page, so a bloom filter for it adds nothing a reader cannot already get exactly, while every value is still hashed into the filter during the write and the filter is serialized after the chunk. parquet-java stopped writing these in PARQUET-2251 (apache/parquet-java#1033, 1.13.0), so files from Spark, Hive and Iceberg never have a bloom filter on a dictionary-only chunk, and there was no way to get the same output from this crate. Details in apache#10962. # What changes are included in this PR? - `WriterProperties::bloom_filter_for_dictionary_encoded_chunks` with `WriterPropertiesBuilder::set_bloom_filter_for_dictionary_encoded_chunks` and `DEFAULT_BLOOM_FILTER_FOR_DICTIONARY_ENCODED_CHUNKS = true`, so the default output is unchanged. - In `GenericColumnWriter::close`, when the option is `false`, the bloom filter is dropped unless `encoding_stats` records at least one `DATA_PAGE`/`DATA_PAGE_V2` whose encoding is not `PLAIN_DICTIONARY` or `RLE_DICTIONARY`, the same test `ParquetFileWriter.writeColumnChunk` applies in parquet-java. `flush_bloom_filter` is still called so the encoder state is reset as before. # Are these changes tested? Yes, `test_bloom_filter_for_dictionary_encoded_chunks` writes a small dictionary-friendly Int32 column across the dictionary on/off × option on/off matrix and asserts a filter is present in every case except dictionary on with the option off. # Are there any user-facing changes? One new writer property, opt-in, documented on the setter. No breaking changes. --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

@etseidl

…s dictionary encoded (apache#10966) # Which issue does this PR close? - Closes apache#10965. - Related to apache#10963, which skips the filter for chunks whose data pages are all dictionary encoded; the two are independent and merge in either order. # Rationale for this change While a column is dictionary encoded every row was still hashed into the bloom filter, although the interner already holds the distinct values. For a low-cardinality column that is N inserts where D carry the same information, and those are exactly the columns that stay dictionary encoded. Suggested by @etseidl in apache#10963. # What changes are included in this PR? - `ColumnValueEncoderImpl::write_slice` and the byte array `encode` insert into the filter only when no dictionary encoder is active. - `flush_dict_page` on both encoders inserts every interned value before handing the dictionary page over. It runs on fallback and at chunk close, so after a fallback the filter holds the dictionary's values plus every value written plain afterwards. - `DictEncoder::uniques` exposes the interned values for the primitive path. The set of values inserted is unchanged and folding decides from the final fill rate, so the serialized filter is byte-identical; only the write-side cost changes. Benchmark (`cargo bench -p parquet --bench arrow_writer -- '<batch>/bloom_filter'`, Apple M-series, criterion, main vs this branch): | batch | main | this PR | change | |---|---|---|---| | string_dictionary_low_cardinality_100 | 19.20 ms | 14.01 ms | -27.0% | | primitive_non_null | 58.90 ms | 57.98 ms | -1.6% | | string_dictionary | 48.85 ms | 49.10 ms | +0.5% (p = 0.24) | | string_non_null | 108.63 ms | 108.96 ms | +0.3% (p = 0.05) | # Are these changes tested? Yes. New round-trip tests for `StringArray` and `Int64Array` cover a chunk that stays dictionary encoded (asserted through the page encoding mask) and a chunk that falls back to plain after a small dictionary page limit, checking the filter for every written value and for absent ones. The existing bloom filter round-trip tests already sweep dictionary disabled, immediate fallback and dictionary enabled and pass unchanged. # Are there any user-facing changes? No API or output change. `DictEncoder::uniques` is new but the type is not exported from the crate.

Add a writer option to skip bloom filters for column chunks whose dat…

332a1dc

…a pages are all dictionary encoded

github-actions Bot added the parquet Changes to the parquet crate label Sep 2, 2026

etseidl approved these changes Sep 2, 2026

View reviewed changes

Describe the option without the parquet-java comparison

5acb30e

This was referenced Sep 3, 2026

[Parquet] Populate bloom filters from the dictionary instead of per row while a column is dictionary encoded #10965

Closed

[Parquet] Populate bloom filters from the dictionary while a column is dictionary encoded #10966

Merged

Merge remote-tracking branch 'apache/main' into bloom-filter-dictiona…

5c064d9

…ry-encoded-chunks # Conflicts: # parquet/src/column/writer/mod.rs

alamb merged commit 5be0557 into apache:main Sep 9, 2026
31 checks passed

etseidl mentioned this pull request Sep 25, 2026

feat(parquet): provide direct access to dictionary pages #10420

Merged

Navigation Menu

[Parquet] Add writer option to skip bloom filters for column chunks whose data pages are all dictionary encoded - #10963

[Parquet] Add writer option to skip bloom filters for column chunks whose data pages are all dictionary encoded#10963
alamb merged 3 commits into
apache:mainfrom
ranflarion:bloom-filter-dictionary-encoded-chunks

ranflarion commented Sep 2, 2026

Uh oh!

etseidl left a comment

Uh oh!

etseidl Sep 2, 2026

Uh oh!

alamb commented Sep 8, 2026

Uh oh!

etseidl commented Sep 8, 2026

Uh oh!

Uh oh!

alamb commented Sep 9, 2026

Uh oh!

ranflarion commented Sep 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		/// The dictionary page of such a chunk already lists every distinct value, so parquet-java
		/// skips the bloom filter for it; set this to `false` to write files the same way.

Conversation

ranflarion commented Sep 2, 2026

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

etseidl left a comment

Choose a reason for hiding this comment

Uh oh!

etseidl Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

alamb commented Sep 8, 2026

Uh oh!

etseidl commented Sep 8, 2026

Uh oh!

Uh oh!

alamb commented Sep 9, 2026

Uh oh!

ranflarion commented Sep 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants