GH-50689: [C++][Parquet] Add IEEE-754 total order and nan count for floating types by HuaHuaY · Pull Request #50807 · apache/arrow

HuaHuaY · 2026-08-04T15:11:28Z

Rationale for this change

Implement IEEE 754 total order and NaN counts from apache/parquet-format#514.

What changes are included in this PR?

Use IEEE total order by default, with a writer-level TYPE order fallback.
Add nan_count to statistics and nan_counts to PageIndex.
Preserve NaN payloads and signed zero in statistics and dictionary encoding.
Make Dataset pruning NaN-aware (skip FLOAT16 numeric bounds until comparison kernels are available).

Are these changes tested?

Yes.

Are there any user-facing changes?

cpp/src/parquet/types.h: Adds ColumnOrder::IEEE_754_TOTAL_ORDER.
cpp/src/parquet/properties.h: Adds the floating-point column-order writer property.
cpp/src/parquet/schema.h: Allows column descriptors to use IEEE-ordered min/max statistics.
cpp/src/parquet/page_index.h: Exposes virtual std::optional<std::span<const int64_t>> nan_counts() const.
cpp/src/parquet/statistics.h: Adds NaN fields and presence APIs to EncodedStatistics and Statistics, and extends encoded-state Statistics::Make / MakeStatistics overloads with nan_count and has_nan_count.

GitHub Issue: [C++][Parquet] Add IEEE-754 total order and nan count for floating types #50689

HuaHuaY · 2026-08-05T07:14:52Z

                         ARROW_SUPPRESS_DEPRECATION_WARNING= \
                         ARROW_UNSUPPRESS_DEPRECATION_WARNING= \
                         GANDIVA_EXPORT= \
+                         PARQUET_DEPRECATED(x)= \


Add this because I added a PARQUET_DEPRECATED at cpp/src/parquet/statistics.h and ci failed. https://github.com/apache/arrow/actions/runs/30975844655/job/92209544482

I believe this is a long-standing issue. If someone would like me to submit a separate PR to fix it, I can certainly do so.

Copilot

Pull request overview

This PR updates Arrow C++’s Parquet implementation to support IEEE-754 total ordering for floating-point statistics and to record NaN counts in statistics and page indexes, aligning with parquet-format changes (apache/parquet-format#514) and improving correctness for pruning, encoding, and metadata round-trips involving NaNs and signed zeros.

Changes:

Add ColumnOrder::IEEE_754_TOTAL_ORDER and a writer property to control floating-point column ordering (defaulting to IEEE total order).
Track and serialize nan_count in column/page statistics and expose per-page nan_counts via ColumnIndex.
Update statistics computation, dictionary encoding, metadata/schema handling, and dataset pruning to be NaN-aware (including special handling for FLOAT16 pruning limitations).

Reviewed changes

Copilot reviewed 28 out of 28 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
docs/source/python/parquet.rst	Update example metadata output values to match new behavior/encoding sizes.
cpp/src/parquet/types.h	Add `IEEE_754_TOTAL_ORDER` to `ColumnOrder` enum and declare static instance.
cpp/src/parquet/types.cc	Define `ColumnOrder::ieee_754_total_order_`.
cpp/src/parquet/thrift_internal.h	Serialize/deserialize `nan_count`; treat IEEE order as min/max-value stats field source.
cpp/src/parquet/statistics.h	Extend statistics APIs and encoded state to include optional `nan_count`; update comparator docs.
cpp/src/parquet/statistics.cc	Implement IEEE total-order comparisons, NaN counting, and IEEE-aware min/max handling and merging.
cpp/src/parquet/statistics_test.cc	Add/extend tests for NaN counts, IEEE total order, and float16 behaviors.
cpp/src/parquet/schema.h	Allow `can_use_min_max()` for IEEE order (see review note re: non-float types).
cpp/src/parquet/schema.cc	Add `IsFloatingPointType` helper for schema/metadata decisions.
cpp/src/parquet/schema_test.cc	Test that IEEE column order allows min/max usage.
cpp/src/parquet/schema_internal.h	Expose `IsFloatingPointType` as a non-public schema utility.
cpp/src/parquet/properties.h	Add writer property `floating_point_column_order` with validation and plumbing.
cpp/src/parquet/page_index.h	Add `has_nan_counts()` / `nan_counts()` to `ColumnIndex` API.
cpp/src/parquet/page_index.cc	Persist per-page `nan_counts` when available for floating columns; validate vector lengths.
cpp/src/parquet/page_index_test.cc	Add tests covering nan_counts propagation for IEEE total-order float columns.
cpp/src/parquet/metadata.cc	Plumb encoded stats into `Statistics::Make`; enforce column-order compatibility; parse/write IEEE column order in file metadata.
cpp/src/parquet/file_writer.cc	Build writer schema with per-float column orders from writer properties and stabilize type_length.
cpp/src/parquet/file_serialize_test.cc	Add round-trip test validating floating-point column order behavior and schema stability.
cpp/src/parquet/encoding_test.cc	Add tests ensuring dictionary encoding preserves float bit patterns and avoids NaN hash collisions.
cpp/src/parquet/encoder.cc	Change float/double dictionary memoization keys to use bitwise representations for NaN payload stability.
cpp/src/parquet/column_writer.cc	Gate legacy min/max field population based on effective ordering; enable stats collection via `can_use_min_max()`.
cpp/src/parquet/arrow/reader_internal.cc	Decode FLOAT16 min/max statistics into HalfFloat scalars for Arrow conversion.
cpp/src/parquet/arrow/index_test.cc	Add nan_counts to column index round-trip expectations; add test parquet file coverage for mixed orders/nan counts.
cpp/src/parquet/arrow/arrow_reader_writer_test.cc	Add end-to-end test ensuring float dictionary round-trips preserve NaN payloads and signed zeros with IEEE order and page index.
cpp/src/arrow/dataset/file_parquet.cc	Make dataset pruning NaN-aware using `nan_count`; skip FLOAT16 numeric pruning until kernels exist.
cpp/src/arrow/dataset/file_parquet_test.cc	Add tests validating pruning expressions with/without nan_count and for FLOAT16 behavior.
cpp/apidoc/Doxyfile	Teach Doxygen about `PARQUET_DEPRECATED` macro for API docs.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

HuaHuaY · 2026-08-27T02:26:50Z

Run git rebase to resolve conflicts

Copilot

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated no new comments.

Copilot

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated no new comments.

Copilot

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated 1 comment.

Copilot

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated no new comments.

Suppressed comments (1)

cpp/src/parquet/schema.cc:635

schema::IsFloatingPointType dereferences descr.logical_type() without a null check. ColumnDescriptor::logical_type() can be empty (e.g. FLBA columns created via PrimitiveNode::Make with only physical/converted type), which would crash when this helper is called (e.g. from page index / metadata code).

bool IsFloatingPointType(const ColumnDescriptor& descr) {
  return descr.physical_type() == Type::FLOAT || descr.physical_type() == Type::DOUBLE ||
         descr.logical_type()->type() == LogicalType::Type::FLOAT16;

wgtmac

Thanks for addressing my comments! Looks good to me now. I still have two comments with open questions.

wgtmac · 2026-09-01T16:14:22Z

+  /// \param[in] is_min_value_exact whether the min value is exact
+  /// \param[in] is_max_value_exact whether the max value is exact
+  /// \param[in] pool a memory pool to use for any memory allocations, optional
+  static std::shared_ptr<Statistics> Make(


I think that's a different concern. Here I mean we don't have to add yet another overload for adding an optional nan_count because we can add it to the end of an existing one with a default value. We don't need a breaking change this time. The issue you've mentioned is to replace has_xxx and xxx with a single std::optional<int64_t> which has to be a breaking change.

wgtmac · 2026-09-01T16:15:47Z

                                                          Repetition::type rep) {
-  return PrimitiveNode::Make(name, rep, LogicalType::Float16(),
-                             Type::FIXED_LEN_BYTE_ARRAY, 2);
+  auto node = PrimitiveNode::Make(name, rep, LogicalType::Float16(),


nit: should we expand PrimitiveNode::Make to accept column order as well? Currently we cannot create a primitive node of floating type with type-defined-order in a single shot.

wgtmac · 2026-09-01T16:19:51Z

@pitrou Do you want to take a look?

pitrou · 2026-09-01T16:28:51Z

Yes, I'll take a look. Sorry!

pitrou · 2026-09-02T08:43:08Z

                                         *logical_type, min, max);
  }
+  if (logical_type->type() == LogicalType::Type::FLOAT16) {
+    *min = std::make_shared<::arrow::HalfFloatScalar>(


If this is a new feature, is it tested somewhere?

I think it is more of a fix for incorrect behavior. Previously, float16 statistics were treated as fixed_size_binary[2].

It does not have a standalone unit test, but it is tested in TEST(ParquetPageIndex, FloatingPointOrdersInterop).

pitrou · 2026-09-02T09:28:47Z

+
+TEST(TestFloatStatistics, TotalOrderFloat16) {
+  // -qNaN(payload=1), +qNaN(payload=1).
+  BufferedFloat16 negative_nan(Float16::FromBits(0xfe01));


Unrelated, but I wonder if at some point TypedStatistics<FixedLenByteArray> could have a Update that doesn't involve an array of FLBAs, e.g.:

template <int kByteLength> virtual void Update(std::span<const std::array<uint8_t, kByteLength> values, int64_t null_count) = 0 { UpdateFromLinearBuffer(reinterpret_cast<const uint8_t*>(values.data()), kByteLength, values.size(), null_count); } template <typename Value> virtual std::enable_if_t<std::is_arithmetic_v<Value>> Update( std::span<const Value> values, int64_t null_count) = 0 { UpdateFromLinearBuffer( reinterpret_cast<const uint8_t*>(values.data()), static_cast<int>(sizeof(Value)), values.size(), null_count); } protected: virtual void UpdateFromLinearBuffer( const uint8_t* valyes, int value_length, int64_t num_values, int64_t null_count) = 0;

That could perhaps help us get rid of the BufferedFloat16 mess.

And/or add a Float16Statistics class deriving from TypedStatistics<FixedLenByteArray>, with overloads that take/return Float16 values?

In the refactoring mentioned earlier, I removed BufferedFloat16. However, the issue where FLBA only stores pointers remains. I feel that accepting Float16 might be a direction.

P.S. A member template function cannot be virtual in C++. So the API needs to be adjusted.

pitrou · 2026-09-02T09:33:13Z

+  return lhs_bits <=> rhs_bits;
+}
+
+std::strong_ordering TotalOrderCompare(float lhs, float rhs) {


Can we expose these functions as internal APIs somewhere (parquet/statistics_internal.h perhaps?) and unit-test them directly?

pitrou · 2026-09-02T09:55:24Z


    if (comparator_ == nullptr) return;
+
+    if constexpr (IsOneOf<DType, FloatType, DoubleType, FLBAType>::value) {


It's... the amount of bespoke code that we have to add in this file for such a conceptually simple addition is scary. This is going to become difficult to maintain.

Perhaps the whole internal organization here has become impossible to work with and we should think of something else?

HuaHuaY · 2026-09-19T16:41:11Z

I have addressed most of pitrou's review comments, except for the last two regarding statistics.cc. Let me take a little more time to resolve the remaining comments.

Copilot

Copilot review overview

🟡 Changes recommended

NaN-count correctness, the missing presence API, and null logical-type handling remain unresolved.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1 · 1

Open (2)

Count dictionary NaNs using value multiplicity · New
Add missing nan_counts presence accessor or revise API contract · New

+        UpdateFloatingBounds(
+            [&](auto&& visit) {
+              visit_valid_indices(
+                  [&](int64_t value_index) { visit(array.Value(value_index)); });
+            },
+            update_counts);


Copilot

Copilot review overview

🟡 Changes recommended

Unresolved critical correctness, build, fixture, and API issues remain.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 5

Open (5)

Missing parquet fixture causes test failure · New
Allowing IEEE total order for non-floating types · New
Missing direct <bit> include for std::popcount · New
Bitmap offsets are double-counted for sliced arrays · New
Count dictionary NaNs using value multiplicity

Resolved since last review (1)

Add missing nan_counts presence accessor or revise API contract

+TEST(ParquetPageIndex, FloatingPointOrdersInterop) {
+  auto reader = ParquetFileReader::OpenFile(
+      test::get_data_file("floating_orders_nan_count.parquet"));


+        case ColumnOrder::IEEE_754_TOTAL_ORDER:
+          column_order.__set_IEEE_754_TOTAL_ORDER(format::IEEE754TotalOrder{});
+          break;


 #include "parquet/page_index.h"

 #include <gtest/gtest.h>
+#include <algorithm>


+      auto visit_valid_indices = [&](auto&& visit) {
+        ::arrow::internal::VisitSetBitRunsVoid(
+            values.null_bitmap_data(), values.offset(), values.length(),
+            [&](int64_t position, int64_t run_length) {
+              for (int64_t value_index = 0; value_index < run_length; ++value_index) {
+                visit(position + value_index);
+              }
+            });
+      };


Copilot

Copilot review overview

🔵 Needs a closer look

Restrict IEEE ordering to supported floating types and preserve row-weighted NaN counts for dictionary-encoded columns.

Review effort: Lite
Findings: 5

Open (5)

Bitmap offsets are double-counted for sliced arrays
Missing direct <bit> include for std::popcount
Allowing IEEE total order for non-floating types
Missing parquet fixture causes test failure
Count dictionary NaNs using value multiplicity

HuaHuaY requested review from pitrou and wgtmac as code owners August 4, 2026 15:11

github-actions Bot added awaiting review Awaiting review Component: Parquet Component: C++ labels Aug 4, 2026

HuaHuaY force-pushed the nan_count branch 2 times, most recently from 9b4091a to 5812e60 Compare August 5, 2026 07:09

HuaHuaY commented Aug 5, 2026

View reviewed changes

github-actions Bot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels Aug 5, 2026

HuaHuaY force-pushed the nan_count branch 2 times, most recently from bd13067 to f51ffc1 Compare August 5, 2026 08:03

github-actions Bot added the Component: Documentation label Aug 5, 2026

HuaHuaY force-pushed the nan_count branch from f51ffc1 to 9b29954 Compare August 5, 2026 08:10

wgtmac reviewed Aug 5, 2026

View reviewed changes

Comment thread cpp/src/parquet/types.h Outdated

Comment thread cpp/src/parquet/schema.h Outdated

Comment thread cpp/src/parquet/schema.cc Outdated

Comment thread cpp/src/parquet/statistics.h Outdated

wgtmac reviewed Aug 18, 2026

View reviewed changes

HuaHuaY force-pushed the nan_count branch 3 times, most recently from a306ade to 2f24bb6 Compare August 19, 2026 09:19

Copilot AI lite review requested due to automatic review settings August 25, 2026 06:49

HuaHuaY force-pushed the nan_count branch from 2f24bb6 to 2f25eeb Compare August 25, 2026 06:49

Copilot started reviewing on behalf of HuaHuaY August 25, 2026 06:51 View session

Copilot AI reviewed Aug 25, 2026

View reviewed changes

Comment thread cpp/src/parquet/schema.h

Comment thread cpp/src/parquet/statistics.cc

Copilot AI review requested due to automatic review settings August 27, 2026 02:26

HuaHuaY force-pushed the nan_count branch from 2f25eeb to 295a14a Compare August 27, 2026 02:26

Copilot started reviewing on behalf of HuaHuaY August 27, 2026 02:26 View session

Copilot AI reviewed Aug 27, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings August 27, 2026 09:22

Copilot started reviewing on behalf of HuaHuaY August 27, 2026 09:24 View session

Copilot started reviewing on behalf of HuaHuaY September 1, 2026 09:42 View session

Copilot AI reviewed Sep 1, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings September 1, 2026 09:58

HuaHuaY force-pushed the nan_count branch from 43a2ca5 to 50e7835 Compare September 1, 2026 09:58

Copilot started reviewing on behalf of HuaHuaY September 1, 2026 10:02 View session

Copilot AI reviewed Sep 1, 2026

View reviewed changes

Comment thread cpp/src/parquet/statistics.cc

Copilot AI review requested due to automatic review settings September 1, 2026 10:18

HuaHuaY force-pushed the nan_count branch from 50e7835 to 18569b8 Compare September 1, 2026 10:18

Copilot started reviewing on behalf of HuaHuaY September 1, 2026 10:18 View session

Copilot AI reviewed Sep 1, 2026

View reviewed changes

wgtmac approved these changes Sep 1, 2026

View reviewed changes

pitrou requested changes Sep 2, 2026

View reviewed changes

HuaHuaY mentioned this pull request Sep 17, 2026

GH-50517: [C++] RunEndEncodedBuilder does not merge consecutive NaN values #51325

Open

Add IEEE-754 total order and nan count for floating types

35c2f9d

Copilot AI review requested due to automatic review settings September 19, 2026 16:40

HuaHuaY force-pushed the nan_count branch from 18569b8 to 20e0e3a Compare September 19, 2026 16:40

Copilot started reviewing on behalf of HuaHuaY September 19, 2026 16:40 View session

HuaHuaY force-pushed the nan_count branch from 20e0e3a to b5756b7 Compare September 19, 2026 16:43

Copilot AI reviewed Sep 19, 2026

View reviewed changes

Comment thread cpp/src/parquet/statistics.cc

Comment on lines +992 to +997

UpdateFloatingBounds(

[&](auto&& visit) {

visit_valid_indices(

[&](int64_t value_index) { visit(array.Value(value_index)); });

},

update_counts);

Comment thread cpp/src/parquet/page_index.h

Copilot AI review requested due to automatic review settings September 19, 2026 16:45

Copilot started reviewing on behalf of HuaHuaY September 19, 2026 16:45 View session

Copilot AI reviewed Sep 19, 2026

View reviewed changes

address some review comments

a9c0020

Copilot AI review requested due to automatic review settings September 19, 2026 16:59

HuaHuaY force-pushed the nan_count branch from b5756b7 to a9c0020 Compare September 19, 2026 16:59

Copilot started reviewing on behalf of HuaHuaY September 19, 2026 17:00 View session

Copilot AI reviewed Sep 19, 2026

View reviewed changes


		if (comparator_ == nullptr) return;

		if constexpr (IsOneOf<DType, FloatType, DoubleType, FLBAType>::value) {

Conversation

HuaHuaY commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

HuaHuaY commented Aug 27, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

wgtmac left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wgtmac commented Sep 1, 2026

Uh oh!

pitrou commented Sep 1, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HuaHuaY Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HuaHuaY Sep 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

HuaHuaY commented Aug 4, 2026 •

edited

Loading

HuaHuaY Sep 14, 2026 •

edited

Loading

HuaHuaY Sep 19, 2026 •

edited

Loading

HuaHuaY commented Sep 19, 2026 •

edited

Loading