Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
89 commits
Select commit Hold shift + click to select a range
04776a6
GH-17211: refresh history for scalar_hash kernel
drin Jan 12, 2024
bef8c0b
GH-17211: refresh history for updates to key_hash
drin Jan 12, 2024
a1adc88
GH-17211: refresh history for new compute fn infra
drin Jan 12, 2024
98aaaf3
GH-17211: refresh history of scalar hash benchmark
drin Jan 12, 2024
362a5b5
added KeyColumnArray alias
drin Oct 7, 2024
ab02dd1
task: revert style and other minor changes
drin Oct 11, 2024
b427d5e
task: reducing support to some nested types
drin Oct 11, 2024
4502af2
minor: added type check for ListArray
drin Oct 11, 2024
77b42eb
Update the implementation to use column oriented hash-combine.
kszucs Dec 10, 2024
0e947ae
add hypothesis test case and fix list hashing
kszucs Dec 11, 2024
43d2be4
ensure to keep newly allocated arrays alive until hash gets computed
kszucs Dec 11, 2024
6a52388
minor cleanup
kszucs Dec 11, 2024
02ad712
wrong include
kszucs Dec 11, 2024
776c201
add hash32 function
kszucs Dec 13, 2024
484ae79
add hash32 function and address review comments
kszucs Dec 13, 2024
bf6d0d7
update testing suite
kszucs Dec 13, 2024
6d2d913
generated struct tests
kszucs Dec 13, 2024
910361c
prepare adding hash32 tests
kszucs Dec 13, 2024
fe2fb0a
add tests for hash32, dictionary encoded type, extension type
kszucs Dec 16, 2024
dfe2bcc
ensure that NULL and 0 hashes to different numbers
kszucs Dec 16, 2024
cfd66e8
fix the benchmarks
kszucs Dec 16, 2024
db499a6
update docs to highlight 0 return for NULL values
kszucs Dec 16, 2024
6465886
use resize instead of reserve
kszucs Dec 17, 2024
c037429
fix R compute function filtering test
kszucs Dec 17, 2024
38340a1
address review comments
kszucs Dec 18, 2024
374d566
update boolean test case
kszucs Dec 18, 2024
3200d6d
update exec checks
kszucs Dec 18, 2024
8c25799
use Hasher::kHashBatchTempStackUsage
kszucs Dec 18, 2024
28aa8a4
ensure that slices properly work
kszucs Dec 18, 2024
516abfd
add tests for fixed_size_binary()
kszucs Dec 18, 2024
1c167a5
add rudimentary quality tests
kszucs Dec 18, 2024
7a812f6
test temporal types too
kszucs Dec 19, 2024
6feca33
test temporal and decimal types
kszucs Dec 19, 2024
6e8a096
add type matcher to explicitly raise NotImplemented for unsupported t…
kszucs Dec 19, 2024
8df07a9
update error message
kszucs Dec 19, 2024
6293b44
update docs
kszucs Dec 19, 2024
1ea5599
set a tolerance for the randomly generated numbers
kszucs Dec 19, 2024
93c4add
make msvc happy
kszucs Dec 20, 2024
141d3d4
Rename DCHECK macros to ARROW_DCHECK
kszucs May 13, 2025
59d254e
Missing blank lines
kszucs May 14, 2025
a9686c8
chore: move key_hash benchmarks to key_hash_benchmark.cc under compute
kszucs Jun 20, 2025
63aaf30
Apply suggestions from code review
kszucs Jun 25, 2025
4d2a8e3
use type singleton
kszucs Jun 25, 2025
06a6773
run clang format on scalar_hash_benchmark.cc
kszucs Jun 25, 2025
2914268
run cmake format
kszucs Jun 25, 2025
e60dbd7
Address review comments and fix missing-declarations build failure
kszucs Jul 23, 2026
901c6dd
Fix unreachable slicing check and remove unused hashing_rng in benchm…
kszucs Jul 23, 2026
96e7fcc
Fix nested-type null and hash-corruption bugs in hash32/hash64
kszucs Jul 23, 2026
c5e12b7
Remove dead list-like branches from ToColumnArray
kszucs Jul 23, 2026
96039e5
Fix independently-null nested fields hashing as non-zero garbage
kszucs Jul 23, 2026
5ffc10d
Size key_hash_benchmark's TempVectorStack from kHashBatchTempStackUsage
kszucs Jul 23, 2026
1e8d5ed
Hoist KeyColumnArray construction out of key_hash_benchmark's timed loop
kszucs Jul 23, 2026
85b090a
Fix GCC -Werror=range-loop-construct in scalar_hash_test.cc
kszucs Jul 23, 2026
7872dce
Guard against a null values buffer in ToColumnArray's binary path
kszucs Jul 23, 2026
a099eed
Add R test asserting hash32/hash64 are discoverable via list_compute_…
kszucs Jul 23, 2026
d090d5e
Avoid hashing unreferenced child data for sliced list/map arrays, fix…
kszucs Jul 24, 2026
31101f9
Add hypothesis tests asserting hash32/hash64 respect Array.slice()
kszucs Jul 24, 2026
b01857d
Revert kAddend change to the shared HashIntImp; scope the null/0 fix …
kszucs Jul 24, 2026
6b65e6d
Unwrap extension types in HashableMatcher so unsupported storage type…
kszucs Jul 24, 2026
3193d05
Add test coverage for zero/null collision across byte widths and FIXE…
kszucs Jul 24, 2026
859414c
Fix HashChild reusing a nested field's unshifted validity buffer
kszucs Jul 24, 2026
5bdf006
Avoid the bitmap copy in HashChild by omitting the validity buffer en…
kszucs Jul 24, 2026
1af6e2c
Fix double blank lines around the new Hash Functions doc section
kszucs Jul 24, 2026
3ef5649
Avoid hashing unreferenced struct nested-field data, avoid gtest macr…
kszucs Jul 24, 2026
372f0b1
Hoist the per-column vector out of KeyHashIntegers benchmark loops
kszucs Jul 24, 2026
9877f6b
Fix Hash32/Hash64 API docstrings
kszucs Jul 24, 2026
b6691d5
Fix values-child offset cancellation bug in list/map hashing
kszucs Jul 24, 2026
ed7c83e
Fix valid struct rows coincidentally colliding with the null hash sen…
kszucs Jul 24, 2026
746050c
Loosen RandomPrimitive's hash-quality tolerance for cross-platform RN…
kszucs Jul 24, 2026
57f24ac
Fix independently-null fields of multi-field structs not hashing to 0
kszucs Jul 27, 2026
11bfed4
Dedupe Hash64Struct{Small,Medium,Large}Strings benchmarks
kszucs Jul 27, 2026
015485e
Make hash32/hash64 null-aware instead of reserving 0 for nulls
kszucs Jul 27, 2026
1e62e95
Fix zero-width fixed_size_binary hashing, and simplify
kszucs Jul 28, 2026
22e3e0b
Fix NullType struct fields not invalidating their row's hash
kszucs Aug 24, 2026
b64054c
Add DoNotOptimize/ClobberMemory to key hash benchmarks
kszucs Sep 2, 2026
3c6c2b7
Bulk-copy hash output validity, and fix doc nits
kszucs Sep 2, 2026
01e2916
Reject unsupported nested child types during hash dispatch
kszucs Sep 2, 2026
0e2d8b0
Drop unused builder_primitive.h include from hashing_benchmark
kszucs Sep 2, 2026
28fb6a9
Route zero-width fixed_size_binary struct fields through HashArray
kszucs Sep 2, 2026
8615f2f
Add the new scalar-hash sources to the Meson build
kszucs Sep 2, 2026
c65c43e
Distinguish a null child element from one hashing to zero
kszucs Sep 3, 2026
987c5d3
Hash map keys even when the item is null
kszucs Sep 3, 2026
1e06942
Move zero-width struct comment to its own test
kszucs Sep 7, 2026
a3c1c8b
Close scalar-hash test coverage gaps found in review
kszucs Sep 7, 2026
defc465
Add missing standard-library includes flagged in PR review
kszucs Sep 7, 2026
dd8323f
Cover a scalar argument to hash32/hash64
kszucs Sep 9, 2026
ddba847
Document the scalar and chunked input shapes for hash32/hash64
kszucs Sep 9, 2026
f32786c
Spell out the null and stability semantics in the hash FunctionDocs
kszucs Sep 11, 2026
e052771
Feed HashMultiColumn row counts it can represent
kszucs Sep 11, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions cpp/src/arrow/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -869,6 +869,7 @@ if(ARROW_COMPUTE)
compute/kernels/scalar_arithmetic.cc
compute/kernels/scalar_boolean.cc
compute/kernels/scalar_compare.cc
compute/kernels/scalar_hash.cc
compute/kernels/scalar_if_else.cc
Comment thread
kszucs marked this conversation as resolved.
compute/kernels/scalar_nested.cc
compute/kernels/scalar_random.cc
Expand Down
1 change: 1 addition & 0 deletions cpp/src/arrow/compute/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -185,6 +185,7 @@ add_arrow_compute_test(row_test
arrow_compute_testing)

add_arrow_compute_benchmark(function_benchmark)
add_arrow_compute_benchmark(key_hash_benchmark)

add_subdirectory(kernels)

Expand Down
10 changes: 10 additions & 0 deletions cpp/src/arrow/compute/api_scalar.cc
Original file line number Diff line number Diff line change
Expand Up @@ -957,6 +957,16 @@ Result<Datum> MapLookup(const Datum& arg, MapLookupOptions options, ExecContext*
return CallFunction("map_lookup", {arg}, &options, ctx);
}

// ----------------------------------------------------------------------
// Hash functions
Result<Datum> Hash32(const Datum& input_array, ExecContext* ctx) {
return CallFunction("hash32", {input_array}, ctx);
}

Result<Datum> Hash64(const Datum& input_array, ExecContext* ctx) {
return CallFunction("hash64", {input_array}, ctx);
}

// ----------------------------------------------------------------------

} // namespace compute
Expand Down
43 changes: 43 additions & 0 deletions cpp/src/arrow/compute/api_scalar.h
Original file line number Diff line number Diff line change
Expand Up @@ -1807,5 +1807,48 @@ ARROW_EXPORT Result<Datum> NanosecondsBetween(const Datum& left, const Datum& ri
/// \note API not yet finalized
ARROW_EXPORT Result<Datum> MapLookup(const Datum& map, MapLookupOptions options,
ExecContext* ctx = NULLPTR);

/// \brief Construct a hash value for each row of the input.
///
/// The result has the same length and shape as the input (Scalar in, Scalar out;
/// Array in, Array out; ChunkedArray in, ChunkedArray out), but with element type
/// UInt32. For a nested input type (struct, list, map, etc.), each row's child values
/// are combined into a single hash for that row, recursively. A null input row produces
/// a null output row; within a struct, a null field makes that whole row null, while
/// within a list or map a null element does not (only the row's own validity matters
/// there). Hash values are not guaranteed to be stable across different versions of the
/// library, and this function does not currently take options, though these may be
/// added in the future.
///
/// \param[in] input_array input data to hash
/// \param[in] ctx function execution context, optional
/// \return elementwise hash values
///
/// \since 26.0.0
/// \note API not yet finalized
ARROW_EXPORT
Result<Datum> Hash32(const Datum& input_array, ExecContext* ctx = NULLPTR);

/// \brief Construct a hash value for each row of the input.
///
/// The result has the same length and shape as the input (Scalar in, Scalar out;
/// Array in, Array out; ChunkedArray in, ChunkedArray out), but with element type
/// UInt64. For a nested input type (struct, list, map, etc.), each row's child values
/// are combined into a single hash for that row, recursively. A null input row produces
/// a null output row; within a struct, a null field makes that whole row null, while
/// within a list or map a null element does not (only the row's own validity matters
/// there). Hash values are not guaranteed to be stable across different versions of the
/// library, and this function does not currently take options, though these may be
/// added in the future.
///
/// \param[in] input_array input data to hash
/// \param[in] ctx function execution context, optional
/// \return elementwise hash values
///
/// \since 26.0.0
/// \note API not yet finalized
ARROW_EXPORT
Result<Datum> Hash64(const Datum& input_array, ExecContext* ctx = NULLPTR);

} // namespace compute
} // namespace arrow
1 change: 1 addition & 0 deletions cpp/src/arrow/compute/initialize.cc
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ Status RegisterComputeKernels() {
internal::RegisterScalarArithmetic(registry);
internal::RegisterScalarBoolean(registry);
internal::RegisterScalarComparison(registry);
internal::RegisterScalarHash(registry);
internal::RegisterScalarIfElse(registry);
internal::RegisterScalarNested(registry);
internal::RegisterScalarRandom(registry); // Nullary
Expand Down
2 changes: 2 additions & 0 deletions cpp/src/arrow/compute/kernels/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,7 @@ add_arrow_compute_test(scalar_math_test

add_arrow_compute_test(scalar_utility_test
SOURCES
scalar_hash_test.cc
scalar_random_test.cc
scalar_set_lookup_test.cc
scalar_validity_test.cc
Expand All @@ -89,6 +90,7 @@ add_arrow_benchmark(scalar_cast_benchmark PREFIX "arrow-compute")
add_arrow_compute_benchmark(scalar_arithmetic_benchmark)
add_arrow_compute_benchmark(scalar_boolean_benchmark)
add_arrow_compute_benchmark(scalar_compare_benchmark)
add_arrow_compute_benchmark(scalar_hash_benchmark)
add_arrow_compute_benchmark(scalar_if_else_benchmark)
add_arrow_compute_benchmark(scalar_list_benchmark)
add_arrow_compute_benchmark(scalar_random_benchmark)
Expand Down
2 changes: 2 additions & 0 deletions cpp/src/arrow/compute/kernels/meson.build
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,7 @@ scalar_kernel_tests = {
},
'arrow-compute-scalar-utility-test': {
'sources': [
'scalar_hash_test.cc',
'scalar_random_test.cc',
'scalar_set_lookup_test.cc',
'scalar_validity_test.cc',
Expand All @@ -76,6 +77,7 @@ scalar_kernel_benchmarks = [
'scalar_boolean_benchmark',
'scalar_cast_benchmark',
'scalar_compare_benchmark',
'scalar_hash_benchmark',
'scalar_if_else_benchmark',
'scalar_list_benchmark',
'scalar_random_benchmark',
Expand Down
Loading
Loading