Skip to content

feat(compute): add dictionary_encode - #1140

Draft
fallintoplace wants to merge 15 commits into
apache:mainfrom
fallintoplace:feat/compute-dictionary-encode
Draft

feat(compute): add dictionary_encode#1140
fallintoplace wants to merge 15 commits into
apache:mainfrom
fallintoplace:feat/compute-dictionary-encode

Conversation

@fallintoplace

@fallintoplace fallintoplace commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Rationale for this change

Arrow-Go currently has a vector hash kernel for unique, but not for dictionary encoding. The existing memo table already assigns stable positions to distinct values, so dictionary encoding can build on that state and provide a useful output form for repeated values.

What changes are included in this PR?

  • Add the dictionary_encode compute function and DictionaryEncodeOptions.
  • Support primitive, binary, temporal, decimal, interval, fixed-size binary, and boolean input types covered by the existing vector hash kernels.
  • Use int32 dictionary indexes and preserve first-seen dictionary order.
  • Mask nulls by default. With NullEncodingEncode, null is included in the dictionary and encoded as a regular index.
  • Keep one shared dictionary across chunked input while returning one index per input value.
  • Pass through already dictionary-encoded input instead of creating a nested dictionary type.
  • Register boolean inputs in the shared vector hash kernels, which also adds unique(bool) support while preserving its existing null-as-a-distinct-value behavior.
  • Correct the unique documentation to state that null is treated as a distinct value.

Are these changes tested?

  • go test ./arrow/compute/... -count=1
  • Added coverage for repeated and all-null values in both null modes, chunked input, dictionary input, dictionary order, output cardinality, boolean unique and dictionary encoding, sliced boolean input, and memo-table resizing.

Are there any user-facing changes?

Yes. This adds the public compute.DictionaryEncode function, DictionaryEncodeOptions, and the NullEncodingMask and NullEncodingEncode options. It also adds boolean input support to unique through the shared vector hash registration.

@fallintoplace
fallintoplace marked this pull request as ready for review August 8, 2026 14:31
@fallintoplace
fallintoplace marked this pull request as draft August 8, 2026 23:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant