perf(array): accumulate nested builder validity without a null buffer [builders-child-stack] - #8966
Conversation
cc36904 to
6f3ef13
Compare
85f8dcd to
b67473e
Compare
6f3ef13 to
20dd597
Compare
Merging this PR will not alter performance
Comparing Footnotes
|
b67473e to
8561525
Compare
8561525 to
a8ebf30
Compare
df75205 to
c30fa2a
Compare
a8ebf30 to
9f2a3d8
Compare
c30fa2a to
e8186e0
Compare
9f2a3d8 to
7b61176
Compare
|
Do we still need LazyBitBufferBuilder? |
e8186e0 to
fe7132d
Compare
7b61176 to
32c0dbd
Compare
fe7132d to
9b0fd4b
Compare
32c0dbd to
9c58b3c
Compare
|
On
What this PR does remove is the nested builders' use of it: I did not migrate the flat builders on purpose: a flat builder copies its values into one contiguous buffer, so there is no chunk boundary for validity runs to line up with, and keeping runs there would hand downstream a chunked bool array for an otherwise flat array. Say the word if you would rather have one accumulator everywhere and I will do it. |
9b0fd4b to
bc03a46
Compare
9c58b3c to
4e607da
Compare
bc03a46 to
79dc1ec
Compare
4e607da to
6541973
Compare
79dc1ec to
c9bacae
Compare
6541973 to
dd4420b
Compare
c9bacae to
edb04e4
Compare
5e51643 to
421b6d4
Compare
A nested builder learns about validity from two sources: one row at a time as scalars are appended, and a whole array's worth at a time as arrays are. Only the first needs a null buffer, but `LazyBitBufferBuilder` treated both the same, so every appended array had its validity executed into a `Mask` and its bits copied. `ValidityBuilder` keeps a whole array's validity as a run and concatenates the runs at the end, the way `Validity::concat` already does for `StructArray::try_concat`. `AllValid` and `AllInvalid` runs cost nothing, array-backed runs are bool arrays that are already built, and a builder that only ever saw uniform validity still answers from its nullability rather than producing a bool array. However few values a run covers, it is kept as it arrived, so a builder's validity is split on exactly the boundaries its children are. `StructBuilder`, `ListBuilder`, `ListViewBuilder` and `FixedSizeListBuilder` use it; the leaf builders keep `LazyBitBufferBuilder`. Signed-off-by: Claude <noreply@anthropic.com> Signed-off-by: Robert Kruszewski <robert@spiraldb.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
421b6d4 to
b24a656
Compare
edb04e4 to
76b988c
Compare
Rationale for this change
Stacked on #8964 — review that first.
After the parent merges the arrays are lazy but validity is not. With this change the validity is lazy as well