feat(array): builders no longer canonicalize their children [builders-child-stack] - #8964
Closed
robert3005 wants to merge 1 commit into
Closed
Conversation
Merging this PR will improve performance by 46.53%
Performance Changes
Tip Curious why this is faster? Comment Comparing Footnotes
|
This was referenced Jul 25, 2026
Closed
Closed
robert3005
force-pushed
the
claude/builders-canonical-children-9ze0t6
branch
from
July 25, 2026 21:07
2e987a3 to
b4cb8fb
Compare
robert3005
commented
Jul 27, 2026
robert3005
force-pushed
the
claude/builders-canonical-children-9ze0t6
branch
from
July 29, 2026 07:35
b4cb8fb to
c78ed3c
Compare
robert3005
changed the base branch from
develop
to
claude/listview-builder-keep-overlaps-9ze0t6
July 29, 2026 07:35
robert3005
force-pushed
the
claude/builders-canonical-children-9ze0t6
branch
from
July 29, 2026 13:48
c78ed3c to
e45996f
Compare
robert3005
changed the base branch from
claude/listview-builder-keep-overlaps-9ze0t6
to
claude/builders-bulk-loop-callers-9ze0t6
July 29, 2026 13:50
robert3005
force-pushed
the
claude/builders-canonical-children-9ze0t6
branch
from
July 29, 2026 14:19
e45996f to
df75205
Compare
robert3005
force-pushed
the
claude/builders-bulk-loop-callers-9ze0t6
branch
from
July 29, 2026 14:19
b622d0f to
3304731
Compare
robert3005
force-pushed
the
claude/builders-bulk-loop-callers-9ze0t6
branch
from
July 31, 2026 13:35
3304731 to
118384a
Compare
robert3005
force-pushed
the
claude/builders-canonical-children-9ze0t6
branch
from
July 31, 2026 13:35
df75205 to
c30fa2a
Compare
robert3005
force-pushed
the
claude/builders-bulk-loop-callers-9ze0t6
branch
2 times, most recently
from
July 31, 2026 15:29
022be8b to
10d22d5
Compare
robert3005
force-pushed
the
claude/builders-canonical-children-9ze0t6
branch
2 times, most recently
from
July 31, 2026 16:54
e8186e0 to
fe7132d
Compare
robert3005
force-pushed
the
claude/builders-bulk-loop-callers-9ze0t6
branch
from
July 31, 2026 16:54
7e3b8e7 to
39e5a06
Compare
robert3005
force-pushed
the
claude/builders-canonical-children-9ze0t6
branch
2 times, most recently
from
July 31, 2026 18:38
9b0fd4b to
bc03a46
Compare
robert3005
force-pushed
the
claude/builders-bulk-loop-callers-9ze0t6
branch
from
July 31, 2026 18:39
39e5a06 to
24b26b3
Compare
robert3005
force-pushed
the
claude/builders-canonical-children-9ze0t6
branch
from
July 31, 2026 18:43
bc03a46 to
79dc1ec
Compare
robert3005
force-pushed
the
claude/builders-bulk-loop-callers-9ze0t6
branch
from
July 31, 2026 19:19
24b26b3 to
ade8dbe
Compare
robert3005
force-pushed
the
claude/builders-canonical-children-9ze0t6
branch
from
July 31, 2026 19:20
79dc1ec to
c9bacae
Compare
Contributor
Author
|
This is the meat of this change |
robert3005
force-pushed
the
claude/builders-bulk-loop-callers-9ze0t6
branch
from
August 5, 2026 12:50
ade8dbe to
043d08a
Compare
robert3005
force-pushed
the
claude/builders-canonical-children-9ze0t6
branch
from
August 5, 2026 12:51
c9bacae to
edb04e4
Compare
Nested builders used to push every appended child array through `append_to_builder`, which decoded it into the child's canonical builder. That work is wasted: `Canonical` only promises a canonical *top level*, so struct fields, list elements and extension storage are free to stay compressed. Introduce `ChildBuilder`, which accumulates a child as a `Vec<ArrayRef>` of chunks plus a scalar builder for the values that cannot come from an array, and stitches them into a `ChunkedArray` on `finish` when more than one chunk accumulated. `StructBuilder`, `ListBuilder`, `ListViewBuilder`, `FixedSizeListBuilder` and `ExtensionBuilder` now hold their children this way. However short the appended array, it becomes a chunk. Deciding on the caller's behalf that its values are cheaper copied than referenced would be guessing at a boundary only the caller can see, and a caller that wants them copied has `append_scalar`. A child is therefore chunked on exactly the boundaries it was appended on. Signed-off-by: Claude <noreply@anthropic.com> Signed-off-by: Robert Kruszewski <robert@spiraldb.com> Signed-off-by: Robert Kruszewski <github@robertk.io> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
robert3005
force-pushed
the
claude/builders-bulk-loop-callers-9ze0t6
branch
from
August 7, 2026 10:40
043d08a to
e570927
Compare
robert3005
force-pushed
the
claude/builders-canonical-children-9ze0t6
branch
from
August 7, 2026 10:40
edb04e4 to
76b988c
Compare
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builders are lazy. Fully canonical builders force a lot of unnecessary decompression which goes against what vortex aims for. With this change builders behave more like canonical arrays where the top level isn't lazy but the children are. Practically speaking this means that all nested type builders store lazy children and validity