Skip to content

feat(array): builders no longer canonicalize their children [builders-child-stack] - #8964

Closed
robert3005 wants to merge 1 commit into
claude/builders-bulk-loop-callers-9ze0t6from
claude/builders-canonical-children-9ze0t6
Closed

feat(array): builders no longer canonicalize their children [builders-child-stack]#8964
robert3005 wants to merge 1 commit into
claude/builders-bulk-loop-callers-9ze0t6from
claude/builders-canonical-children-9ze0t6

Conversation

@robert3005

@robert3005 robert3005 commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

Builders are lazy. Fully canonical builders force a lot of unnecessary decompression which goes against what vortex aims for. With this change builders behave more like canonical arrays where the top level isn't lazy but the children are. Practically speaking this means that all nested type builders store lazy children and validity

@codspeed-hq

codspeed-hq Bot commented Jul 25, 2026

Copy link
Copy Markdown

Merging this PR will improve performance by 46.53%

⚡ 8 improved benchmarks
✅ 1921 untouched benchmarks
⏩ 51 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation extend_from_array_zctl[(1000, 64)] 2,184.4 µs 722.9 µs ×3
Simulation extend_from_array_zctl[(10000, 8)] 3.6 ms 1.8 ms ×2
Simulation decompress[u64, (1000, 16)] 130 µs 73.1 µs +77.78%
Simulation extend_from_array_zctl[(1000, 8)] 903.1 µs 715.5 µs +26.21%
Simulation chunked_dict_primitive_canonical_into[f32, (1000, 100, 100)] 1.7 ms 1.5 ms +13.09%
Simulation extend_from_array_non_zctl_overlapping[(10000, 8)] 2.1 ms 1.8 ms +12.77%
Simulation chunked_dict_primitive_canonical_into[u32, (1000, 10, 100)] 1.6 ms 1.5 ms +11.11%
Simulation search_index_in_range_chunked 5.9 ms 5.3 ms +10.12%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.


Comparing claude/builders-canonical-children-9ze0t6 (76b988c) with claude/builders-bulk-loop-callers-9ze0t6 (e570927)

Open in CodSpeed

Footnotes

  1. 51 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@claude claude Bot changed the title feat(array): builders no longer canonicalize their children feat(array): builders no longer canonicalize their children [builders-child-stack] Jul 25, 2026
@robert3005 robert3005 added the changelog/break A breaking API change label Jul 25, 2026
@robert3005
robert3005 force-pushed the claude/builders-canonical-children-9ze0t6 branch from 2e987a3 to b4cb8fb Compare July 25, 2026 21:07
@robert3005
robert3005 requested a review from joseph-isaacs July 27, 2026 13:36
@robert3005
robert3005 requested a review from AdamGS July 27, 2026 16:17
Comment thread vortex-array/src/builders/child.rs Outdated
@robert3005
robert3005 force-pushed the claude/builders-canonical-children-9ze0t6 branch from b4cb8fb to c78ed3c Compare July 29, 2026 07:35
@robert3005
robert3005 changed the base branch from develop to claude/listview-builder-keep-overlaps-9ze0t6 July 29, 2026 07:35
@robert3005
robert3005 force-pushed the claude/builders-canonical-children-9ze0t6 branch from c78ed3c to e45996f Compare July 29, 2026 13:48
@robert3005
robert3005 changed the base branch from claude/listview-builder-keep-overlaps-9ze0t6 to claude/builders-bulk-loop-callers-9ze0t6 July 29, 2026 13:50
@robert3005
robert3005 force-pushed the claude/builders-canonical-children-9ze0t6 branch from e45996f to df75205 Compare July 29, 2026 14:19
@robert3005
robert3005 force-pushed the claude/builders-bulk-loop-callers-9ze0t6 branch from b622d0f to 3304731 Compare July 29, 2026 14:19
@robert3005
robert3005 force-pushed the claude/builders-bulk-loop-callers-9ze0t6 branch from 3304731 to 118384a Compare July 31, 2026 13:35
@robert3005
robert3005 force-pushed the claude/builders-canonical-children-9ze0t6 branch from df75205 to c30fa2a Compare July 31, 2026 13:35
@robert3005
robert3005 force-pushed the claude/builders-bulk-loop-callers-9ze0t6 branch 2 times, most recently from 022be8b to 10d22d5 Compare July 31, 2026 15:29
@robert3005
robert3005 force-pushed the claude/builders-canonical-children-9ze0t6 branch 2 times, most recently from e8186e0 to fe7132d Compare July 31, 2026 16:54
@robert3005
robert3005 force-pushed the claude/builders-bulk-loop-callers-9ze0t6 branch from 7e3b8e7 to 39e5a06 Compare July 31, 2026 16:54
@robert3005
robert3005 force-pushed the claude/builders-canonical-children-9ze0t6 branch 2 times, most recently from 9b0fd4b to bc03a46 Compare July 31, 2026 18:38
@robert3005
robert3005 force-pushed the claude/builders-bulk-loop-callers-9ze0t6 branch from 39e5a06 to 24b26b3 Compare July 31, 2026 18:39
@robert3005
robert3005 force-pushed the claude/builders-canonical-children-9ze0t6 branch from bc03a46 to 79dc1ec Compare July 31, 2026 18:43
@robert3005
robert3005 force-pushed the claude/builders-bulk-loop-callers-9ze0t6 branch from 24b26b3 to ade8dbe Compare July 31, 2026 19:19
@robert3005
robert3005 force-pushed the claude/builders-canonical-children-9ze0t6 branch from 79dc1ec to c9bacae Compare July 31, 2026 19:20
@robert3005
robert3005 requested a review from onursatici August 4, 2026 15:47
@robert3005

Copy link
Copy Markdown
Contributor Author

This is the meat of this change

@robert3005
robert3005 force-pushed the claude/builders-bulk-loop-callers-9ze0t6 branch from ade8dbe to 043d08a Compare August 5, 2026 12:50
@robert3005
robert3005 force-pushed the claude/builders-canonical-children-9ze0t6 branch from c9bacae to edb04e4 Compare August 5, 2026 12:51
Nested builders used to push every appended child array through
`append_to_builder`, which decoded it into the child's canonical builder.
That work is wasted: `Canonical` only promises a canonical *top level*, so
struct fields, list elements and extension storage are free to stay
compressed.

Introduce `ChildBuilder`, which accumulates a child as a `Vec<ArrayRef>` of
chunks plus a scalar builder for the values that cannot come from an array,
and stitches them into a `ChunkedArray` on `finish` when more than one chunk
accumulated. `StructBuilder`, `ListBuilder`, `ListViewBuilder`,
`FixedSizeListBuilder` and `ExtensionBuilder` now hold their children this
way.

However short the appended array, it becomes a chunk. Deciding on the caller's
behalf that its values are cheaper copied than referenced would be guessing at
a boundary only the caller can see, and a caller that wants them copied has
`append_scalar`. A child is therefore chunked on exactly the boundaries it was
appended on.

Signed-off-by: Claude <noreply@anthropic.com>
Signed-off-by: Robert Kruszewski <robert@spiraldb.com>
Signed-off-by: Robert Kruszewski <github@robertk.io>

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@robert3005
robert3005 force-pushed the claude/builders-bulk-loop-callers-9ze0t6 branch from 043d08a to e570927 Compare August 7, 2026 10:40
@robert3005
robert3005 force-pushed the claude/builders-canonical-children-9ze0t6 branch from edb04e4 to 76b988c Compare August 7, 2026 10:40
@robert3005

Copy link
Copy Markdown
Contributor Author

Collapsed into #9064 along with #8966 and #9131.

@robert3005 robert3005 closed this Aug 7, 2026
@robert3005
robert3005 deleted the claude/builders-canonical-children-9ze0t6 branch August 7, 2026 11:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/break A breaking API change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant