Skip to content

perf(numeric): batch safe Decimal64 grouped aggregates - #33

Merged
aunjgr merged 3 commits into
matrixorigin:upstream-dev-mergefrom
aunjgr:perf/exact-decimal64-groupby
Oct 13, 2026
Merged

aunjgr merged 3 commits into
matrixorigin:upstream-dev-mergefrom
aunjgr:perf/exact-decimal64-groupby

Conversation

@aunjgr

@aunjgr aunjgr commented Oct 12, 2026 •

Copy link
Copy Markdown
Collaborator

Grouped exact Decimal64 SUM/AVG currently sort all input rows and run the general checked decimal reduction. Local inputs have at most INT32_MAX rows, so the full signed 64-bit coefficient domain fits in signed 128-bit sums. This lets eligible nodes use one batched cuDF groupby without changing exact output semantics.

Widen operands at their existing scale, share SUM and COUNT requests by input column, normalize all-NULL groups to zero sum/count states, and sign-extend sums when the immutable layout selects Decimal256. Sort only the resulting groups to preserve current group ordering. The whole-node gate accepts ordinary integer/boolean/date/string keys and supported COUNT slots without DISTINCT; other nodes keep the existing path. Checked global merge, final precision validation, AVG rounding, and operation-owned errors stay on their existing implementations. Temporary ownership is retained until stream quiescence, including exceptions.

Refs matrixorigin/matrixone#28966 and matrixorigin/matrixone#28968. Consumer integration: matrixorigin/matrixone#30021. The logical-payload initialization change has merged separately as Sirius #32; this PR retains only the aggregation scope relative to the current base.

Validation on the frozen MO Pixi provider, using an isolated incremental build over merged native b953bb5a:

  • All 14 owning GPU cases and 1,329 assertions pass normally and under memcheck; memcheck reports zero errors.
  • The complete native GPU suite also passes: 39 cases and 35,107 assertions, including scheduler cancellation/reuse and result consumers.
  • Independent expected coefficients and equivalent Decimal128 generic-path controls cover physical INT64 extremes, mixed SUM/AVG/COUNT slots, NULL keys, all-NULL/empty groups, string keys, both state widths, and multiple partials.
  • Final precision overflow remains rejected and a subsequent healthy operation succeeds. Sixty-four allocation cutpoints verify temporary release; existing cancellation and spill/restore cases remain enabled.
  • Pinned changed-file hooks, clang-format 20.1.4, Markdown checks, and diff checks pass.

A separate native-stage benchmark measures 262,144 rows with 21 post-warmup samples per case. Median local-stage time falls from 4.159 to 0.796 ms for 4 groups and from 11.846 to 1.425 ms for 131,072 groups against the same baseline. It is diagnostic evidence; the public Q20 improvement target and full-suite regression gate remain part of #30021's fresh guarded release qualification after merged native pins. No public end-to-end speedup is claimed here.

After merging current upstream-dev-merge (87aaaa24, Sirius #32), conflict resolution 9e137b9d preserves both approved design sections. The aggregation implementation and tests are unchanged. A freshly relinked combined native build passes all 39 GPU cases/35,107 assertions; the 14 aggregation cases/1,329 assertions also pass memcheck with zero errors. Pinned formatting, changed-file hooks, Markdown, and diff checks pass. Earlier benchmark captures retain their original source identities.

@aunjgr
aunjgr merged commit 0cc6e48 into matrixorigin:upstream-dev-merge Oct 13, 2026
6 of 9 checks passed
zuyu pushed a commit that referenced this pull request Oct 14, 2026
* docs(numeric): record Decimal64 grouped fast-path design

* perf(numeric): batch safe Decimal64 grouped aggregates
zuyu pushed a commit that referenced this pull request Oct 21, 2026
* docs(numeric): record Decimal64 grouped fast-path design

* perf(numeric): batch safe Decimal64 grouped aggregates
zuyu pushed a commit that referenced this pull request Oct 22, 2026
* docs(numeric): record Decimal64 grouped fast-path design

* perf(numeric): batch safe Decimal64 grouped aggregates
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant