Skip to content

Fix out-of-bounds write in Newton-Raphson division, Improve normalize - #553

Merged
tompng merged 4 commits into
ruby:masterfrom
tompng:fix_newton_division_slice_copy
Sep 5, 2026
Merged

Fix out-of-bounds write in Newton-Raphson division, Improve normalize#553
tompng merged 4 commits into
ruby:masterfrom
tompng:fix_newton_division_slice_copy

Conversation

@tompng

@tompng tompng commented Sep 4, 2026

Copy link
Copy Markdown
Member

VpSetZero, VpSetInf and VpSetNaN left the exponent of the previous value.
VpNmlz wasn'y normalizing the exponent.
Zero, Infinity and NaN have no meaningful exponent and every reader already
ignores it, but the stale exponent reached slice_copy in Newton-Raphson
division. When a quotient block is zero, BigDecimal_fix of the tiny product
x * inv produces a zero that still has a negative exponent. slice_copy
then underflows the copy length and writes one word outside of the quotient
buffer, corrupting the heap.

Reproduction (with the default threshold):

y = 10**1000 + 7
BigDecimal(y * 10**1800 + 5).divmod(BigDecimal(y))

The result is correct, but GC can crash later. With
NEWTON_RAPHSON_DIVISION_THRESHOLD=1, rake test segfaulted in
test_bigmath about once in four runs.

Changes:

  • Reset exponent in VpSetZero, VpSetInf and VpSetNaN. This alone
    fixes the crash.
  • Fix VpNmlz to normalize exponent
  • Rewrite slice_copy to copy only the intersection of the requested range
    and the existing words, so it can never write outside dest[0, length).
  • Add a regression test for the zero quotient block.

tompng and others added 4 commits September 5, 2026 01:58
These macros left the exponent of the previous value. Zero, Infinity and
NaN have no meaningful exponent and every reader (VpExponent10, VpAsgn, ...)
already ignores it, but the internal inconsistency reached slice_copy in
Newton-Raphson division: BigDecimal_fix of a tiny positive value (a quotient
block that is zero) produced a zero with a negative exponent, and slice_copy
underflowed the copy length and wrote outside of the quotient buffer.
It corrupted the heap for e.g. BigDecimal(y * 10**1800 + 5).divmod(BigDecimal(y))
with y = 10**1000 + 7, and rake test with NEWTON_RAPHSON_DIVISION_THRESHOLD=1
crashed in GC about once in four runs.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The range can be entirely above the most significant word or below the least
significant word of the source. Clamp both ends instead of adjusting the start
afterwards, so that the copy never exceeds dest[0, length).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
VpDivdNewtonInner ignored the result of AddExponent and kept adding to the
exponent after an overflow turned the quotient into Infinity, and also added
to the exponent of a zero quotient or remainder. Apply the total shift in one
call and skip it for zero.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Rounding a small value with ROUND_CEILING or ROUND_FLOOR beyond its first
digit can result in zero, and the exponent compensation for the rounding
position was applied to that zero.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@tompng
tompng merged commit 655823d into ruby:master Sep 5, 2026
83 checks passed
@tompng
tompng deleted the fix_newton_division_slice_copy branch September 5, 2026 04:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant