quantize() violates IEEE 754-2008 §5.3.5; test encodes the same wrong formula

[acko1980]

Summary

Two related bugs:

1. quantize() does not rescale the operand significand, so it returns a value that differs from the input by orders of magnitude whenever the input cohort is not already at the target exponent.
2. The unit test for quantize() checks the result against an "expected" value computed with the same broken formula as the implementation, so the test passes regardless of whether quantize is conformant.

This has been verified present in every tagged release back to v1.0.1.

---

Bug 1: quantize() implementation does not rescale

Location (v6.0.1, commit 9429c29f): include/boost/decimal/decimal64_t.hpp ~line 2205. Almost certainly the same shape in decimal32_t.hpp, decimal128_t.hpp, and the _fast variants.

return {lhs.full_significand(), rhs.biased_exponent(), lhs.isneg()};

This constructs a result from lhs's raw cohort significand combined with rhs's biased exponent — i.e. lhs.full_significand() × 10^rhs.biased_exponent() — without any rescaling step. The numeric value of lhs is full_significand × 10^biased_exponent, so this only equals lhs in the trivial case where lhs's significand happens to already be expressed at rhs's scale.

Per IEEE 754-2008 §5.3.5, quantize(x, y) must return a value whose quantum equals y's quantum and whose numeric value equals (a correctly rounded) x. After any arithmetic that normalizes the cohort to the full 16-digit precision (decimal64), results are off by orders of magnitude.

Minimal reproducer

#include <boost/decimal/decimal64_t.hpp>
#include <boost/decimal/cmath.hpp>
#include <iostream>

int main() {
    using namespace boost::decimal;
    decimal64_t a {2, -4};      // 0.0002 (sig=2, exp=-4)
    decimal64_t b = decimal64_t{15, -5} + decimal64_t{5, -5};  // also 0.0002, but cohort sig is much larger after addition
    decimal64_t target {1, -4}; // 0.0001 — desired quantum
    std::cout << "a == b: " << (a == b) << " (both numerically 0.0002)\n";
    std::cout << "quantize(a, target): " << quantize(a, target) << " (correct: 0.0002)\n";
    std::cout << "quantize(b, target): " << quantize(b, target) << " (wrong: ~2e+11)\n";
}

a and b compare equal numerically (both are 0.0002), but quantize(a, target) returns 0.0002 while quantize(b, target) returns ~2e+11. A conformant implementation must return 0.0002 in both cases.

---

Bug 2: the test encodes the same wrong formula as its expected value

Location: test/test_decimal_quantum.cpp lines 124–158, function test_quantize.

const Dec val1 {sig1, exp1};
const Dec val2 {sig2, exp2};
const Dec quantized_val {sig1, exp2};                       // <- "expected" value
if (!BOOST_TEST_EQ(quantize(val1, val2), quantized_val))    // <- actual

The expected value quantized_val is constructed as {sig1, exp2} — i.e. {lhs.sig, rhs.exp} — exactly the same broken formula the implementation uses. So quantize(val1, val2) == Dec{sig1, exp2} will always hold under the current implementation, regardless of whether quantize is conformant. This is presumably why the bug has shipped undetected.

The test should compare against an independently computed expected value. The IEEE 754-2008 / decTest conformance vectors maintained by Mike Cowlishaw (https://speleotrove.com/decimal/dectest.html) have been the gold-standard test corpus for decimal arithmetic conformance for ~20 years and include quantize cases. They would be a natural basis for new tests, though the maintainers may want to consider how (or whether) to vendor them.

---

Version history

Verified the same {lhs.sig, rhs.exp} shape is present in every tagged release: v1.0.1, v2.0.0, v3.0.0, v4.0.0, v5.0.0, v5.2.0, v6.0.0, v6.0.1. The library has shipped a non-conformant quantize and a test that affirms the non-conformance for its entire public history.

---

Not proposing a PR — happy to leave the fix shape and the test approach (vendor decTest vs. write hand-rolled vectors against decTest's expected values, etc.) to the maintainers.

[mborland]

I'll take a look. decTest is hosted in a separate repo due to license incompatibilities, but it is pulled in here for the CI runs. You can find that repo here: https://github.com/cppalliance/decimal-dectest