Skip to content

Respect rounding mode when quantizing sub-ULP values - #152

Open
chiliec wants to merge 1 commit into
cockroachdb:masterfrom
chiliec:fix-quantize-subulp-rounding
Open

Respect rounding mode when quantizing sub-ULP values#152
chiliec wants to merge 1 commit into
cockroachdb:masterfrom
chiliec:fix-quantize-subulp-rounding

Conversation

@chiliec

@chiliec chiliec commented Sep 2, 2026

Copy link
Copy Markdown

Fixes #140

What

Context.Quantize ignored the rounding mode when the input had fewer significant digits than the target exponent (i.e. its magnitude was smaller than one unit in the last place). It truncated straight to zero regardless of the configured Rounder:

ctx := apd.Context{Precision: 100, Rounding: apd.RoundCeiling, MaxExponent: 1000, MinExponent: -1000, Traps: apd.DefaultTraps}
var d apd.Decimal
d.SetString("0.002")
ctx.Quantize(&d, &d, -1)
// got "0.0", want "0.1"

RoundCeiling(1.002, -1) already returned 1.1; only the sub-ULP case was wrong.

Fix

In the p < 0 branch of quantize, run the discarded coefficient through the context's Rounding.ShouldAddOne (the same mechanism the normal rounding path uses) instead of unconditionally zeroing it.

Tests

Added TestQuantizeRounding covering the reported case plus every rounding mode (ceiling/floor/up/down/half-up/half-even, positive and negative). Verified genuine RED→GREEN:

# with fix
$ go test -run TestQuantizeRounding .
ok  github.com/cockroachdb/apd/v3

# revert context.go, keep the test
$ git stash push context.go && go test -run TestQuantizeRounding .
--- FAIL: TestQuantizeRounding/0.002:_-1_ceiling
    expected: 0.1, got: 0.0
--- FAIL: TestQuantizeRounding/-0.002:_-1_floor
    expected: -0.1, got: -0.0
FAIL

Full suite (including the GDA conformance tests) is green:

$ go test ./...
ok  github.com/cockroachdb/apd/v3  1.524s

Happy to adjust.

Quantize truncated any value with fewer digits than the target exponent
straight to zero, ignoring the context's Rounding mode. RoundCeiling of a
positive sub-ULP value (e.g. 0.002 to exponent -1) returned 0.0 instead of
0.1, and RoundFloor of a negative one lost the rounding entirely.

Round the discarded coefficient through the configured Rounder instead.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RoundCeil(0.002, -1) != 0.1

1 participant