fix: Consolidate string-to-decimal parsing into a single code path - #10850
Open
neilconway wants to merge 1 commit into
Open
fix: Consolidate string-to-decimal parsing into a single code path#10850neilconway wants to merge 1 commit into
neilconway wants to merge 1 commit into
Conversation
arrow-cast previously contained two different string-to-decimal parsers: `parse_string_to_decimal_native`, used by `cast`, and `parse_decimal`, used by the CSV and JSON readers. Aside from redundancy, these code paths behaved differently (e.g., truncating vs rounding for digits beyond the target type's scale, whitespace trimming, support for e-notation, etc.), so decimal conversion behaved differently depending on how the decimal value arrived into arrow-rs. This branch replaces these two parsers with a single unified parser; `parse_decimal(s, precision_scale)` is now the public entry point and `parse_string_to_decimal_native` is a thin, deprecated wrapper over it. The new parser is based on the one-pass, u64-chunked parser in apache#10668, extended with support for e-notation and negative scales. This fixes a lot of bugs and ensures consistent behavior, but it does result in some behavior changes and a small performance regression for the CSV/JSON path; more details below. Bugs fixed: (all in the JSON/CSV path) - divide-by-zero panic or wrong values for some inputs in exponent notation (apache#10788, apache#5762) - overflow on inputs with >= 256 digits or long exponents (apache#10787) - negative scales were ignored (apache#10791) - `0e0`/`-0e0` were rejected (apache#10789) while `e5` and `-.` parsed as 0 (apache#10790) Behaviour changes: - CSV and JSON readers now round half away from zero instead of truncating digits beyond the scale (apache#9410, apache#9422, apache#7355) - `cast` from strings accepts exponent notation (apache#5068) and negative scales, which `can_cast_types` already advertised (apache#10792), and validates the target precision and scale before parsing any values - CSV and JSON readers now trim whitespace (apache#10793). Only ASCII whitespace characters are trimmed, which matches the behavior of the CSV float/int parsers; previously, the `cast` path trimmed Unicode whitespace as well, but it will no longer do so. - parse errors use `ArrowError::ParseError` with unified messages - `variant_get` validates the target precision for string inputs (apache#10794) Tests: new tests cover rounding, exponents, negative scale, whitespace, long inputs and the four widths, a seeded differential test checks 20k random inputs against a BigInt reference (num-bigint was added as a dev-dependency), and the CSV/JSON readers gain end-to-end tests for the new behavior and bugfixes listed above. Performance: - `cast` string-to-decimal: ~unchanged. The cast path already used the fast single-pass parser from apache#10668; the unified parser benchmarks within Criterion noise (~4%) of it. - CSV/JSON reader path: short inputs cost 1-2 ns more per value, while long Decimal256 inputs are ~35% faster. End-to-end, CSV reads of decimal columns are ~8% slower. I suspect there is room for further optimization here (which will now benefit both code paths!) to reach or exceed the previous performance, but I'd like to land the unified parser first before we tackle further optimizations. Closes apache#10787, closes apache#10788, closes apache#10789, closes apache#10790, closes apache#10791, closes apache#10792, closes apache#10793, closes apache#10794
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
parse_decimalnumeric overflow on long inputs #10787, closesparse_decimalpanics or wrong result value on some valid inputs with negative exponents #10788, closesparse_decimalrejects0e0#10789, closesparse_decimalallows invalid inputse5and-#10790, closesparse_decimalignores negative scales #10791, closescan_cast_typesreports string → negative-scale decimal as supported, but cast always errors #10792, closesparse_decimalrejects input with leading/trailing whitespace #10793, closesvariant_getconverts strings to decimals without checking the target precision #10794Rationale for this change
arrow-cast had two different string-to-decimal parsers:
parse_string_to_decimal_native, used bycast, andparse_decimal, used by the CSV and JSON readers. Aside from redundancy, these code paths behaved differently (e.g., truncating vs rounding for digits beyond the target type's scale, whitespace trimming, support for e-notation, etc.), so decimal conversion behaved differently depending on how the decimal value arrived into arrow-rs.This PR replaces these parsers with a single unified parser;
parse_decimal(s, precision_scale)is now the public entry point andparse_string_to_decimal_nativeis a thin, deprecated wrapper over it. The new parser is based on the one-pass, u64-chunked parser in #10668, extended with support for e-notation and negative scales. This fixes a lot of bugs and ensures consistent behavior, but it does result in some behavior changes and a small performance regression for the CSV/JSON path; more details below.Bugs fixed: (all in the JSON/CSV path)
parse_decimalpanics or wrong result value on some valid inputs with negative exponents #10788, Parsing Decimal With Negative Exponent And Zero Scale Can Overflow #5762)parse_decimalnumeric overflow on long inputs #10787)parse_decimalignores negative scales #10791)0e0/-0e0were incorrectly rejected (parse_decimalrejects0e0#10789), whilee5and-.were incorrectly accepted (parse_decimalallows invalid inputse5and-#10790)Behaviour changes:
castfrom strings accepts exponent notation (Decimal enhancements in arrow-cast #5068) and negative scales, whichcan_cast_typesalready advertised (can_cast_typesreports string → negative-scale decimal as supported, but cast always errors #10792), and validates the target precision and scale before parsing any valuesparse_decimalrejects input with leading/trailing whitespace #10793). Only ASCII whitespace characters are trimmed, which matches the behavior of the CSV float/int parsers; previously, thecastpath trimmed Unicode whitespace as well, but it will no longer do so.ArrowError::ParseErrorwith unified messagesvariant_getvalidates the target precision for string inputs (variant_getconverts strings to decimals without checking the target precision #10794)Performance:
caststring-to-decimal: ~unchanged. The cast path already used the fast single-pass parser from perf(arrow-cast): optimize parsing of decimals from strings #10668; the unified parser benchmarks within Criterion noise (~4%) of it.CSV/JSON reader path: short inputs cost 1-2 ns more per value, while long Decimal256 inputs are ~35% faster. End-to-end, CSV reads of decimal columns are ~8% slower. I suspect there is room for further optimization here (which will now benefit both code paths!) to reach or exceed the previous performance, but I'd like to land the unified parser first before we tackle further optimizations.
What changes are included in this PR?
See above.
Are these changes tested?
New tests added to cover rounding, exponents, negative scale, whitespace, long inputs and the four widths, a seeded differential test checks 20k random inputs against a BigInt reference (num-bigint was added as a dev-dependency), and the CSV/JSON readers gain end-to-end tests for the new behavior and bugfixes listed above.
Are there any user-facing changes?
Yes; a deprecated public API, and user-visible behavioral changes in decimal parsing.
AI usage
Iterated primarily with CC Fable 5; code reviewed by Codex GPT 5.6. I read, understand, and revised the resulting code.