Skip to content

refactor: harden library usability before 0.5.0 - #253

Merged
tisonkun merged 7 commits into
apache:mainfrom
tisonkun:codex/prepare-0.5.0
Aug 31, 2026
Merged

refactor: harden library usability before 0.5.0#253
tisonkun merged 7 commits into
apache:mainfrom
tisonkun:codex/prepare-0.5.0

Conversation

@tisonkun

@tisonkun tisonkun commented Aug 31, 2026

Copy link
Copy Markdown
Member

Summary

This PR addresses a small set of release-facing issues found during a pre-0.5.0 audit:

  • make the new REQ iterator actually zero-allocation by borrowing retained items instead of cloning them;
  • use the existing NumStdDev type for REQ rank confidence bounds so unsupported confidence levels are not representable;
  • reject oversized encoded string lengths before FrequentItemsSketch<String> allocates from untrusted input;
  • reject REQ images whose serialized extrema contradict their retained items;
  • document the panic contracts of mutable T-Digest queries at their own call sites;
  • remove two impossible signed-count assertions from unsigned frequent-item updates.

REQ is still unreleased, so tightening its iterator and confidence-bound APIs here avoids carrying immediately regrettable interfaces into 0.5.0.

Audit boundary

The audit intentionally excluded changes without a concrete failure mode or an established contract. In particular:

  • Count-Min bound semantics were left unchanged after checking them against the Apache Java implementation.
  • Count overflow behavior was left unchanged because saturation would alter merge semantics and the crate has no agreed overflow contract.
  • REQ deep-size accounting was not added while the generic-value accounting question remains open in Define deep estimated-size accounting for generic sketch values #193.
  • Existing tests were not removed merely for looking repetitive; the relevant suites are fast and exercise observable behavior boundaries.
  • Open feature work such as Display implementations and KLL was not folded into this release-hardening PR.
  • REQ deserialization hardening is not listed as a separate changelog item because REQ is new since 0.4.0; the release notes describe its final 0.5.0 contract instead of internal development history.

Commit structure

Each concern is isolated in a reviewable commit whose message records the reason for the change:

  1. refactor(req): borrow items from the public iterator
  2. fix(frequencies): validate string lengths before allocation
  3. fix(req): validate serialized extrema against retained items
  4. docs(tdigest): state mutable query panic contracts
  5. refactor(req): type the confidence level parameter
  6. refactor(frequencies): remove impossible count assertions
  7. docs(changelog): omit unreleased REQ hardening

Validation

  • cargo x prepare-testdata
  • cargo x check
  • cargo x test
  • cargo x lint
  • cargo package --list -p datasketches
  • cargo publish --dry-run --locked -p datasketches

The publish dry-run packaged 99 files (1.9 MiB, 274.9 KiB compressed) and verified the package with Rust/Cargo 1.86.0.

Returning owned items cloned every retained value while the iterator was
documented as zero-allocation. Borrow the items from the sketch so callers
can inspect heap-backed values without hidden cloning or allocation.
String lengths are read from untrusted sketch images. Check the declared
length against the remaining payload before allocating so a four-byte length
field cannot trigger an allocation much larger than the supplied input.
REQ estimation images store stream extrema separately from retained items.
Reject images whose extrema do not bound those items so min_item and max_item
cannot expose values contradicted by the sketch state.
The mutable query methods perform the same input assertions as TDigest, but
their own API docs omitted those contracts. Document them at the call sites so
users do not need to follow cross-links to discover when a query can panic.
REQ confidence bounds support only one, two, or three standard deviations, but
a u8 accepted every value and differed from the rest of the crate. Reuse
NumStdDev so invalid confidence levels are unrepresentable at the call site.
Update counts are unsigned and zero already returns early, so the following
"not negative" assertions were always true. Remove the dead checks and their
misleading signed-count error message.
REQ is new since the latest release, so its deserialization fix is part of the
initial 0.5.0 contract rather than a user-visible change between releases.
Keep the changelog focused on the feature's final behavior.
@tisonkun
tisonkun marked this pull request as ready for review August 31, 2026 12:06
@tisonkun
tisonkun enabled auto-merge (squash) August 31, 2026 12:07
@tisonkun
tisonkun merged commit 292000d into apache:main Aug 31, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant