Skip to content

Refresh SizeMeasure support and make archived row decoding fallible - #51

Merged
pathscale merged 4 commits into
mainfrom
size-measure-update
Aug 3, 2026
Merged

Refresh SizeMeasure support and make archived row decoding fallible#51
pathscale merged 4 commits into
mainfrom
size-measure-update

Conversation

@Handy-caT

@Handy-caT Handy-caT commented Aug 13, 2025

Copy link
Copy Markdown
Collaborator

Refreshes the original SizeMeasure work onto current main, modernizes dependencies, and closes the archived-row safe-API soundness issue.

Changes

  • adds an overridable SizeMeasurable::default_aligned_size type-level helper
  • adds generic primitive-array sizing while preserving the existing array alignment contract
  • adds SizeMeasure support for non-generic fieldless enums, measured from the explicitly named rkyv archived representation
  • rejects payload-bearing enums instead of guessing a data-dependent size
  • handles struct and enum macro generation through the same syn::Result path
  • updates DataBucket to 0.5.0 and data_bucket_derive to 0.3.17
  • updates WorkTablesIndex to 0.0.3 and declares rkyv 0.8.17 as the minimum
  • removes the unused rkyv 0.7 and other unused dependencies from the proc-macro crate
  • replaces archived-row raw-pointer walking with bounds-checked offsets and unconditional rkyv validation
  • returns structured DataDecodeError values for unsupported schemas and short, malformed, or misaligned archives
  • documents and tests why field validation retains the complete serialized prefix for out-of-line relative pointers

Breaking API and versioning

DataBucket is bumped to 0.5.0 because this soundness correction intentionally changes public decoding contracts:

  • parse_archived_row now returns Result<Vec, DataDecodeError>
  • DataType::from_archived_bytes returns Result
  • the legacy safe raw-pointer DataType methods are removed
  • DataIterator items carry DataDecodeError

The derive crate remains 0.3.17 because its public changes are additive and its generated-code correction is compatible.

Persistence compatibility

This PR does not intentionally change the archive or page format. Generic primitive arrays deliberately retain align() == None, matching the existing u8 array implementations, and regression tests cover the old 20-byte and 32-byte behavior. rkyv guarantees format compatibility across the 0.8 release line.

Validation

  • cargo build --verbose
  • cargo test --verbose: 49 DataBucket tests passed
  • cargo test -p data_bucket_derive
  • cargo test -p data_bucket --no-default-features
  • cargo clippy --all-targets --all-features -- -D warnings
  • data_bucket_derive 0.3.17 publish dry-run
  • WorkTable 0.9.4 all-target/all-feature downstream check with DataBucket 0.5.0 and WorkTablesIndex 0.0.3
  • web3.trading-backend downstream check
  • api.support.cafe workspace/all-target check with s3-sync enabled

The DataBucket package dry-run must follow publication of its exact data_bucket_derive 0.3.17 dependency. Live api.support.cafe S3 validation is intentionally deferred.

Release order after merge: data_bucket_derive 0.3.17, then data_bucket 0.5.0.

Fixes #61

@Handy-caT Handy-caT self-assigned this Aug 13, 2025
@pathscale
pathscale force-pushed the size-measure-update branch from e979130 to 78c9ca7 Compare August 3, 2026 06:04
@pathscale pathscale changed the title SizeMeasure update Refresh SizeMeasure support and harden archived row parsing Aug 3, 2026
@pathscale pathscale changed the title Refresh SizeMeasure support and harden archived row parsing Refresh SizeMeasure support and make archived row decoding fallible Aug 3, 2026
@pathscale

Copy link
Copy Markdown
Owner

Disposition of the latest blocker review:

  1. codegen/src/size_measure/mod.rs: the original branches already compiled because ? unwraps the enum Result to TokenStream, but the inconsistency was valid maintenance feedback. Struct generation now also returns syn::Result<TokenStream> and both branches propagate with ?.
  2. enum_generator.rs: the emitted associated-type syntax was already compile-tested successfully. It now names <Enum as rkyv::Archive>::Archived explicitly and adds an explicit Archived: Sized bound. The integration derive test remains green.
  3. archived-row slice anchoring: &buf[..field_end] is intentional. rkyv::access roots the field at the slice end, while the complete prefix is required for out-of-line string payloads serialized before the fixed-width row root. This is now documented inline and covered by a primitive-followed-by-long-string archive test.
  4. panic-based decoding: removed. parse_archived_row and DataType::from_archived_bytes now return structured DataDecodeError values. Safe raw-pointer trait methods were removed. DataIterator propagates decode errors.
  5. versioning: the public decoding change is breaking, so DataBucket is now 0.5.0. data_bucket_derive remains 0.3.17.
  6. blanket DefaultSizeMeasurable: removed. default_aligned_size is now a default method on SizeMeasurable, so each concrete implementation may override it without coherence conflicts. A regression test uses a type whose Default panics and proves the override does not construct it.
  7. heavy default performance: the docs state the default construction behavior and override contract. Empty String and Vec defaults do not allocate; custom expensive defaults can override directly.
  8. panic tests: converted to explicit Result/error assertions for short buffers, malformed relative pointers, misalignment, and unsupported schema types.
  9. dead pointer helpers: removed; repository search finds no remaining raw-pointer decode helper use.

Validation is listed in the PR body. WorkTable, web3.trading-backend, and api.support.cafe compile against the DataBucket 0.5.0 dependency chain. Live S3 validation remains deferred by request.

@pathscale
pathscale merged commit 8408f0a into main Aug 3, 2026
2 checks passed
@pathscale
pathscale deleted the size-measure-update branch August 3, 2026 06:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Potential unsoundness: OOB

2 participants