Skip to content

feat(value): introduce Array storage abstraction - #63

Open
anakrish wants to merge 8 commits into
mainfrom
storage-abstraction-array-foundation
Open

feat(value): introduce Array storage abstraction#63
anakrish wants to merge 8 commits into
mainfrom
storage-abstraction-array-foundation

Conversation

@anakrish

@anakrish anakrish commented Jun 4, 2026

Copy link
Copy Markdown
Owner

Adds an opaque Array newtype paralleling Object, living under src/value/array/ with the same module structure as src/value/object/ (mod.rs / iter.rs / serde.rs).

Array wraps Vec<Value> today but exposes only a curated surface: constructors, indexed access, first / last, push / pop, insert / remove, truncate, sort / sort_by, dedup, extend_from_slice, reverse, iter / iter_mut, cursor (resumable), serde, conversions to Value, and a hand-written lexicographic Ord. The cursor type is re-exported behind the rvm feature so the follow-up IterationState::Array swap can land additively.

No crate-root Array re-export is added; the new type lives under regorus::value::Array, matching the storage-abstraction namespace strategy for Object and Set.

Value::Array is unchanged in this PR (still Rc<Vec<Value>>); the payload swap and call-site migration ship in the next PR.

Tests live in src/value/tests.rs alongside the Object tests. This also repairs pre-existing doctest examples exercised by cargo test --doc --all-features, so the full requested validation suite passes.

@anakrish
anakrish force-pushed the storage-abstraction-array-foundation branch from a76f04a to 2fe9ec2 Compare June 4, 2026 19:15
@anakrish
anakrish requested a review from Copilot June 4, 2026 19:41
@anakrish

anakrish commented Jun 4, 2026

Copy link
Copy Markdown
Owner Author

@copilot Review this PR using all the review skills in this repo. Use a separate agent for each skill.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Introduces a new value::Array storage abstraction (parallel to Object) with opaque iterator/cursor types, serde support, and an explicit lexicographic Ord, plus associated unit tests and several doctest fixes across the crate.

Changes:

  • Added src/value/array/ module implementing Array, iterator newtypes, (feature-gated) cursor, and serde impls.
  • Added comprehensive Array tests to src/value/tests.rs.
  • Repaired multiple doctest imports/paths and an example target-loading snippet to keep cargo test --doc --all-features working.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
src/value/tests.rs Adds unit tests covering Array constructors, mutation APIs, iteration, cursor (rvm), ordering, conversions, and serde.
src/value/mod.rs Wires in the new array module and re-exports Array.
src/value/array/mod.rs Core Array type and API surface, including ordering, indexing, and cursor iteration.
src/value/array/iter.rs Opaque iterator newtypes and IntoIterator impls for Array and references.
src/value/array/serde.rs Serialize/Deserialize implementations with allocator limit checks during decode.
src/schema/validate.rs Updates doctest imports to use crate-root re-exports.
src/schema.rs Adds missing doctest imports and updates example use paths.
src/engine.rs Fixes a doctest indexing path into the AST JSON.
src/compiled_policy.rs Fixes doctest target-loading by reading JSON then calling from_json_str.
docs/value/array.md Adds documentation describing the new Array abstraction and its design goals.

Comment thread src/value/array/mod.rs
Comment thread src/value/array/mod.rs

Copilot AI commented Jun 4, 2026

Copy link
Copy Markdown

@copilot Review this PR using all the review skills in this repo. Use a separate agent for each skill.

Here are the results from both review skills run in parallel:


code-review skill — 2 findings (0 critical, 1 high, 1 medium)

HIGH | High confidence

Location: src/value/array/mod.rs:117
Issue: Array::remove, Array::insert, and Index<usize> all panic on out-of-bounds input — violating regorus's no-panic rule.

pub fn remove(&mut self, index: usize) -> Value { self.inner.remove(index) }
pub fn insert(&mut self, index: usize, value: Value) { self.inner.insert(index, value); }
fn index(&self, index: usize) -> &Self::Output { ops::Index::index(self.inner.as_slice(), index) }

Any out-of-bounds call panics. If Array reaches a FFI path this poisons the whole process.
Fix: Make these fallible (Option/Result), matching Object::remove's convention. Avoid publicly exposing Index<usize>.


MEDIUM | Medium confidence

Location: src/value/array/mod.rs:68
Issue: as_slice(), get_mut(), iter_mut(), and Index hand out Rust references into contiguous memory, already committing to Vec-specific semantics before Value::Array migrates.

Any future lazy/FFI-backed backend would need to fully materialize to satisfy these contracts, defeating the stated "no call-site changes" abstraction goal.
Fix: Defer as_slice, mutable borrows, and Index until the backend contract is settled.


deep-review skill — 7 findings (0 critical, 2 high, 3 medium, 2 low)

HIGH | High confidence — Panic-on-invalid-index (confirmed)

Same as above, confirmed by multiple agents and adversarial verifier.


HIGH | Medium confidence — Array growth bypasses memory-limit checks

Location: src/value/array/mod.rs:46–48, 102–151, 222–239

push, extend_from_slice, and with_capacity allocate directly through Vec with no enforce_limit() / check_memory_limit_if_needed() calls. The crate-wide accumulation pattern requires these guards; the only limit-aware path in this PR is Deserialize.

pub fn push(&mut self, value: Value) { self.inner.push(value); }
pub fn extend_from_slice(&mut self, other: &[Value]) { self.inner.extend_from_slice(other); }

Fix: Either add checked growth helpers or document that Array is not limit-enforcing outside serde/evaluation code.


MEDIUM | High confidence — remove freezes a panic-based semver contract

Location: src/value/array/mod.rs:122

Object::remove returns Option<Value>; Array::remove returns Value directly. Since the type is already public, changing this later is a breaking change.
Fix: Decide the API contract now; prefer Option<Value> for consistency with Object.


MEDIUM | High confidence — Serde is lossy for non-JSON-native Value variants (confirmed)

Location: src/value/array/serde.rs:15–18, 30–43

Arrays containing Undefined, Set, or objects with non-string keys don't round-trip correctly through JSON serde:

  • [Undefined]["<undefined>"][String("<undefined>")]
  • [Set({1,2})][[1,2]][Array([1,2])]

This is inherited from existing Value serde but may warrant documentation or a rejected-variant guard.


MEDIUM | Medium confidence — Public surface leaks Vec backend

Location: src/value/array/mod.rs:82–90, 159–163, 213–219

as_slice() and into_value() → Value::Array(Rc::new(self.inner)) make the current Vec-backed representation directly observable; tests also assert Value::Array(crate::Rc::new(vec![...])) directly, baking in the backend.


LOW | High confidence — No test coverage for invalid-index contract

Location: src/value/tests.rs:591

array_mutators_and_accessors only exercises valid indexes. Add a #[should_panic] test (or a test for fallible return if the API is changed) to document the intended contract.


LOW | Medium confidence — Test edge-case and abstraction gaps

Location: src/value/tests.rs:574, 680

with_capacity is only tested with 2; conversion assertions compare against the concrete Rc<Vec<_>> representation rather than public semantics. Add a limit-boundary test and rewrite conversion assertions through Value equality or public accessors.


Combined summary

Both skills independently identified the same two highest-priority issues: panic-on-bad-index in public mutating APIs and premature commitment to Vec semantics that undermines the abstraction goal. The deep-review additionally flagged missing enforce_limit() calls on growth paths (a crate-wide convention) and Object-inconsistent remove return type. These are the four items I'd prioritize before the payload-swap follow-up PR lands.

@anakrish
anakrish force-pushed the storage-abstraction-array-foundation branch from 872e582 to 05a1592 Compare June 4, 2026 20:30
@anakrish
anakrish requested a review from Copilot June 4, 2026 21:00
@anakrish

anakrish commented Jun 4, 2026

Copy link
Copy Markdown
Owner Author

@copilot Review this PR using the skills in the repo. Use a separate agent for each skill.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.

Comment thread docs/value/array.md Outdated
Comment on lines +9 to +13
`Array` wraps a `Vec<Value>` today but exposes only a curated method surface
(`get`, `get_mut`, `push`, `pop`, `insert`, `remove`, `iter`, `iter_mut`,
`cursor`, serde). The inner vector is private — callers cannot pattern-match it
or hand out mutable references to the backing store, so the backend can change
without churn at call sites that currently assume `Vec<Value>`.

Copilot AI commented Jun 4, 2026

Copy link
Copy Markdown

@copilot Review this PR using the skills in the repo. Use a separate agent for each skill.

Completed. I ran separate agents for both repo review skills (code-review and deep-review) and consolidated the results.

Validated findings:

  • src/value/array/mod.rs: Array::with_capacity can panic on oversized capacities (confirmed).
  • docs/value/array.md: docs list cursor unconditionally, but cursor/next are #[cfg(feature = "rvm")] (confirmed).
  • src/value/array/mod.rs: growth helpers (push/extend_from_slice/Extend) currently rely on caller-side limit enforcement (likely).
  • src/value/array/mod.rs: FromIterator<Value> likely shares the same overflow-panic class as capacity construction.

Dropped during adversarial verification:

  • Undefined serde roundtrip concern is pre-existing in Value serde, not introduced by this PR.
  • dual export path (pub mod array + pub use array::Array) treated as intentional API choice.

kusha and others added 3 commits June 5, 2026 16:00
…rosoft#742)

`run_opa_tests` builds `path_dir_str` from `path.strip_prefix(...).to_string_lossy()`,
which on Windows yields strings with backslash separators (e.g.
`v0\aggregates`). The folder filter then does an exact-string
comparison against the CLI arguments:

    let run_test = folders.is_empty()
        || folders.iter().any(|f| &path_dir_str == f);

CLI arguments use forward slashes (`v0/aggregates`), so on Windows
the comparison never matches, no tests are selected, and the function
bails with `"no matching tests found"`. This blocks the
`cargo xtask pre-push` hook for any Windows contributor.

Normalize `path_dir_str` to use forward slashes at construction
time. Reproduces before the fix as `cargo test ... --test opa --
v1/aggregates` exiting 1 with `no matching tests found`; after the
fix the same command runs 72 cases and the full hook command runs
2861 / 0 across 188 folders.

The duplicate platform check at the `is_rego_v0_test` site
(`path_dir_str.starts_with("v0/") || path_dir.starts_with("v0\\")`)
is left intact to keep the change minimal — the backslash branch
becomes redundant but is harmless.

Co-authored-by: Mark Birger <markbirger@microsoft.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…icrosoft#736)

Builds on #57. Swap Value::Object's payload from Rc<BTreeMap<Value, Value>>
to Rc<Object> and migrate all call sites to the Object API.

as_object / as_object_mut keep their names but return &Object / &mut Object.
The mutable accessor handles Rc::make_mut internally, so callers no longer
do it themselves. Object grows into_value() and From<Object> for Value.
Value's serializer now delegates to Object::serialize, dropping a duplicate
non-string-key stringification path.

RVM IterationState::Object is rewritten around ObjectCursor: O(log n)
steps over a shared Rc<Object>, no eager pair snapshot. Snapshot
independence is preserved by Rc copy-on-write; setup_next_iteration
advances the cursor inline and advance() becomes a no-op for this variant.
A new iteration_state_object_is_snapshot_independent_of_source test
covers CoW against a mutated alias.

Value::Set still wraps Rc<BTreeSet<Value>>; the matching Set abstraction
and its swap ship in follow-up PRs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add an opaque `Array` newtype paralleling `Object`, living under
`src/value/array/` with the same module structure (`mod.rs` /
`iter.rs` / `serde.rs`). `Array` wraps `Vec<Value>` today but exposes
only a curated surface: constructors, indexed access, push/pop/insert/
remove/truncate/sort/dedup/reverse, iteration, serde, conversion to
`Value`, and a hand-written lexicographic `Ord`. The cursor type is
re-exported behind the `rvm` feature so the follow-up
`IterationState::Array` swap can land additively.

No crate-root `Array` re-export is added; the new type lives under
`regorus::value::Array`, matching the storage-abstraction namespace
strategy for `Object` and `Set`.

`Value::Array` is unchanged in this commit (still wraps
`Rc<Vec<Value>>`); the payload swap and call-site migration ship in the
next PR.

Also repair pre-existing doctest examples that `cargo test --doc
--all-features` exercises, so the required validation suite passes with
this branch.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@anakrish
anakrish force-pushed the storage-abstraction-array-foundation branch from 05a1592 to 81ae161 Compare June 6, 2026 12:58
@anakrish
anakrish requested a review from Copilot June 8, 2026 14:25
anakrish and others added 4 commits June 8, 2026 09:27
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@anakrish

anakrish commented Jun 8, 2026

Copy link
Copy Markdown
Owner Author

Addressed the outstanding Array foundation review items:

  • Updated docs/value/array.md to accurately describe the public API, rvm-gated cursor/next, and Vec-like surfaces that future backends may materialize.
  • Made Array::with_capacity checked via try_reserve and return Option for oversized/unreservable capacities.
  • Added memory-limit checks to Array growth paths: push/extend_from_slice now return Result; Extend/FromIterator route through the checked path.
  • Added capacity boundary/overflow coverage and removed tests that asserted Value::Array with Rc<Vec<_>> directly.

Commits:

  • 84d9b53 docs(value/array): accurate API surface and feature gating
  • 87b26c3 feat(value/array): checked with_capacity
  • ebc5165 feat(value/array): enforce memory limit on growth paths
  • 25ba655 test(value/array): cover overflow + remove Vec-backend assertions

Intentionally deferred: no Value::Array migration or removal of ergonomic Vec-like APIs in this foundation PR; the docs now call out that future backends may need to materialize those views.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 39 out of 39 changed files in this pull request and generated 1 comment.

Comment thread src/value/mod.rs Outdated
clippy::as_conversions
)] // value helpers index paths directly for performance

pub mod array;
@anakrish

anakrish commented Jun 8, 2026

Copy link
Copy Markdown
Owner Author

@copilot Review this PR using all the review skills in this repo. Launch a separate agent for each skill.

Copilot AI commented Jun 8, 2026

Copy link
Copy Markdown

@copilot Review this PR using all the review skills in this repo. Launch a separate agent for each skill.

Ran both review skills with separate agents (code-review and deep-review) and consolidated results.

Findings:

  1. CriticalArray::Extend/FromIterator can abort the process on recoverable memory-limit errors (src/value/array/mod.rs:259-277).
  2. HighArray::insert can grow without memory-limit enforcement (src/value/array/mod.rs:132-137).
  3. Mediumpush/extend_from_slice can return Err after mutation (partial writes) (src/value/array/mod.rs:112-115,180-183).
  4. Mediumwith_capacity reserves without memory-limit enforcement (src/value/array/mod.rs:52-55).
  5. MediumFrom<Vec<Value>> for Array bypasses cooperative limit checks (src/value/array/mod.rs:281-285).
  6. Medium (API risk)pub mod array broadens semver surface (src/value/mod.rs:14-23).

Also noted test gaps for the memory-limit/error-path cases above in src/value/tests.rs.

Addresses follow-up review on PR #63 plus our own strict re-review:

- Drop the process-aborting Extend/FromIterator impls. They now match the
  Object convention: delegate directly to the inner container without
  consulting the cooperative memory limit (rely on the allocator to fail
  on true OOM). Limit-aware bulk insertion has explicit fallible methods.
- Add Array::try_from_vec and Array::try_from_iter as the limit-aware
  alternatives to the From<Vec<Value>> / FromIterator pairs.
- with_capacity now also consults check_memory_limit_if_needed after
  the inner try_reserve so capacity reservations cannot silently push
  usage past the cooperative ceiling.
- insert now returns Result<()>: out-of-bounds and limit failures both
  flow through the same anyhow error channel. The limit case rolls back
  the inserted element so the array length is unchanged.
- push and extend_from_slice roll back the pushed element(s) on a limit
  failure rather than leaving the array in a half-mutated state.
- Make the array module itself private (was `pub mod array`); only
  `Array` (always) and `ArrayCursor` (rvm feature) are re-exported from
  `crate::value`. This matches the Object/ObjectCursor pattern and keeps
  the module layout out of the public API surface.
- Rename the rvm-only Cursor type to ArrayCursor for symmetry with
  ObjectCursor and to disambiguate at the value-module level.
- Update docs/value/array.md to reflect the private module, the
  ArrayCursor name, and the limit-enforcement contract.
- Extend the test suite with try_from_vec, try_from_iter, and an
  insert out-of-bounds error case. Update existing insert call sites
  to the Result API.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@anakrish

anakrish commented Jun 8, 2026

Copy link
Copy Markdown
Owner Author

Addressed the follow-up review pass (Copilot consolidated findings + line comment on pub mod array) plus our own strict re-review.

Memory-limit correctness

  • Dropped the process-aborting Extend/FromIterator impls; they now match the Object convention (delegate to inner container, no per-element cooperative-limit check). Limit-aware bulk insertion now goes through explicit fallible APIs.
  • Added Array::try_from_vec and Array::try_from_iter as limit-aware counterparts to From<Vec<Value>> / FromIterator.
  • with_capacity now also calls check_memory_limit_if_needed after try_reserve so capacity reservations cannot silently cross the cap.
  • insert now returns Result<()>; bounds and limit failures both flow through anyhow. The limit case rolls back the inserted element.
  • push and extend_from_slice roll back partial writes on limit failure so the array never observes a half-mutated state.

API surface

  • Made array module private (was pub mod). Only Array (always) and ArrayCursor (rvm feature) are re-exported from crate::value — matching the Object / ObjectCursor pattern, no longer locking the module layout into the public API.
  • Renamed the rvm-only Cursor to ArrayCursor for symmetry with ObjectCursor and to disambiguate at the value-module level.

Docs + tests

  • Updated docs/value/array.md to reflect the private module, ArrayCursor rename, and the limit-enforcement contract (which methods enforce vs bypass).
  • Added try_from_vec / try_from_iter happy-path tests and an out-of-bounds insert error test. Existing insert call sites migrated to the Result API.

Commit: 93216b2 fix(value/array): tighten memory-limit & API-surface review findings

Gates green: fmt, clippy (default + all-features + wasm bindings), 627 lib tests with --all-features, 242 lib tests default, 107 doc tests.

@anakrish

anakrish commented Jun 8, 2026

Copy link
Copy Markdown
Owner Author

@copilot Review this PR using all the review skills in this repo. Launch a separate agent for each skill.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 39 out of 39 changed files in this pull request and generated no new comments.

Copilot AI commented Jun 8, 2026

Copy link
Copy Markdown

@copilot Review this PR using all the review skills in this repo. Launch a separate agent for each skill.

Completed. I ran separate agents for both review skills (code-review and deep-review) and consolidated the outputs.

Findings:

  • code-review: 1 finding

    • High — memory-limit rollback in Array mutators restores length but may retain grown capacity (src/value/array/mod.rs).
  • deep-review: 7 findings

    • High — throttled check_memory_limit_if_needed() can allow over-limit Array mutations/serde to return success (src/value/array/mod.rs, src/value/array/serde.rs).
    • Mediumwith_capacity reserves before limit check (src/value/array/mod.rs).
    • Mediumiter/iter_mut return concrete iterator types that are not re-exported/nameable (src/value/array/mod.rs, src/value/mod.rs).
    • Medium (likely) — rollback may leave elevated capacity.
    • Low — three test gaps around limit/throttle semantics and iterator API exposure.

CI signal check: workflow run 27162681343 reports no failed jobs/logs (failed_jobs: 0).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants