feat: holdout support — normalized models, assignment precedence, exposure fields (ISS-61) - #12
Open
marcio-absmartly wants to merge 10 commits into
Open
feat: holdout support — normalized models, assignment precedence, exposure fields (ISS-61)#12marcio-absmartly wants to merge 10 commits into
marcio-absmartly wants to merge 10 commits into
Conversation
…osure fields, tests Squashed restore of feat/holdouts work (original commits lost to a tooling accident; content fully preserved): - ExperimentHoldout json model; ContextData.holdouts[] + Experiment.holdoutIds[] (normalized payload: definitions not duplicated per experiment) - Context: holdout check after override, before audience/traffic/variant (Override > Holdout > Audience > Traffic > Variant); id->def map with graceful skip of unknown ids; cache invalidation on holdoutIds and resolved definition changes - Exposure heldOut + holdoutId, serializer round-trip - ContextHoldoutTest matrix + serializer/deserializer coverage
Add missing exposure assertion for the not-held-out-but-has-holdouts path (heldOut=false/holdoutId=0 after holdouts are actually evaluated), plus empty-holdoutIds and empty-split branch coverage. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reuse the already-bound unitType local in the traffic-split branch instead of re-reading experiment.data.unitType, matching the new holdout branch. Behaviour-preserving. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (3)
🚧 Files skipped from review as they are similar to previous changes (1)
WalkthroughAdds experiment holdout models and context data, resolves configured holdouts, evaluates holdout membership during assignment, and records held-out state in exposures. Cache matching now includes holdout metadata. The change also adds deserialisation, assignment, precedence, refresh, full-on, publication, and serialisation coverage with supporting fixtures. Estimated code review effort: 4 (Complex) | ~60 minutes Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (2)
core-api/src/test/java/com/absmartly/sdk/ContextHoldoutTest.java (1)
223-250: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueRepeated publish-assertion boilerplate.
The
PublishEvent/when(eventHandler.publish...)/verify(...)sequence is duplicated acrossskipsHoldoutWhenUnitMissingForUnitType,exposureCarriesHeldOutAndHoldoutId, andexposureCarriesNotHeldOutFieldsWhenUnitNotInHoldout. Consider extracting a small helper (e.g.assertPublishedExposure(context, expectedExposures)) to reduce repetition.Also applies to: 358-414
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@core-api/src/test/java/com/absmartly/sdk/ContextHoldoutTest.java` around lines 223 - 250, The publish assertion setup is duplicated in ContextHoldoutTest across multiple holdout tests, including skipsHoldoutWhenUnitMissingForUnitType, exposureCarriesHeldOutAndHoldoutId, and exposureCarriesNotHeldOutFieldsWhenUnitNotInHoldout. Extract the repeated PublishEvent construction, when(eventHandler.publish(...)) stubbing, and verify(...) timeout assertion into a small helper such as assertPublishedExposure or assertPublishEvent so the tests stay focused on the scenario-specific expectations.core-api/src/main/java/com/absmartly/sdk/Context.java (1)
786-865: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winConsider de-duplicating the unit lookup in the holdout/traffic-split branches.
The
uid = units_.get(unitType)lookup (plus the follow-ongetUnitHash/getVariantAssignercalls) is repeated once for holdout evaluation and again for traffic-split evaluation. Hoisting theuidlookup once at the top of this block would remove the duplication and slightly reduce the complexity of an already dense method (holdout precedence, audience matching, full-on/traffic logic are all interleaved here).♻️ Suggested simplification
if (experiment != null) { final String unitType = experiment.data.unitType; + final String uid = units_.get(unitType); assignment.holdoutIds = experiment.data.holdoutIds; assignment.holdouts = experiment.holdouts; - if (experiment.holdouts != null && experiment.holdouts.length > 0) { - final String uid = units_.get(unitType); - if (uid != null) { - final byte[] unitHash = Context.this.getUnitHash(unitType, uid); - final VariantAssigner assigner = Context.this.getVariantAssigner(unitType, unitHash); - for (final ExperimentHoldout holdout : experiment.holdouts) { + if (uid != null && experiment.holdouts != null && experiment.holdouts.length > 0) { + final byte[] unitHash = Context.this.getUnitHash(unitType, uid); + final VariantAssigner assigner = Context.this.getVariantAssigner(unitType, unitHash); + for (final ExperimentHoldout holdout : experiment.holdouts) { if (assigner.assign(holdout.split, holdout.seedHi, holdout.seedLo) == 0) { assignment.heldOut = true; assignment.holdoutId = holdout.id; assignment.variant = 0; assignment.assigned = true; break; } - } - } + } } if (!assignment.heldOut) { ... } else if (experiment.data.fullOnVariant == 0) { - final String uid = units_.get(unitType); if (uid != null) { final byte[] unitHash = Context.this.getUnitHash(unitType, uid); final VariantAssigner assigner = Context.this.getVariantAssigner(unitType, unitHash); ... } } else { ... } } ... }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@core-api/src/main/java/com/absmartly/sdk/Context.java` around lines 786 - 865, The unit lookup and assigner setup in the experiment assignment flow is duplicated in both the holdout and traffic-split branches. In Context’s assignment block, hoist the units_.get(unitType) lookup (and, if available, the derived getUnitHash/getVariantAssigner setup) once before the conditional branches, then reuse it for holdout and eligibility evaluation. Keep the existing holdout precedence and audience/full-on behavior unchanged while simplifying the repeated logic.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@core-api/src/main/java/com/absmartly/sdk/Context.java`:
- Around line 786-865: The unit lookup and assigner setup in the experiment
assignment flow is duplicated in both the holdout and traffic-split branches. In
Context’s assignment block, hoist the units_.get(unitType) lookup (and, if
available, the derived getUnitHash/getVariantAssigner setup) once before the
conditional branches, then reuse it for holdout and eligibility evaluation. Keep
the existing holdout precedence and audience/full-on behavior unchanged while
simplifying the repeated logic.
In `@core-api/src/test/java/com/absmartly/sdk/ContextHoldoutTest.java`:
- Around line 223-250: The publish assertion setup is duplicated in
ContextHoldoutTest across multiple holdout tests, including
skipsHoldoutWhenUnitMissingForUnitType, exposureCarriesHeldOutAndHoldoutId, and
exposureCarriesNotHeldOutFieldsWhenUnitNotInHoldout. Extract the repeated
PublishEvent construction, when(eventHandler.publish(...)) stubbing, and
verify(...) timeout assertion into a small helper such as
assertPublishedExposure or assertPublishEvent so the tests stay focused on the
scenario-specific expectations.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 00fd9122-82f1-4830-b808-f01c239e7192
📒 Files selected for processing (12)
core-api/src/main/java/com/absmartly/sdk/Context.javacore-api/src/main/java/com/absmartly/sdk/json/ContextData.javacore-api/src/main/java/com/absmartly/sdk/json/Experiment.javacore-api/src/main/java/com/absmartly/sdk/json/ExperimentHoldout.javacore-api/src/main/java/com/absmartly/sdk/json/Exposure.javacore-api/src/test/java/com/absmartly/sdk/ContextHoldoutTest.javacore-api/src/test/java/com/absmartly/sdk/ContextTest.javacore-api/src/test/java/com/absmartly/sdk/DefaultContextDataDeserializerTest.javacore-api/src/test/java/com/absmartly/sdk/DefaultContextEventSerializerTest.javacore-api/src/test/java/com/absmartly/sdk/json/ContextDataTest.javacore-api/src/test/java/com/absmartly/sdk/json/ExperimentHoldoutTest.javacore-api/src/test/resources/holdouts_context.json
- Assignment cache: treat held-out assignments as up-to-date even when a custom assignment is set. Holdout wins over custom assignment, so the cached variant (0) can never equal the custom variant; the old condition forced a cache miss and a fresh Assignment (exposed=false) on every call, queueing a duplicate exposure per getTreatment. - Tests: cached held-out assignment with custom assignment set; holdout precedence over fullOnVariant. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
When a custom assignment is set but the cached variant was forced by a higher-precedence rule (holdout, full-on, traffic ineligibility, or a strict audience mismatch), the custom value can never equal that variant. The old cache-validity check compared them directly, so it forced a cache miss and a fresh Assignment (exposed=false) on every getTreatment call, queueing a duplicate exposure each time. Introduce variantForcedRegardlessOfCustom() covering all forced-variant cases and treat them as cache-valid. 5af036a fixed the holdout case only; this generalizes the same fix to the audience-strict-mismatch and traffic-ineligible cases, which are pre-existing on main. Tests: cached forced-variant assignments with a custom assignment set for both the strict audience mismatch and traffic ineligible cases (stable pending count across repeated calls). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Assignment.holdoutIds/holdouts alias the arrays owned by ContextExperiment instead of copying them. This is safe only because experiment data is treated as immutable after setData and the arrays are read-only here. Note the invariant at the assignment site. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ash parity The gpt-5.6-sol pass changed the SDK normalizer to 1.0/0x100000000 to make the probability range half-open [0,1). But the collector (and every other ABsmartly SDK) uses 1.0/0xffffffff, where the max 32-bit hash maps to exactly 1.0. Assignment requires bit-exact hash parity between the SDK and the collector -- diverging the normalizer would make the SDK and server disagree on borderline units (including holdout membership). Reverted VariantAssigner to match main/collector; removed the range test that asserted the divergent behaviour.
Add optional fullOn field to ExperimentHoldout (tolerant, defaults to null/absent = full holdout, unchanged behavior). A full_on holdout only applies to full-on experiments (fullOnVariant != 0); for other experiments the SDK skips it entirely during heldOut evaluation, matching the collector's server-side behavior bit-exactly.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Client-side holdout support (ISS-61, Phase 3) — companion to absmartly/abs#4539 (server side).
ContextData.holdouts[](definitions:{id, seedHi, seedLo, split}) +Experiment.holdoutIds[]references — definitions are not duplicated per experiment.assigned=true; overrides bypass holdouts (debugging escape hatch).VariantAssignerwith the holdout's own seed — same verdict across all experiments referencing the holdout.holdoutIdsor a referenced definition (seed/split) invalidates cached assignments.heldOut+holdoutIdfor downstream analysis.holdoutIdsand malformed definitions (null/short split) are skipped gracefully.Verification
:core-api:test), findbugsMain cleanNotes
Summary by CodeRabbit
New Features
Bug Fixes
Tests