fix: order strings byte-for-byte in the VM and search generators - #59
Open
junichi-cstk wants to merge 1 commit into
Open
fix: order strings byte-for-byte in the VM and search generators#59junichi-cstk wants to merge 1 commit into
junichi-cstk wants to merge 1 commit into
Conversation
An audience filtering `country < "Argentina"` matched an entity with
country "Albania" in Elasticsearch and matched nobody in the VM. Three
engines implemented ordering against a string column three ways:
ES {"range":{"plain_country":{"lt":"Argentina"}}} on a keyword
field, so a byte-order range -> matches
VM operateStrings had no ordering case, returned an ErrorValue
-> always false, for every entity
bleve NumericRangeQuery whose int/float type switch matched no
string bound, leaving both bounds nil -> predicate dropped
`<` and `>=` were both false in the VM, so the two halves were not
complementary -- the tell that it was an error result rather than a
comparison.
Make all three compare the way the keyword index already does:
- operateStrings handles <, <=, > and >=; the StringsValue case gets
the same operators with any-element semantics, matching how an index
evaluates a term range over a multi-valued field.
- walkTernary gains a StringValue BETWEEN branch, exclusive on both
ends like the numeric branches and like the gt/lt pair esgen emits.
- makeRange and coerceScalar (which feeds makeBetween) stop coercing
the literal for string-typed columns. This was not cosmetic: an ISO
bound became epoch millis, so ES compared the keyword "2026-05-09"
against "1778284800000" and every term sorted greater.
- blevegen emits term ranges for string columns in both makeRange and
makeBetween.
Every new test was confirmed failing against the pre-fix code with
`go test -overlay`. The bleve tests search a real in-memory index with
a keyword mapping rather than asserting query shape; pre-fix they
returned 0 hits for every ordering and BETWEEN case. Full suite green.
Two pre-existing inconsistencies left alone: blevegen's numeric
makeBetween uses inclusive bounds while esgen and the VM are exclusive,
and vm/vm_test.go's fixture map was already not gofmt-clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What was wrong
An audience filtering
country < "Argentina"returned an entity withcountry = "Albania"from Elasticsearch and returned nobody from the VM. Ordering against a string column was implemented three different ways:country < "Argentina", entitycountry = "Albania"{"range":{"plain_country":{"lt":"Argentina"}}}— akeywordfield, so a byte-order range → matchesoperateStringshad no ordering case →ErrorValue→ always false, for every entityNumericRangeQuerywhoseint/float64type switch matched no string bound, leaving both bounds nil → predicate silently dropped<and>=were both false in the VM. The two halves not being complementary is the tell that it was an error result rather than a comparison — and downstream that reads as a definitive "no", not "unknown".What changes
All three now compare the way the keyword index already does.
str < <= > >=ErrorValuestrings.Compare[]str < <= > >=(nil, false)str BETWEEN a AND bwalkTernaryhad Int/Number/Time only →(nil, false)gt/ltpair esgen emitsstr = != CONTAINS LIKE INThe generator coercion was not cosmetic
makeRangeandcoerceScalar(which feedsmakeBetween) triedParseFloatthendateparse.ParseAnyon the literal regardless of column type. For a string column that meant:ES then compared the keyword term
"2026-05-09"against"1778284800000"—"2">"1", so every ISO date in the index sorted greater. Both now keep the literal verbatim forStringType/StringsType/MapStringType; other types are untouched, sovisitct < "10"still coerces toint64(10).Verification
go test -overlay, which never touches the worktree. Representative pre-fix output: esgen producedLT: 1.7782848e+12where"2026-05-09"was expected; the VM cases reportedshould have non-nil result but was nil. The VM's table test fatals on first mismatch, so the[]stringand BETWEEN cases were isolated under a second overlay to confirm each fails on its own rather than riding along.go build ./...,go vet ./...andgo test ./...all clean on this base.replace. The paired lio change is https://github.com/lytics/lio/pull/39220, which stays red until this merges and lio's pin moves — that is deliberate, since lio's compiled evaluator bypassesoperateStringsand the two semantics must not diverge per-account.Left alone deliberately
blevegen.makeBetween's numeric path uses inclusive bounds while esgen and the VM are exclusive. Pre-existing, out of scope, and bleve is not a production query path in lio.vm/vm_test.go's fixture map was already notgofmt-clean on master; reformatting it would bury this diff in noise.user_id > "abc"assertedevalErrorand now assertsfalse, which is the behavior this PR is introducing.