Skip to content

docs(tinygo): correct the xxh3 limitation and point to murmur3 - #799

Merged
mparrett merged 1 commit into
mainfrom
docs/tinygo-xxh3-limitation
Sep 7, 2026
Merged

docs(tinygo): correct the xxh3 limitation and point to murmur3#799
mparrett merged 1 commit into
mainfrom
docs/tinygo-xxh3-limitation

Conversation

@mparrett

@mparrett mparrett commented Sep 6, 2026

Copy link
Copy Markdown
Collaborator

What this changes

The TinyGo guide's xxh3 bullet describes a substitution that does not happen. It currently reads:

xxh3 is not bit-compatible. The xxh3 dependency is assembly-only on
arm64/amd64, so TinyGo links a pure-Go substitute hash. It is deterministic
but produces different digests, making a TinyGo build its own determinism
domain.

Two claims there send a reader the wrong way.

No substitute is linked. pkg/rt/interop_xxh3.go carries //go:build !tinygo, so a TinyGo build has no xxh3 namespace. A program calling xxh3/* does not get different digests; it gets nil and fails at the call site.

murmur3 is the portable hash, not a divergent one. pkg/rt/hash_murmur3.go is untagged and registered on every build, and it masks results to 31 bits so a value is identical on 64-bit native and 32-bit wasm. Its own header states this intent. The current text points at a determinism domain to avoid, when the available hash is the one that removes the divergence.

The replacement bullet says the namespace is absent, names murmur3 as the call to make instead, notes that the surface matches (Hash, HashSeed, HashString, HashStringSeed), and gives the 31-bit masking so the reader can see how it composes with the int-width limitation directly above it.

How this surfaced

Building a real program: a game whose determinism layer is keyed on a seed fold. Under TinyGo it builds and reaches its title screen, then dies:

FATAL ERROR: TypeError: nil is not a function
  new-game <- show-title-screen <- generate-title <- pick <- int-in <- xxh3-64

The guide sends you looking for a digest mismatch. The failure is a missing namespace, and the fix is a one-token swap to murmur3. After that swap the game runs, and its generated world is byte-identical between the TinyGo wasm build (32-bit int) and the stock-Go wasm build (64-bit int): same sha256 over the rendered dungeon at a fixed seed. That is the masking doing what its implementation comment describes.

Verification

  • The !tinygo tag on interop_xxh3.go and the absence of a tag on hash_murmur3.go both read off origin/main.
  • murmur3/HashSeed exercised on a native build; results land inside 31 bits.
  • The crash above reproduced from a clean origin/main TinyGo wasm build.
  • Cross-width determinism checked by diffing the rendered world from both builds at one fixed seed. One seed, not a sweep.

Docs only. No code changes, nothing to regenerate, no CI surface touched.

@nnunley

nnunley commented Sep 6, 2026

Copy link
Copy Markdown
Collaborator

So, it might be worth pointing out that I had a pure let-go implementation of xxh3 that might be worth trying the aot on to see if it can't provide that fingerprint hash.

@mparrett

mparrett commented Sep 6, 2026

Copy link
Copy Markdown
Collaborator Author

So, it might be worth pointing out that I had a pure let-go implementation of xxh3 that might be worth trying the aot on to see if it can't provide that fingerprint hash.

Yeah, thanks for the reminder. I remember you had that. Would be neat to see how it goes with aot. I'll take a look!

@mparrett

mparrett commented Sep 6, 2026

Copy link
Copy Markdown
Collaborator Author

@nnunley I tried it. The pure xxh3 is the xsofy/hash.lg that nooga/xsofy#27 landed and your #64 replaced; the last full version is the parent of that merge, nooga/xsofy d9be3034. It still passes its golden suite on current let-go (61 tests, 171 assertions), so it made a clean corpus.

What AOT does for it: scripts/lg-compile lowers all 52 defns, and linking the output into an lg binary runs the hash about 7× faster natively, from roughly 500 ms to 73 ms per 20,000 twenty-byte hashes. For scale, the reflect-boxed xxh3/Hash sits at 40–55 ms and murmur3/Hash at 2–8 ms on the same loop.

What it cannot do is change the integer width. The lowerer maps :int to Go int, and the u64+ / u64* helpers came out as calls to rt.CoreUncheckedAdd on vm.Int, so the lowered Go follows the platform the same way the interpreter does. Same probe file, same wasmtime:

build (bit-shift-left 1 40) pure xxh3 of "abc" goldens
stock Go wasip1 (64-bit int) 1099511627776 8696274497037089104 171 pass, 0 fail
TinyGo wasip1 (32-bit int) 0 -1078166263 107 pass, 55 fail, 9 error

Linking the lowered package into a tinygo build -target=wasip1 binary does not compile: constant 4294967295 overflows vm.Int at 18 sites, which are the implementation's own 32-bit masks. That is at least a louder failure than the interpreter's silent truncation.

So a 64-bit hash under TinyGo wasm needs vm.Int to be int64, or an int64-typed lowering plus a 64-bit boxed value, whichever algorithm computes it. The 31-bit mask in nooga/xsofy#192's murmur3 sidesteps that; a pure let-go XXH32 would too.

The probe did turn up one gap on the let-go side: unchecked-add and unchecked-multiply had no IR op, so even a fully typed kernel lowered them as prim calls. #811 (draft) makes them first-class on both backends. On an LCG step that is about 2.4× lowered and 30% interpreted. It does not move the xxh3 number, where the cost is the boxed call boundary between fifty small defns (#722), but it is the gap a typed hash kernel hits first.

@mparrett
mparrett requested a review from nnunley September 6, 2026 23:57
The bullet described a pure-Go substitute hash that is never linked.
`pkg/rt/interop_xxh3.go` is gated `//go:build !tinygo`, so a TinyGo build has
no `xxh3` namespace at all: a program calling `xxh3/*` gets nil and fails at
the call site rather than producing different digests.

It also had the determinism story backwards. `pkg/rt/hash_murmur3.go` is
untagged and registered on every build, and masks results to 31 bits so a
value is identical on 64-bit native and 32-bit wasm — the header comment says
so explicitly. murmur3 is the hash that removes the divergence, not one that
introduces it, so the guide should name it as the port target instead of
warning about a determinism domain to accept.

Found by building a downstream program whose seed fold calls `xxh3-64`: it
reaches its title screen under TinyGo and then dies on the nil namespace.
After swapping to `murmur3` the program runs, and its generated world is
byte-identical between the TinyGo wasm build and the stock-Go wasm build at a
fixed seed.

Docs only; no code changes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@mparrett
mparrett force-pushed the docs/tinygo-xxh3-limitation branch from 8afed0d to 067888a Compare September 7, 2026 00:54
@mparrett
mparrett merged commit f26eb49 into main Sep 7, 2026
13 checks passed
@mparrett
mparrett deleted the docs/tinygo-xxh3-limitation branch September 7, 2026 05:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants