Measure hdrgen: native, Node and browsers, and correct the PRD - #261
Merged
Conversation
The PRD claims the WebAssembly build costs roughly 2x native and nothing here measures it. What has been observed pulls in two directions: the shipped wasm merges the 18-frame bracket in 22s headless under Node, matching the ~24s #245 records for the macOS desktop app, while the same bracket in the browser build under Playwright's Chromium did not finish in fifteen minutes on the same machine, and CI's chromium job does it inside a six-minute job. The comparison taken so far was not fair either: /usr/local/bin/hdrgen is an x86_64 binary on an arm64 machine, so every native timing ran under Rosetta against a native arm64 JIT. Hence building from the pinned fork commit rather than measuring only what happens to be installed. The load-bearing decision is that the browser leg is a bare page calling the same hdrgen.js on the same bytes, not the app. Only that shape can say whether the engine is slow or the application around it is. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Six tasks: shared fixtures and reporter, the two native legs including building arm64 from the pinned fork commit, the wasm-on-node control, the browser legs in a bare page, an orchestrator, and a run that writes down what it found. Two decisions the plan makes explicit. The browser legs serve themselves by request interception rather than a static server, so there is no port and nothing added to public/ or out/. And the wasm-node leg runs in a child process, because callMain is synchronous and a ceiling that cannot be enforced is not a ceiling. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The benchmark only means anything if each environment runs identical work, so the bracket, the argument vector and the staged naming live in one module that every leg has to go through. The reporter refuses to drop runs that did not finish: dropping them would report the slowest environment as absent rather than slow. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The summarise() sample had a truncated template literal that could not parse and could not have grouped by frame count. And 'node --test <dir>' fails on Node 24.15.0 in this environment, reproduced independently outside the repo, so every step now uses the glob form that works. Three later tasks would have hit the same wall. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The plan's record shape defines three statuses and its summarise() handled two, folding every error into the timeout count. A benchmark built to find out why a leg does not finish would then have reported a module that aborted immediately as one that was merely slow. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
summarise folded any non-ok status into "N/M timed out", so a leg that crashed immediately would be reported as one that merely ran slow. Telling a hang apart from a crash is most of why this benchmark exists, so the note now counts "timed out" and "errored" separately and lists whichever occurred. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ampling Two native binaries, wasm in Node, and wasm in Chromium and WebKit, all merging the same frames through the same argv so a difference between them is the environment and nothing else. The browser legs run in a bare page served by request interception: no Next build, no app, no worker, no UI, because a number taken through the application cannot tell a slow engine from a slow app. Frames are now sampled evenly across the bracket with the endpoints included, not taken from the front. A bracket is an exposure sequence, so a prefix is a run of near-identical long exposures: on the first four the pipeline's own filter reported "kept 1 of 4", which would have timed a one-frame merge while the table said four. The orchestrator hands its selection to the browser leg so every leg merges the same files from one source, and the selection itself is pinned to an explicit expected list in the tests. The wasm-node leg runs in a child process because callMain is synchronous: a run that hangs blocks the thread that would otherwise time it out, and a ceiling that cannot be enforced is not a ceiling. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
In a browser the WebAssembly build runs the merge at native speed: WebKit 13.4s against native arm64's 13.6s at the full bracket, Chromium 14.5s. PRD.md:79's 'roughly 2x native' is wrong, and pessimistically so. Rosetta costs about 1.35x, which is why the earlier ad-hoc native-versus-wasm comparison read as parity: the native side was emulated. wasm-node is the slowest leg and no explanation is offered, because none has been tested. The suspicion is the harness rather than Node. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The table is the output; the JSON is an artefact of one machine on one afternoon. Writing it to the repository root left it sitting in git status looking like something to commit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The claim was never measured. Merging the 18-frame reference bracket takes 13.4s in WebKit and 14.5s in Chromium against 13.6s for a native arm64 build of the same commit, so the figure was pessimistic for the browser, which is the case the web build actually runs. Keeps the two caveats that make the number honest rather than triumphal: the native leg reads frames from disk inside the timed region and the WebAssembly legs do not, and a merge is not pure compute. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Merged
CI cancelled after fifteen minutes with no failed step, which read as infrastructure and was not. Jest does collect `.mjs` despite its default testMatch listing only `[jt]s`, so `bench.test.mjs` was being pulled into the jsdom environment, where `node:test` means nothing. `scripts/` now sits beside the two end-to-end directories in testPathIgnorePatterns, for the same reason they are there: it has its own runner. The rest is lint. The browser leg's evaluate body was one 26-branch block and is now load, time, classify; a helper in the reporter shadowed a parameter; and three await-in-loop suppressions sat one line above the await they were meant to cover, which silently made them apply to nothing. Verified against every gate CI runs: 61 suites and 406 tests with no bench file collected, tsc clean at the root and in e2e-web, no lint errors, and the browser leg still measuring. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
adulbrich
added a commit
that referenced
this pull request
Aug 7, 2026
* docs: design for measuring hdrgen across native, node and browsers The PRD claims the WebAssembly build costs roughly 2x native and nothing here measures it. What has been observed pulls in two directions: the shipped wasm merges the 18-frame bracket in 22s headless under Node, matching the ~24s #245 records for the macOS desktop app, while the same bracket in the browser build under Playwright's Chromium did not finish in fifteen minutes on the same machine, and CI's chromium job does it inside a six-minute job. The comparison taken so far was not fair either: /usr/local/bin/hdrgen is an x86_64 binary on an arm64 machine, so every native timing ran under Rosetta against a native arm64 JIT. Hence building from the pinned fork commit rather than measuring only what happens to be installed. The load-bearing decision is that the browser leg is a bare page calling the same hdrgen.js on the same bytes, not the app. Only that shape can say whether the engine is slow or the application around it is. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: implementation plan for the hdrgen benchmark Six tasks: shared fixtures and reporter, the two native legs including building arm64 from the pinned fork commit, the wasm-on-node control, the browser legs in a bare page, an orchestrator, and a run that writes down what it found. Two decisions the plan makes explicit. The browser legs serve themselves by request interception rather than a static server, so there is no port and nothing added to public/ or out/. And the wasm-node leg runs in a child process, because callMain is synchronous and a ceiling that cannot be enforced is not a ceiling. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * bench(hdrgen): one argv builder and one reporter for every leg The benchmark only means anything if each environment runs identical work, so the bracket, the argument vector and the staged naming live in one module that every leg has to go through. The reporter refuses to drop runs that did not finish: dropping them would report the slowest environment as absent rather than slow. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: repair two defects in the benchmark plan found during Task 1 The summarise() sample had a truncated template literal that could not parse and could not have grouped by frame count. And 'node --test <dir>' fails on Node 24.15.0 in this environment, reproduced independently outside the repo, so every step now uses the glob form that works. Three later tasks would have hit the same wall. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: let the reporter tell a crash apart from a hang The plan's record shape defines three statuses and its summarise() handled two, folding every error into the timeout count. A benchmark built to find out why a leg does not finish would then have reported a module that aborted immediately as one that was merely slow. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * bench(hdrgen): tell an errored run apart from a timed-out one summarise folded any non-ok status into "N/M timed out", so a leg that crashed immediately would be reported as one that merely ran slow. Telling a hang apart from a crash is most of why this benchmark exists, so the note now counts "timed out" and "errored" separately and lists whichever occurred. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * bench(hdrgen): the remaining legs, the orchestrator, and even frame sampling Two native binaries, wasm in Node, and wasm in Chromium and WebKit, all merging the same frames through the same argv so a difference between them is the environment and nothing else. The browser legs run in a bare page served by request interception: no Next build, no app, no worker, no UI, because a number taken through the application cannot tell a slow engine from a slow app. Frames are now sampled evenly across the bracket with the endpoints included, not taken from the front. A bracket is an exposure sequence, so a prefix is a run of near-identical long exposures: on the first four the pipeline's own filter reported "kept 1 of 4", which would have timed a one-frame merge while the table said four. The orchestrator hands its selection to the browser leg so every leg merges the same files from one source, and the selection itself is pinned to an explicit expected list in the tests. The wasm-node leg runs in a child process because callMain is synchronous: a run that hangs blocks the thread that would otherwise time it out, and a ceiling that cannot be enforced is not a ceiling. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: record what the hdrgen benchmark measured In a browser the WebAssembly build runs the merge at native speed: WebKit 13.4s against native arm64's 13.6s at the full bracket, Chromium 14.5s. PRD.md:79's 'roughly 2x native' is wrong, and pessimistically so. Rosetta costs about 1.35x, which is why the earlier ad-hoc native-versus-wasm comparison read as parity: the native side was emulated. wasm-node is the slowest leg and no explanation is offered, because none has been tested. The suspicion is the harness rather than Node. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * bench(hdrgen): write results beside the run's scratch, not into the repo The table is the output; the JSON is an artefact of one machine on one afternoon. Writing it to the repository root left it sitting in git status looking like something to commit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: the WebAssembly build is not 2x native, it is about native The claim was never measured. Merging the 18-frame reference bracket takes 13.4s in WebKit and 14.5s in Chromium against 13.6s for a native arm64 build of the same commit, so the figure was pessimistic for the browser, which is the case the web build actually runs. Keeps the two caveats that make the number honest rather than triumphal: the native leg reads frames from disk inside the timed region and the WebAssembly legs do not, and a merge is not pure compute. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(bench): keep the benchmark out of Jest, and satisfy the linter CI cancelled after fifteen minutes with no failed step, which read as infrastructure and was not. Jest does collect `.mjs` despite its default testMatch listing only `[jt]s`, so `bench.test.mjs` was being pulled into the jsdom environment, where `node:test` means nothing. `scripts/` now sits beside the two end-to-end directories in testPathIgnorePatterns, for the same reason they are there: it has its own runner. The rest is lint. The browser leg's evaluate body was one 26-branch block and is now load, time, classify; a helper in the reporter shadowed a parameter; and three await-in-loop suppressions sat one line above the await they were meant to cover, which silently made them apply to nothing. Verified against every gate CI runs: 61 suites and 406 tests with no bench file collected, tsc clean at the root and in e2e-web, no lint errors, and the browser leg still measuring. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
scripts/bench-hdrgen, a committed instrument outside the test suite, and records what it measured.Why
PRD.mdclaimed the WebAssembly build costs "roughly 2x native". Nothing in the repository measured that. The claim then got repeated in discussion as though it were established, which is how an unverified number becomes a design constraint.Results
Apple Silicon, arm64, machine idle. Median of three, seconds, for the merge alone.
In a browser the WebAssembly build runs the merge at native speed. WebKit 13.4s against native arm64's 13.6s; Chromium 14.5s, about 1.07x. The PRD is corrected here: it was wrong, and pessimistically so, for the case the web build actually runs.
Rosetta costs about 1.35x. That matters beyond curiosity: the only native hdrgen installed on the machine is x86_64, so every earlier ad-hoc comparison was against an emulated binary. An informal "native 4.0s vs wasm 4.2s, essentially parity" reading came from exactly that, and was flattering wasm for the wrong reason. Building arm64 from the commit
versions.jsonpins is what makes the native column mean anything.wasm-nodeis the odd row at 1.68x native, slower than either browser despite sharing V8 with Chromium. No explanation is offered because none was tested. The suspicion is the harness: the module cannot fetch its own.wasmunder Node, so that leg hands over a compiled module throughinstantiateWasm, which may not tier up like a browser's streaming compile. Flagged rather than explained away.Design decisions worth reviewing
The browser legs are a bare page, no Next build, no app, no worker, no UI, served by request interception. A number taken through the application cannot separate a slow engine from a slow application, and that separation was the point.
WebKit is measured, not just Chromium, unlike
perf.config.tswhich deliberately fixes the engine to compare hosts. The desktop build runs WKWebView, so WebKit is what tells "browsers are slow at this" apart from "Chromium is slow at this".Frames are sampled evenly across the bracket, endpoints included, not taken from the front. A bracket is an exposure sequence, so a prefix is a run of near-identical long exposures: on the first four the pipeline's own filter reported "kept 1 of 4", which would have timed a one-frame merge while the table said four. The orchestrator hands its selection to the browser leg so every leg merges the same files from one source.
A timeout is recorded as a result, not an error, and a crash is distinguished from a hang, because those send an investigation in different directions.
What this does not answer
The deployed site. A full merge is ~14s in a browser, so a deployment that feels far slower is not slow because of the merge; delivery of the wasm payload and the app around it are what remain.
e2e-web/tests/perf.bench.tsalready takesTARGET_URLand exists to separate those.🤖 Generated with Claude Code