Filed after two independent sweeps (mine and a peer session's) both scored bench_typed_array_untyped_access as parity when it is a 7.7x loss.
What happened
Every runner in use extracts the benchmark's time as the first label:number line:
./$b | grep -oE ":[0-9]+" | head -1
For that fixture the first labelled line is ta_untyped_typed_ratio — a perry-to-perry ratio (0.945 for perry, 0.883 for node), deliberately so, because the fixture's own header names the untyped/typed ratio as the #5525 metric it exists to protect. Comparing perry's ratio to node's ratio is meaningless across runtimes, and it happens to land near 1.0, so the row scored 1.00-1.07x.
The absolute numbers, same box:
|
perry |
node |
|
ta_untyped_access |
2955 |
384 |
7.7x |
ta_typed_access |
1593 |
369 |
4.3x |
Why this is worse than the zero-millisecond rows (#9338)
A 0ms row is obviously wrong and invites a second look. This one prints a plausible number in the right range, so nothing prompts one. Two sessions independently mis-scored it for days. A harness that cannot distinguish "parity" from "I read the wrong number" is worse than no harness, because it manufactures confidence.
Proposed fix
- Match an explicit time label, never a position. The runner should look for a known time key (
<name>:<ms> where <name> matches the benchmark, or a dedicated elapsed_ms: line), not head -1.
- Fail loudly on no match. A row whose value cannot be tied to a time label must report
UNSCOREABLE, not a number. Silence is the bug; an error is a finding.
- Rename the ratio line in
bench_typed_array_untyped_access.ts (e.g. ratio_untyped_over_typed:) so it cannot be mistaken for a duration even by a positional reader, and keep the absolute ta_untyped_access / ta_typed_access lines as the scored ones.
- Audit the remaining fixtures for the same shape — any first-line checksum, count or ratio has the identical failure mode.
Related: #9338 (fixtures whose work is dead-code-eliminated, scoring 0 ms). Same family — the runner trusting a number it has not established is a time.
The 7.7x row itself is being taken separately as a typed-array kind-dispatch issue (my lane, adjacent to #9342/#9360); this issue is only about the measurement apparatus that hid it.
Filed after two independent sweeps (mine and a peer session's) both scored
bench_typed_array_untyped_accessas parity when it is a 7.7x loss.What happened
Every runner in use extracts the benchmark's time as the first
label:numberline:For that fixture the first labelled line is
ta_untyped_typed_ratio— a perry-to-perry ratio (0.945 for perry, 0.883 for node), deliberately so, because the fixture's own header names the untyped/typed ratio as the #5525 metric it exists to protect. Comparing perry's ratio to node's ratio is meaningless across runtimes, and it happens to land near 1.0, so the row scored 1.00-1.07x.The absolute numbers, same box:
ta_untyped_accessta_typed_accessWhy this is worse than the zero-millisecond rows (#9338)
A
0msrow is obviously wrong and invites a second look. This one prints a plausible number in the right range, so nothing prompts one. Two sessions independently mis-scored it for days. A harness that cannot distinguish "parity" from "I read the wrong number" is worse than no harness, because it manufactures confidence.Proposed fix
<name>:<ms>where<name>matches the benchmark, or a dedicatedelapsed_ms:line), nothead -1.UNSCOREABLE, not a number. Silence is the bug; an error is a finding.bench_typed_array_untyped_access.ts(e.g.ratio_untyped_over_typed:) so it cannot be mistaken for a duration even by a positional reader, and keep the absoluteta_untyped_access/ta_typed_accesslines as the scored ones.Related: #9338 (fixtures whose work is dead-code-eliminated, scoring 0 ms). Same family — the runner trusting a number it has not established is a time.
The 7.7x row itself is being taken separately as a typed-array kind-dispatch issue (my lane, adjacent to #9342/#9360); this issue is only about the measurement apparatus that hid it.