Skip to content

harness: the suite runner identifies "the time" positionally, so a fixture whose first labelled line is a ratio scores as parity #9374

Description

@proggeramlug

Filed after two independent sweeps (mine and a peer session's) both scored bench_typed_array_untyped_access as parity when it is a 7.7x loss.

What happened

Every runner in use extracts the benchmark's time as the first label:number line:

./$b | grep -oE ":[0-9]+" | head -1

For that fixture the first labelled line is ta_untyped_typed_ratio — a perry-to-perry ratio (0.945 for perry, 0.883 for node), deliberately so, because the fixture's own header names the untyped/typed ratio as the #5525 metric it exists to protect. Comparing perry's ratio to node's ratio is meaningless across runtimes, and it happens to land near 1.0, so the row scored 1.00-1.07x.

The absolute numbers, same box:

perry node
ta_untyped_access 2955 384 7.7x
ta_typed_access 1593 369 4.3x

Why this is worse than the zero-millisecond rows (#9338)

A 0ms row is obviously wrong and invites a second look. This one prints a plausible number in the right range, so nothing prompts one. Two sessions independently mis-scored it for days. A harness that cannot distinguish "parity" from "I read the wrong number" is worse than no harness, because it manufactures confidence.

Proposed fix

  1. Match an explicit time label, never a position. The runner should look for a known time key (<name>:<ms> where <name> matches the benchmark, or a dedicated elapsed_ms: line), not head -1.
  2. Fail loudly on no match. A row whose value cannot be tied to a time label must report UNSCOREABLE, not a number. Silence is the bug; an error is a finding.
  3. Rename the ratio line in bench_typed_array_untyped_access.ts (e.g. ratio_untyped_over_typed:) so it cannot be mistaken for a duration even by a positional reader, and keep the absolute ta_untyped_access / ta_typed_access lines as the scored ones.
  4. Audit the remaining fixtures for the same shape — any first-line checksum, count or ratio has the identical failure mode.

Related: #9338 (fixtures whose work is dead-code-eliminated, scoring 0 ms). Same family — the runner trusting a number it has not established is a time.

The 7.7x row itself is being taken separately as a typed-array kind-dispatch issue (my lane, adjacent to #9342/#9360); this issue is only about the measurement apparatus that hid it.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions