fix: repair benchmark infrastructure (9 bugs blocking any run) - #88
Closed
txloc1909 wants to merge 6 commits into
Closed
fix: repair benchmark infrastructure (9 bugs blocking any run)#88txloc1909 wants to merge 6 commits into
txloc1909 wants to merge 6 commits into
Conversation
…ren) Implements all 16 AWFY benchmarks, 6 CLBG benchmarks, and 4 Wren suite benchmarks in Lox++, plus 4 clox-compatible variants. Includes Dockerfiles for Python/Lua/JS/Wren/clox (each bundling third-party sources at pinned commits), manifest.toml, langs.toml, and runner.py with 4 output adapters (awfy/wren/clbg/raw) and podman execution support. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Update AWFY commit hash from non-existent SHA to master HEAD (74306fec) - Update Wren commit hash to an accessible commit (99d2f0b8) - clox: switch to master.tar.gz (commit-only SHAs 404 on GitHub archives); add libc6-dev (missing on Ubuntu 24.04); fix COPY path to benchmarks/clox/ - JS/Lua: Alpine uses busybox tar which lacks --wildcards; replace with strip-components=1 extraction then explicit subdir copy - Python: likewise switch to strip-components=1; files live in Python/ subdir - Add run_awfy_python.py, run_awfy_js.js, run_awfy_lua.sh harness wrappers: AWFY benchmarks must be invoked via their Run harness, not executed directly, and require PascalCase class names (NBody, DeltaBlue, CD) not filenames - Wren: document that standalone wren CLI was removed upstream; update Makefile path to projects/make/ and switch to strip-components extraction Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The original manifest used `[bench.lang] file = "..."` syntax — placing a
key-value pair on the same line as a section header — which is invalid per the
TOML spec (and rejected by Python 3.14's stricter tomllib).
Reformatted every entry as an inline table:
lox = {file = "bounce.lox"}
Also updated AWFY filenames: the current AWFY master renamed all benchmarks
from PascalCase (Bounce.py, Fib.py) to lowercase (bounce.py, fib.py).
Removed fib and earley from Python/JS/Lua entries — those files don't exist
in AWFY (lox-specific benchmarks).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- python: use run_bench.py wrapper (direct file exec produces no output) - js: use run_bench.js wrapper (same reason) - lua: use run_bench.sh wrapper (module search path must be set via cd) - clox: change adapter from 'raw' (wall clock incl. container startup) to 'clox' (self-reported elapsed time from stdout last line) - wren: comment out — standalone CLI no longer exists in upstream repo Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1. Scientific notation in AWFY output: Lox++ str() prints large µs counts in scientific notation (e.g. 3.99e+06 us). The old regex [\\d.]+ didn't match 'e+06', so timings above ~1 s showed N/A. Extended to _NUM = r'[\\d.eE+\\-]+' throughout _parse_awfy. Also added a fallback: if the summary line isn't found, compute average from iter_us. 2. --no-container applied to all languages: The condition `if no_container or lang == "lox"` ran Python/JS/Lua directly on the host when --no-container was set, causing FileNotFoundError because the wrapper scripts live inside containers. Fixed to `if lang == "lox" and no_container` — only lox bypasses containers. 3. clox adapter added: clox benchmarks self-report elapsed time as the last stdout line (seconds). The old 'raw' adapter used wall-clock time which included ~200 ms container startup overhead. New _parse_clox() reads the last non-empty stdout line as seconds and converts to µs. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- AWFY: update commit from non-existent SHA to the actual master HEAD - AWFY: note that filenames are now lowercase; fib/earley have no Python/JS/Lua variants - Wren: update commit; document that standalone CLI was removed upstream - clox: document why master branch ref is used instead of a commit SHA - Updating a pinned commit: add pre-flight check for GitHub archive reachability; fix build context (must be repo root, not benchmarks/) and smoke-test command Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
txloc1909
force-pushed
the
feat/benchmarks
branch
from
June 8, 2026 16:16
82ec489 to
250a3fd
Compare
txloc1909
force-pushed
the
fix/benchmarks-infrafix
branch
from
June 8, 2026 16:17
3708557 to
8f36735
Compare
Owner
Author
|
Superseded — all 9 infrastructure fixes plus the deltablue and havlak benchmark fixes have been folded directly into feat/benchmarks (PR #86). Closing this PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes 9 bugs in the benchmark setup from PR #86 that prevented any language other than Lox++ from running. All fixes were validated by a full benchmark run — see
notes/benchmark_report_2026-06-08.mdfor the results.Root cause breakdown
5e9fa8e…→ GitHub 404 (archives only served for branch/tag refs)74306fec4a8e084…→ GitHub 40499d2f0b8d9a658f…→ GitHub 404archive/master.tar.gzmanifest.tomlinvalid TOML — key-value on the same line as[section]headerlox = {file = "..."}--wildcards— not supported by Alpine's busybox tar--strip-components=1+ explicit subdir copyrun_awfy_python.py,run_awfy_js.js,run_awfy_lua.shlibc6-devon Ubuntu 24.04; wrongCOPYpathlibc6-dev; fixed COPY tobenchmarks/clox/3.99e+06 us) — AWFY regex missed it, timing showed N/A[\d.eE+\-]+; added fallback to compute average from per-iteration dataBonus:
--no-containerwas applied to all languages instead of lox-only, causing Python/JS/Lua to run on the host (and fail with FileNotFoundError) instead of their containers. Fixed.Wren note: The standalone
wrenCLI was removed from the wren upstream repo. The Wren suite benchmarks currently run Lox++ only.Dockerfile.wrenis kept for future use but documents the situation.Benchmark results (from the fixed run)
AWFY suite, average per iteration, 5 runs:
Lox++ ≈ CPython on object-heavy code; beats Python/Lua on json; 4–8× behind Node.js V8 (JIT).
Full report in
notes/benchmark_report_2026-06-08.md.🤖 Generated with Claude Code