bencch

Structured compiler runner and reporting bench for armfortas.

This repo holds:

• bench-core/ — bench-owned compiler-facing types
• bench/ — the afs-tests runner
• suites/ — authored bench suites
• fixtures/ — reusable fixture programs
• reports/ — failure and consistency bundles

Current Setup

bencch is no longer aiming to be the primary product story for a generic compiler bench. Its role is clearer now:

• the root armfortas harness is the fast armfortas-first runner and the default home for new source-directed testing ideas
• bencch is the structured matrix/reporting/differential runner around that same testing language

Source comments in shared fixtures are the canonical leaf-assertion language. bencch should consume those directives where supported and explain unsupported directives clearly, rather than inventing a separate assertion dialect.

Today bencch is wired to a surrounding armfortas checkout. The practical way to use it is from the armfortas workspace root. CLI-side compiler and tool paths are overridable now; linked capture still comes from the surrounding workspace. That linked compiler surface is currently isolated in bench/src/compiler.rs, and the bench-owned compiler-facing types now live in bench-core/.

cargo run -p afs-tests -- list
cargo run -p afs-tests -- run --suite frontend

Standalone compiler adapters are not finished yet.

Usage

List suites:

cargo run -p afs-tests -- list

Run one suite family:

cargo run -p afs-tests -- run --suite consistency/runtime

Run against an explicit compiler binary:

cargo run -p afs-tests -- run --suite consistency/runtime-control-flow --armfortas-bin ./target/debug/armfortas

Run differential checks with explicit reference compiler paths:

cargo run -p afs-tests -- run --suite differential/runtime-control-flow --gfortran-bin /opt/homebrew/bin/gfortran --flang-bin /opt/homebrew/bin/flang-new

List the staged real-project differential ladder:

cargo run -p afs-tests -- projects list

Run one real project through its native build system with armfortas and flang-new:

cargo run -p afs-tests -- projects run --project fortbite --armfortas-bin ./target/release/armfortas

Run one case with full stage capture:

cargo run -p afs-tests -- run --suite frontend --case stage_walk --all --verbose

Run consistency coverage:

cargo run -p afs-tests -- run --suite consistency --all

Run differential coverage:

cargo run -p afs-tests -- run --suite differential

Reports are written under bencch/reports/.

Project differential reports land under bencch/reports/projects/.

Environment overrides work too:

BENCCH_ARMFORTAS_BIN=./target/debug/armfortas cargo run -p afs-tests -- run --suite consistency/object

Every compiler, generated program, inspection tool, and project command runs in a managed process group. The runner drains both output streams concurrently, retains at most 16 MiB per stream, and treats timeout, cancellation, incomplete capture, and output truncation as hard harness failures. On Unix, expiry terminates the command's entire process group, including descendants that inherited the capture pipes, and reaps the direct child.

The conservative defaults can be overridden with positive integer values:

• BENCCH_COMPILE_TIMEOUT_SECS (default 120)
• BENCCH_RUN_TIMEOUT_SECS (default 30)
• BENCCH_TOOL_TIMEOUT_SECS (default 60)
• BENCCH_PROJECT_TIMEOUT_SECS (default 1800)
• BENCCH_KILL_GRACE_MS (default 1000)
• BENCCH_OUTPUT_LIMIT_BYTES (default 16777216, independently for stdout and stderr)

Malformed, zero, and excessive limit values fail closed before the command is launched. Project reports record TIMEOUT, CANCELLED, OUTPUT_LIMIT, and HARNESS_FAILURE separately and retain the bounded partial output.

Suite Format

Suites are plain text files under suites/.

suite "consistency/runtime"

case "mixed_types_cli_run_reproducible"
source "../../fixtures/runtime/mixed_types.f90"
opts => all
armfortas => run
repeat => 3
consistency => cli_run_reproducible
expect run.stdout check-comments
expect run.exit_code equals 0
end

Graph cases use entry plus file lines in reference-compilation dependency order:

suite "modules/runtime-graphs"

case "module_chain_runtime"
entry "../../fixtures/modules/module_chain/main.f90"
file "../../fixtures/modules/module_chain/math_seed.f90"
file "../../fixtures/modules/module_chain/math_values.f90"
file "../../fixtures/modules/module_chain/main.f90"
opts => O0, O1, O2
armfortas => run
expect run.stdout check-comments
end

Graph members remain separate translation units. The armfortas adapter preprocesses and captures each authored file independently, resolves module dependencies with the compiler's graph scanner, exchanges real .amod/.mod artifacts through an isolated case directory, emits one object per source, and links those objects for obj/run checks. Reference compilers compile the listed files in authored dependency order and then link their distinct objects; they do not reuse armfortas dependency decisions. Failure bundles store the entry source as source.f90 and preserve every authored member under sources/.

Common things the runner understands:

• stage capture like armfortas => tokens, ir, asm, obj, run
• opt matrices like opts => O0, O1, O2
• references like differential => gfortran, flang-new
• expected failures like xfail "reason"
• per-opt status like xfail when O1, O2 because "reason"
• consistency checks like cli_obj_vs_system_as and capture_run_reproducible

The suite DSL is for orchestration:

• opts
• compilers and references
• graph composition
• capability policy
• reporting and bundles

Leaf assertions should come from shared source directives whenever possible. expect run.stdout check-comments enforces ordered ! CHECK: lines together with ! FILE_CHECK: and ! FILE_NOT: against the captured run sandbox. Other ! FILE_* directives fail closed until bencch implements their exact root-harness semantics.

Notes

• .docs/ is local and gitignored.
• The runner is currently strongest on matrices, differential behavior, capability-aware execution, reports, bundles, and graph orchestration.
• The shared-language reset and follow-through testing roadmap live in the parent armfortas repo under .docs/testing/.