diff --git a/bench/METHODOLOGY.md b/bench/METHODOLOGY.md index 8a4aa24..eb2072d 100644 --- a/bench/METHODOLOGY.md +++ b/bench/METHODOLOGY.md @@ -687,6 +687,119 @@ builder-quiescence gate; it is skipped under `--equalize` (the fixed region makes it grid/font-independent by construction). Full design rationale: `docs/specs/bench-doom-fire.md`. +### 8. Latency under load (concurrent DSR probe + bulk producer, one pty) + +**Opt-in, not in the default axis set** (`--axes latency-under-load`): it is +the most expensive axis (`LOADLAT_REPS` independent full launches, each one +running a bulk producer for tens of seconds), and its entire point is to +measure under contention, so it never belongs in a quick default sweep. + +**Why this axis exists.** Axis 1 (throughput) and axis 2 (input latency) +never run at the same time — the DSR probe (`dsr_probe`) runs alone as the +pty child, and the 150 MB `cat` runs alone as the pty child. Neither axis +answers "how fast does a keystroke echo when the terminal is also draining +a flood of output" — the actual shape of typing while a build log or `cat` +scrolls past. This axis measures exactly that: it runs the DSR probe and a +bulk producer **concurrently, in the same pty, in the same launch**. + +**One pty, not two panes.** The two are launched as sibling processes of the +same pty child (`wrapper.sh`'s `NOA_MODE=latload`), both talking to the +single tty the terminal opened — the producer only writes, the probe writes +its query and reads the terminal's reply, so there is no fd contention +between them, only contention for the terminal's own parse/render/respond +loop. This was chosen over a two-pane design because it is the closer match +to the real "typing while output floods" scenario: a single input/output +path under load, not cross-pane isolation. A two-pane variant would answer a +different, narrower question (does terminal-wide contention leak across +independent panes?) — worth adding as a sub-variant later, not implemented +in this pass (see "not implemented" below). + +**Bounded-duration bulk producer, not a bare `cat`.** A 3 GB/s terminal can +finish 150 MB in well under a second — too short a window to hold +contention through hundreds of DSR iterations. `bench/tools/bulk_produce.c` +re-streams the same 150 MB ascii file in a loop under a **wall-clock +duration bound** (`LOADLAT_BULK_DURATION_S`: 20s full / 8s quick), not a +byte-count bound, so the window is long enough regardless of how fast the +terminal drains. Same pty flow-control semantics as axis 1 (`write()` blocks +on the kernel pty buffer until the terminal drains), so its own +`/` gives the sustained MiB/s of the exact concurrent +window it was writing. + +**Per-launch sequence** (`wrapper.sh` `NOA_MODE=latload`, driven by +`run_all.sh: run_latency_under_load`): + +1. **No-load control**: `dsr_probe` runs alone first (`LOADLAT_CTRL_ITERS` + kept iterations, `LOADLAT_CTRL_WARMUP` discarded) — the baseline reading + for *this exact launch*, not a historical number from a different run. +2. `bulk_produce` starts in the background against the same pty. +3. **Discard window** (`LOADLAT_DISCARD_S`: 2s full / 1s quick): the + producer's write loop takes a fraction of a second to reach steady pty + backpressure; this is a deliberate margin, not the unrelated 5-40s + macOS GPU-driver-pool reclaim window documented for the memory axes + (that window is about idle GPU pool teardown, which has nothing to do + with a CPU/pty contention experiment running within seconds of launch — + verified inapplicable here by construction, not assumed). +4. **In-load probe**: `dsr_probe` runs again (`LOADLAT_ITERS` kept + iterations, `LOADLAT_WARMUP` discarded) while the producer is still + actively writing. +5. The wrapper joins the producer (`wait $bulk_pid`) before signaling the + sentinel, so the producer's own result file is always complete by the + time the harness reads it. + +**Reported pairing** (per terminal, `raw.tsv` axis `latency_under_load`): +the no-load control's pooled median/p95/p99/max/count, the in-load probe's +pooled median/p95/p99/max/count, the sustained MiB/s achieved during the +in-load window (median across reps), and a **degradation factor** +(in-load pooled median ÷ control pooled median) computed from this same +run's own numbers — never against a historical baseline. + +**Reps and pooling**: `LOADLAT_REPS` independent launches (5 full / 2 +quick — the same rep philosophy as mem-longevity's 5 cycles and startup's 5 +reps: enough independent process lifetimes to see spread without the +sequence taking minutes per terminal). Raw kept DSR samples (both control +and load, every launch) are pooled before computing percentiles, the same +convention axis 2 uses (`pooled_stats`) — a percentile from one launch's +few hundred samples is noisy; pooling across `LOADLAT_REPS` independent +launches gives the tail more support. The console also prints each rep's +own numbers so the spread across launches is visible, not just the pooled +figure. + +**Overlap is verified, not assumed — by interval containment, not sample +counting.** `bulk_produce` records its own measured active interval +`[t0, t1]` (`CLOCK_MONOTONIC`, written into its result file at exit); +`wrapper.sh` separately records the in-load probe's own `[start, end]` +wall-clock window with the same clock in the same pty child process, so the +two are directly comparable. `run_all.sh` checks that the producer's +interval **contains** the probe's interval (`bulk_t0 <= probe_start` and +`bulk_t1 >= probe_end`, plus `bytes > 0`) and reports +`verified(bulk_active[Nms]_contains_probe[Mms])` or `NOT-VERIFIED(...)` with +the raw intervals attached. An earlier draft of this check instead counted +how many points in `bulk_produce`'s ~200ms-cadence progress log fell inside +the probe's window — that under-samples whenever the probe window (tens of +ms, since the in-load probe's several hundred iterations complete quickly) +is shorter than the logging cadence, which produced false `NOT-VERIFIED` +results on a from-scratch run even though the two processes were genuinely +concurrent (caught during this axis's own smoke test, not shipped +unverified). The containment check has no such blind spot: it only needs +each process's own start/end, not a sampling rate fine enough to land +inside a short window. The progress log itself is kept +(`bulk_produce`'s optional 4th argument) for auxiliary auditing of the +producer's write cadence, but is no longer the overlap oracle. + +**What this is and isn't**: like axis 2, this is the pty→parser→responder +loop, not photon/keyboard-to-glass latency — the same caveat applies here, +amplified by contention. It measures one terminal's own behavior under its +own bulk-output load, not cross-terminal fairness under a shared system +load (the builder-quiescence gate still applies: the harness refuses to +start while a `cargo`/`rustc`/`clang` build is alive system-wide, for the +same contention-sensitivity reason as the scroll axis). + +**Not implemented in this pass** (documented here so it isn't silently +assumed done): a two-pane cross-contamination variant, and reusing the +pooled-samples files for a full session-level histogram artifact beyond the +percentiles already reported. Both are natural follow-ups if the +single-pty result turns out to be interesting enough to warrant them. + ### Ghostty load-active timeout (baseline 2026-07-16) — root-caused & fixed The 20260716-084038 baseline reported Ghostty's two load-active rows as @@ -822,6 +935,7 @@ bench/run_all.sh # full 6-axis suite bench/run_all.sh --quick # smoke (fewer reps/shorter settles) bench/run_all.sh --only noa,kitty # subset of terminals bench/run_all.sh --axes memory,load # subset of axes +bench/run_all.sh --axes latency-under-load # opt-in axis, not in the default set bench/run_all.sh --force # bypass the builder-quiescence gate ``` diff --git a/bench/run_all.sh b/bench/run_all.sh index e4cd8ac..6ee7182 100755 --- a/bench/run_all.sh +++ b/bench/run_all.sh @@ -7,7 +7,10 @@ # scenario dual-reported as active @15s + settled @90s, ranked on # settled), load (idle CPU% + active CPU-time-per-workload), # fire (DOOM-fire IO stress: fixed 80x24 truecolor full-region -# repaint fps under pty flow control). +# repaint fps under pty flow control), latency-under-load (opt-in: +# DSR echo-latency distribution measured WHILE a bulk producer +# floods the same pty, paired with the no-load control and the +# sustained throughput achieved during the same window). # Terminals: noa (target/release), Ghostty, Termy, kitty — whichever exist. # # One command runs the whole suite and writes a machine-readable results.json @@ -35,10 +38,15 @@ WRAPPER="$BENCH_DIR/wrapper.sh" NOWNS="$TOOLS/nowns" PROBE="$TOOLS/dsr_probe" FIRE="$TOOLS/fire" +BULK="$TOOLS/bulk_produce" # ── options ──────────────────────────────────────────────────────── QUICK=0 ONLY="" +# latency-under-load is NOT in the default set: it is the most expensive axis +# (bulk producer + probe per rep, several reps) and its whole point is to run +# under contention, so it is opt-in via --axes latency-under-load (see +# METHODOLOGY.md axis 8). AXES="throughput,scroll,latency,startup,memory,load,fire" EQUALIZE=0 FORCE=0 @@ -152,6 +160,15 @@ if [ "$QUICK" = 1 ]; then MEM_MULTITAB_N=3; MEM_LONGEVITY_CYCLES=2; MEM_LONGEVITY_IDLE_S=1 LOAD_IDLE_SETTLE_S=5; LOAD_IDLE_S=10 FIRE_REPS=1; FIRE_SECS=3 + # latency-under-load (quick): 2 independent launches, each launch runs a + # no-load control probe then a load probe concurrent with an 8s bulk + # producer — smoke only. + LOADLAT_REPS=2 + LOADLAT_CTRL_ITERS=100; LOADLAT_CTRL_WARMUP=10 + LOADLAT_ITERS=150; LOADLAT_WARMUP=15 + LOADLAT_DISCARD_S=1 + LOADLAT_BULK_DURATION_S=8 + LOADLAT_TIMEOUT=30 else TP_REPS=3; SCROLL_REPS=3; LAT_RUNS=10; START_REPS=5 # latency (full): 10 process launches x 1000 kept iterations each (100 @@ -175,6 +192,20 @@ else # fire: median of FIRE_REPS runs, each FIRE_SECS of flat-out rendering # after 60 discarded warmup frames (see docs/specs/bench-doom-fire.md). FIRE_REPS=3; FIRE_SECS=10 + # latency-under-load (full, see METHODOLOGY.md axis 8): 5 independent + # launches — the same rep philosophy as mem-longevity's 5 cycles and + # startup's 5 reps, chosen for the same reason: enough independent + # process lifetimes to see spread without the sequence taking minutes per + # terminal (each launch is itself ~discard + control-probe + a bounded + # bulk-producer window, not a single quick sample). Each launch's raw DSR + # samples (both control and load) are pooled across launches, same + # convention as the plain `latency` axis's pooled percentiles. + LOADLAT_REPS=5 + LOADLAT_CTRL_ITERS=300; LOADLAT_CTRL_WARMUP=30 + LOADLAT_ITERS=500; LOADLAT_WARMUP=50 + LOADLAT_DISCARD_S=2 + LOADLAT_BULK_DURATION_S=20 + LOADLAT_TIMEOUT=90 fi TP_TIMEOUT=180; SCROLL_TIMEOUT=120; LAT_TIMEOUT=60; START_TIMEOUT=30 FIRE_TIMEOUT=60 @@ -184,7 +215,7 @@ MEM_HOLD_TIMEOUT=20 ASCII="$BENCH_DIR/150MB_ascii.txt" UNICODE="$BENCH_DIR/150MB_unicode.txt" SCROLLF="$BENCH_DIR/scroll_stress.txt" -if { axis_selected throughput || axis_selected load; } && { [ ! -f "$ASCII" ] || [ ! -f "$UNICODE" ]; }; then +if { axis_selected throughput || axis_selected load || axis_selected latency-under-load; } && { [ ! -f "$ASCII" ] || [ ! -f "$UNICODE" ]; }; then (cd "$BENCH_DIR" && python3 generate_data.py) fi if axis_selected scroll || axis_selected memory || axis_selected load; then @@ -196,7 +227,7 @@ fi # prebuilt dsr_probe would silently emit the old 4-field result format). tools_fresh() { local t - for t in nowns dsr_probe winwait wincount fire dispinfo; do + for t in nowns dsr_probe winwait wincount fire dispinfo bulk_produce; do [ -x "$TOOLS/$t" ] || return 1 [ "$BENCH_DIR/tools/$t.c" -nt "$TOOLS/$t" ] && return 1 done @@ -205,7 +236,7 @@ tools_fresh() { tools_fresh || \ (cd "$BENCH_DIR/tools" && mkdir -p bin && \ cc -O2 -o bin/nowns nowns.c && cc -O2 -o bin/dsr_probe dsr_probe.c && \ - cc -O2 -o bin/fire fire.c && \ + cc -O2 -o bin/fire fire.c && cc -O2 -o bin/bulk_produce bulk_produce.c && \ cc -O2 -framework ApplicationServices -o bin/winwait winwait.c && \ cc -O2 -framework ApplicationServices -o bin/wincount wincount.c && \ cc -O2 -framework ApplicationServices -o bin/dispinfo dispinfo.c) @@ -1073,6 +1104,92 @@ run_load_active() { echo "$((delta_cs * 10))" } +# run_latency_under_load +# -> prints one line: +# " \ +# \ +# " +# or empty on timeout/failure. A single launch runs, IN THIS ORDER: a +# no-load DSR control, then a bounded-duration bulk producer starts against +# the same pty, then (after a short ramp) a second DSR probe runs while the +# producer is still writing. Raw per-launch DSR samples (both control and +# load) are appended to the given pooled files so the caller can compute +# pooled percentiles across LOADLAT_REPS launches, same convention as the +# plain `latency` axis. See METHODOLOGY.md axis 8 for the full design +# rationale (why one pty, why bounded-duration, how overlap is verified). +run_latency_under_load() { + local term="$1" rep="$2" ctrl_pooled="$3" load_pooled="$4" + local sentinel="$RUNTMP/${term}.latload.$rep.sentinel" + local res_noload="$RUNTMP/${term}.latload.$rep.noload.result" + local res_load="$RUNTMP/${term}.latload.$rep.load.result" + local res_bulk="$RUNTMP/${term}.latload.$rep.bulk.result" + local prog_bulk="$RUNTMP/${term}.latload.$rep.bulk.progress" + local overlap_marker="$RUNTMP/${term}.latload.$rep.overlap" + local samples_noload="$RUNTMP/${term}.latload.$rep.samples.noload" + local samples_load="$RUNTMP/${term}.latload.$rep.samples.load" + kill_term "$term"; sleep 0.4 + rm -f "$sentinel" "$res_noload" "$res_load" "$res_bulk" "$prog_bulk" \ + "$overlap_marker" "$samples_noload" "$samples_load" + + local go="" + if [ "$FULLSCREEN" = 1 ]; then + go="$RUNTMP/${term}.latload.$rep.go.$RANDOM"; rm -f "$go" + fi + export NOA_MODE=latload NOA_SENTINEL="$sentinel" NOA_NOWNS="$NOWNS" \ + NOA_PROBE="$PROBE" NOA_BULK="$BULK" NOA_BULK_FILE="$ASCII" \ + NOA_BULK_DURATION="$LOADLAT_BULK_DURATION_S" \ + NOA_LOAD_WARMUP_DISCARD_S="$LOADLAT_DISCARD_S" \ + NOA_CTRL_ITERS="$LOADLAT_CTRL_ITERS" NOA_CTRL_WARMUP="$LOADLAT_CTRL_WARMUP" \ + NOA_PROBE_ITERS="$LOADLAT_ITERS" NOA_PROBE_WARMUP="$LOADLAT_WARMUP" \ + NOA_RESULT_NOLOAD="$res_noload" NOA_SAMPLES_NOLOAD="$samples_noload" \ + NOA_RESULT="$res_load" NOA_SAMPLES="$samples_load" \ + NOA_BULK_RESULT="$res_bulk" NOA_BULK_PROGRESS="$prog_bulk" \ + NOA_OVERLAP_MARKER="$overlap_marker" NOA_GO="$go" + + launch_term "$term" >/dev/null + if [ -n "$go" ]; then + fullscreen_term "$term" || emit "$term" meta - - fullscreen_window "FAILED (windowed for one latency-under-load rep)" note + : > "$go" + fi + if ! wait_sentinel "$sentinel" "$LOADLAT_TIMEOUT"; then + kill_term "$term"; sleep 0.3 + echo "" + return + fi + kill_term "$term"; sleep 0.4 + + local ctrl="0 0 0 0 0 0" load="0 0 0 0 0 0" + local bulk_bytes=0 bulk_ns=0 bulk_t0=0 bulk_t1=0 overlap="unverified" + [ -f "$res_noload" ] && ctrl="$(cat "$res_noload")" + [ -f "$res_load" ] && load="$(cat "$res_load")" + if [ -f "$res_bulk" ]; then + read -r bulk_bytes bulk_ns bulk_t0 bulk_t1 < "$res_bulk" + fi + cat "$samples_noload" >> "$ctrl_pooled" 2>/dev/null + cat "$samples_load" >> "$load_pooled" 2>/dev/null + + # Verify overlap empirically, cadence-independent: the bulk producer's own + # measured active interval [bulk_t0, bulk_t1] (CLOCK_MONOTONIC, written by + # bulk_produce itself) must CONTAIN the load probe's own recorded + # [probe_start, probe_end] window (same clock, same pty child process, so + # directly comparable) — i.e. the producer was still mid-write-loop for + # the ENTIRE duration the probe was running, not merely launched earlier. + # (An earlier version of this check counted progress-log points landing + # inside the probe window; that under-samples whenever the probe window — + # tens of ms — is shorter than the ~200ms progress-log cadence, which + # produced false NOT-VERIFIED results even though the two were genuinely + # concurrent. This containment check has no such blind spot.) + if [ -f "$overlap_marker" ] && [ "$bulk_t1" != 0 ]; then + read -r pstart pend < "$overlap_marker" + if [ "$bulk_t0" -le "$pstart" ] && [ "$bulk_t1" -ge "$pend" ] && [ "${bulk_bytes:-0}" -gt 0 ]; then + overlap="verified(bulk_active[$(( (bulk_t1 - bulk_t0) / 1000000 ))ms]_contains_probe[$(( (pend - pstart) / 1000000 ))ms])" + else + overlap="NOT-VERIFIED(bulk[${bulk_t0},${bulk_t1}]_probe[${pstart},${pend}])" + fi + fi + echo "$ctrl $load $bulk_bytes $bulk_ns $overlap" +} + # ── select terminals ─────────────────────────────────────────────── # warp is opt-in only (`--only ...,warp`): its $SHELL launch is unverified # and it carries account/AI-agent state that doesn't belong in a default @@ -1250,6 +1367,84 @@ for term in $SELECTED; do done fi # axis: fire +# ── LATENCY UNDER LOAD (concurrent DSR probe + bulk producer, one pty) ── +# See METHODOLOGY.md axis 8. Opt-in (not in the default AXES set) because it +# is the most expensive axis: LOADLAT_REPS independent launches, each one a +# no-load control probe + a bounded-duration bulk producer + a load probe +# run concurrently against the SAME pty. Skipped under --equalize (pinned +# grid geometry isn't the point of this axis). +if axis_selected latency-under-load && [ "$EQUALIZE" != 1 ]; then +for term in $SELECTED; do + echo "[latency-under-load] $term ($LOADLAT_REPS launches: no-load control + ${LOADLAT_BULK_DURATION_S}s concurrent bulk producer + in-load DSR probe)" + ctrl_pooled="$RUNTMP/${term}.llctrl_pooled"; : > "$ctrl_pooled" + load_pooled="$RUNTMP/${term}.llload_pooled"; : > "$load_pooled" + mibps_samples=""; got=0; overlap_fail=0 + for r in $(seq 1 $LOADLAT_REPS); do + out="$(run_latency_under_load "$term" "$r" "$ctrl_pooled" "$load_pooled")" + [ -z "$out" ] && { echo " rep $r: UNMEASURED (timeout)"; continue; } + set -- $out + c_med="$1"; c_p95="$2"; c_p99="$3"; c_max="$4"; c_cnt="$6" + l_med="$7"; l_p95="$8"; l_p99="$9"; l_max="${10}"; l_cnt="${12}" + b_bytes="${13}"; b_ns="${14}"; overlap="${15}" + if [ "${c_cnt:-0}" = 0 ] || [ "${l_cnt:-0}" = 0 ]; then + echo " rep $r: UNMEASURED (no DSR reply — reply appears render-thread/focus-gated)" + continue + fi + got=$((got + 1)) + mibps="$(awk -v b="$b_bytes" -v ns="$b_ns" 'BEGIN{ if(ns>0) printf "%.1f", (b/1048576)/(ns/1e9); else print 0 }')" + emit "$term" latency_under_load load "$r" sustained_mib_per_s "$mibps" mib_s + emit "$term" latency_under_load load "$r" overlap_check "$overlap" note + emit "$term" latency_under_load control "$r" median_ns "$c_med" ns + emit "$term" latency_under_load control "$r" p99_ns "$c_p99" ns + emit "$term" latency_under_load load "$r" median_ns "$l_med" ns + emit "$term" latency_under_load load "$r" p95_ns "$l_p95" ns + emit "$term" latency_under_load load "$r" p99_ns "$l_p99" ns + emit "$term" latency_under_load load "$r" max_ns "$l_max" ns + mibps_samples="$mibps_samples$mibps\n" + case "$overlap" in + verified*) ;; + *) overlap_fail=$((overlap_fail + 1)) ;; + esac + c_med_us="$(awk -v n="$c_med" 'BEGIN{printf "%.1f", n/1000}')" + l_med_us="$(awk -v n="$l_med" 'BEGIN{printf "%.1f", n/1000}')" + l_p99_us="$(awk -v n="$l_p99" 'BEGIN{printf "%.1f", n/1000}')" + l_max_us="$(awk -v n="$l_max" 'BEGIN{printf "%.1f", n/1000}')" + echo " rep $r: control median ${c_med_us}us | under load median ${l_med_us}us p99 ${l_p99_us}us max ${l_max_us}us | sustained ${mibps} MiB/s | overlap $overlap" + done + if [ "$got" -ge 1 ]; then + set -- $(pooled_stats "$ctrl_pooled") + pcmed="${1:-0}"; pcp95="${2:-0}"; pcp99="${3:-0}"; pcmax="${4:-0}"; pccnt="${5:-0}" + set -- $(pooled_stats "$load_pooled") + plmed="${1:-0}"; plp95="${2:-0}"; plp99="${3:-0}"; plmax="${4:-0}"; plcnt="${5:-0}" + mibps_med="$(printf "$mibps_samples" | median_f)" + degradation="$(awk -v l="$plmed" -v c="$pcmed" 'BEGIN{ if(c>0) printf "%.2f", l/c; else print "n/a" }')" + emit "$term" latency_under_load control pooled pooled_median_ns "$pcmed" ns + emit "$term" latency_under_load control pooled pooled_p95_ns "$pcp95" ns + emit "$term" latency_under_load control pooled pooled_p99_ns "$pcp99" ns + emit "$term" latency_under_load control pooled pooled_max_ns "$pcmax" ns + emit "$term" latency_under_load control pooled pooled_count "$pccnt" count + emit "$term" latency_under_load load pooled pooled_median_ns "$plmed" ns + emit "$term" latency_under_load load pooled pooled_p95_ns "$plp95" ns + emit "$term" latency_under_load load pooled pooled_p99_ns "$plp99" ns + emit "$term" latency_under_load load pooled pooled_max_ns "$plmax" ns + emit "$term" latency_under_load load pooled pooled_count "$plcnt" count + emit "$term" latency_under_load load pooled sustained_mib_per_s_median "$mibps_med" mib_s + emit "$term" latency_under_load load pooled degradation_factor_median_vs_control "$degradation" ratio + emit "$term" latency_under_load load pooled reps_ok "$got" count + emit "$term" latency_under_load load pooled reps_overlap_not_verified "$overlap_fail" count + pcmed_us="$(awk -v n="$pcmed" 'BEGIN{printf "%.1f", n/1000}')" + plmed_us="$(awk -v n="$plmed" 'BEGIN{printf "%.1f", n/1000}')" + plp95_us="$(awk -v n="$plp95" 'BEGIN{printf "%.1f", n/1000}')" + plp99_us="$(awk -v n="$plp99" 'BEGIN{printf "%.1f", n/1000}')" + plmax_us="$(awk -v n="$plmax" 'BEGIN{printf "%.1f", n/1000}')" + echo " pooled ($got/$LOADLAT_REPS launches ok, $overlap_fail overlap-unverified): control ${pcmed_us}us (n=$pccnt) -> under load median ${plmed_us}us p95 ${plp95_us}us p99 ${plp99_us}us max ${plmax_us}us (n=$plcnt), ${degradation}x degradation, sustained ${mibps_med} MiB/s" + else + emit "$term" latency_under_load - - status UNMEASURED "no-dsr-reply-in-any-rep" + echo " UNMEASURED (no rep produced a usable DSR reply)" + fi +done +fi # axis: latency-under-load + # Latency + startup are condition-independent; skip them under --equalize # (the equalized re-run targets the render-sensitive throughput+scroll axes). if [ "$EQUALIZE" != 1 ]; then diff --git a/bench/tools/bulk_produce.c b/bench/tools/bulk_produce.c new file mode 100644 index 0000000..eced21c --- /dev/null +++ b/bench/tools/bulk_produce.c @@ -0,0 +1,131 @@ +// bulk_produce — bounded-duration bulk pty writer for the +// "latency-under-load" bench axis (bench/METHODOLOGY.md, axis 8). +// +// Streams a file's bytes to stdout in a loop, under the SAME pty flow +// control as the throughput axis's `cat ` (write() blocks until the +// terminal drains its kernel buffer), until a wall-clock DURATION elapses — +// not until N bytes are sent. This is the "bounded-duration equivalent" of +// the 150MB `cat` workload: a single `cat` of 150MB can complete in well +// under a second on a fast terminal, too short a window to hold a +// concurrent DSR probe through a meaningful number of iterations, so this +// tool re-streams the file for as long as the caller needs contention held. +// +// Two outputs: +// - result-file: " " written once, +// at exit — / gives the harness sustained MiB/s +// over the exact window this process was actually writing; / +// (CLOCK_MONOTONIC, same clock nowns/dsr_probe use) let the +// harness verify this process's own active interval CONTAINED another +// process's interval (e.g. the concurrent DSR probe's), which is a +// cadence-independent overlap proof — unlike counting progress-log +// points, which can under-sample a probe window shorter than the +// progress cadence. +// - progress-file (optional): " " +// appended roughly every 200ms while writing. This is NOT used for the +// throughput number (that comes from the exact start/end result-file +// pair) — it exists so the harness can verify, after the fact, that the +// producer was STILL ACTIVELY WRITING throughout the concurrent DSR +// probe's own wall-clock window (see run_latency_under_load in +// run_all.sh), rather than assuming overlap from the launch order alone. +// +// Usage: bulk_produce [progress-file] +#include +#include +#include +#include +#include +#include + +static long long now_ns(void) { + struct timespec ts; + clock_gettime(CLOCK_MONOTONIC, &ts); + return (long long)ts.tv_sec * 1000000000LL + ts.tv_nsec; +} + +static int write_all(int fd, const char *buf, size_t len) { + const char *p = buf; + while (len > 0) { + ssize_t w = write(fd, p, len); + if (w < 0) { + if (errno == EINTR) continue; + return -1; + } + p += w; + len -= (size_t)w; + } + return 0; +} + +int main(int argc, char **argv) { + if (argc < 4) { + fprintf(stderr, + "usage: %s [progress-file]\n", + argv[0]); + return 64; + } + double secs = atof(argv[1]); + const char *file = argv[2]; + const char *result_path = argv[3]; + const char *progress_path = (argc >= 5 && argv[4][0] != '\0') ? argv[4] : NULL; + if (secs <= 0) { + fprintf(stderr, "bulk_produce: bad duration '%s'\n", argv[1]); + return 64; + } + + FILE *fp = fopen(file, "rb"); + if (!fp) { perror("bulk_produce: fopen"); return 1; } + if (fseek(fp, 0, SEEK_END) != 0) { fclose(fp); return 1; } + long fsize = ftell(fp); + if (fsize <= 0) { fclose(fp); fprintf(stderr, "bulk_produce: empty file\n"); return 1; } + rewind(fp); + char *buf = malloc((size_t)fsize); + if (!buf) { fclose(fp); return 1; } + if (fread(buf, 1, (size_t)fsize, fp) != (size_t)fsize) { + fclose(fp); free(buf); + fprintf(stderr, "bulk_produce: short read\n"); + return 1; + } + fclose(fp); + + FILE *pf = NULL; + if (progress_path) { + pf = fopen(progress_path, "w"); + if (pf) setvbuf(pf, NULL, _IOLBF, 0); // line-buffered: partial log survives a hard kill + } + + long long t0 = now_ns(); + long long deadline = t0 + (long long)(secs * 1e9); + long long total = 0; + long long last_progress = t0; + const long long progress_interval_ns = 200000000LL; // ~200ms cadence + + while (now_ns() < deadline) { + if (write_all(STDOUT_FILENO, buf, (size_t)fsize) < 0) break; // pty/pipe gone + total += fsize; + long long t = now_ns(); + if (pf && (t - last_progress) >= progress_interval_ns) { + fprintf(pf, "%lld %lld\n", t, total); + last_progress = t; + } + } + long long t1 = now_ns(); + if (pf) { + fprintf(pf, "%lld %lld\n", t1, total); // final point, always recorded + fclose(pf); + } + + // " " — t0/t1 are the producer's own + // measured CLOCK_MONOTONIC start/finish, the same clock nowns/dsr_probe + // use in the same pty child, so a caller can directly check whether + // this process's active interval CONTAINED another process's interval + // (see run_all.sh: run_latency_under_load's overlap check) instead of + // relying on the coarser progress-log cadence, which can be wider than + // a short probe window and would otherwise false-negative real overlap. + FILE *rf = fopen(result_path, "w"); + if (rf) { + fprintf(rf, "%lld %lld %lld %lld\n", total, t1 - t0, t0, t1); + fclose(rf); + } + free(buf); + return 0; +} diff --git a/bench/wrapper.sh b/bench/wrapper.sh index 61cd160..cae5021 100755 --- a/bench/wrapper.sh +++ b/bench/wrapper.sh @@ -38,6 +38,32 @@ # pty child exits would otherwise vanish before sampling. # NOA_CYCLES (longevity) number of flood+idle cycles (default 5) # NOA_IDLE_S (longevity) idle seconds between cycles (default 3) +# NOA_MODE=latload (latency-under-load axis, bench/METHODOLOGY.md axis 8): +# runs a no-load DSR control, then starts a bounded-duration bulk +# producer against the SAME pty and runs a second DSR probe while it is +# still actively writing — the pairing (throughput, latency-under-load) +# is the whole point of the axis. +# NOA_BULK path to the bulk_produce tool +# NOA_BULK_FILE file bulk_produce streams in a loop +# NOA_BULK_DURATION seconds bulk_produce keeps writing (bounded, not +# byte-count-bounded — see bulk_produce.c) +# NOA_BULK_RESULT file bulk_produce writes " " to +# NOA_BULK_PROGRESS (optional) file bulk_produce appends +# " " to every ~200ms, so +# the harness can verify the producer was still +# writing during the load-probe's own window +# NOA_LOAD_WARMUP_DISCARD_S seconds to let the producer ramp before the +# load probe starts sampling +# NOA_CTRL_ITERS/NOA_CTRL_WARMUP no-load control probe iteration counts +# NOA_PROBE_ITERS/NOA_PROBE_WARMUP in-load probe iteration counts +# NOA_RESULT_NOLOAD/NOA_SAMPLES_NOLOAD control probe outputs +# NOA_RESULT/NOA_SAMPLES in-load probe outputs (same vars +# the plain `latency` mode uses) +# NOA_OVERLAP_MARKER file wrapper writes " " +# to (CLOCK_MONOTONIC via nowns) — the load probe's +# own wall-clock window, checked against +# NOA_BULK_PROGRESS to verify overlap actually +# happened rather than assuming it from launch order # wait_go — block until the harness signals final window geometry (NOA_GO # file appears). No-op when NOA_GO is unset (memory/load scenarios reuse the @@ -103,6 +129,36 @@ case "$NOA_MODE" in done : > "$NOA_SENTINEL" ;; + latload) + wait_go + # 1. no-load control: baseline echo latency with nothing else running, + # in the SAME launch/session as the load probe below (same-conditions + # pairing — see METHODOLOGY.md axis 8 "Report the pairing"). + "$NOA_PROBE" "${NOA_CTRL_ITERS:-300}" "${NOA_CTRL_WARMUP:-30}" \ + "$NOA_RESULT_NOLOAD" "${NOA_SAMPLES_NOLOAD:-}" + # 2. start the bulk producer against the SAME pty, in the background. + "$NOA_BULK" "${NOA_BULK_DURATION:-20}" "$NOA_BULK_FILE" "$NOA_BULK_RESULT" \ + "${NOA_BULK_PROGRESS:-}" & + bulk_pid=$! + # 3. let contention ramp up before sampling (discard the startup + # transient of the producer's write loop reaching steady pty + # backpressure — a fraction of a second in practice; the seconds-long + # discard here is a deliberate margin, not the unrelated 5-40s + # GPU-driver-pool reclaim window documented for the memory axes). + sleep "${NOA_LOAD_WARMUP_DISCARD_S:-2}" + # 4. echo latency WHILE the bulk producer is actively writing. The + # wrapper's own before/after timestamps are recorded so the harness + # can check the producer's progress log against this exact window. + probe_start=$("$NOA_NOWNS") + "$NOA_PROBE" "${NOA_PROBE_ITERS:-200}" "${NOA_PROBE_WARMUP:-20}" \ + "$NOA_RESULT" "${NOA_SAMPLES:-}" + probe_end=$("$NOA_NOWNS") + if [ -n "${NOA_OVERLAP_MARKER:-}" ]; then + printf '%s %s\n' "$probe_start" "$probe_end" > "$NOA_OVERLAP_MARKER" + fi + wait "$bulk_pid" 2>/dev/null + : > "$NOA_SENTINEL" + ;; *) printf 'wrapper: unknown NOA_MODE=%s\n' "$NOA_MODE" >&2 exit 64