Skip to content

feat(benchmark): add analysis plots for phases, operations, auto and the link - #204

Merged
eiserv merged 1 commit into
mainfrom
feat/benchmark-analysis-plots
Aug 17, 2026
Merged

feat(benchmark): add analysis plots for phases, operations, auto and the link#204
eiserv merged 1 commit into
mainfrom
feat/benchmark-analysis-plots

Conversation

@eiserv

@eiserv eiserv commented Aug 17, 2026

Copy link
Copy Markdown
Owner

The optional Python layer under benchmarks/analysis/ read two of the stored
files and drew two kinds of picture. Everything else the harness measures (the
phase breakdown, the per round-trip costs, the delete sweeps, the link probe,
the auto policy, the drift canary) was only readable as JSON. This adds the
plots for those, plus the structure to keep adding them.

What is new

Command What it draws Reads
heatmap the grid, now with the best cell boxed and the edge of the grid called out matrix.csv
scaling new, the same cells as curves against a perfect-scaling reference matrix.json
auto new, what easySFTP picks for itself against the fastest cell matrix.json
canary new, whether the line held still for the whole sweep matrix.json
phases new, where a deployment's wall clock goes either
operations new, per round-trip cost and share of the work either
deletes new, the clean deployment, the only measurement of deletion either
link new, measured throughput against the link probe's own control either
trend across releases, now two metrics and one marker per runner trend.csv
report new, every plot the given files support plus a report.md index both

Common options on every command: --scenario, --profile, --out,
--format png|svg|pdf, --dpi, plus --metric for the heatmap and
--include-deletes for the phases.

"either" means a stored run or a sweep. For a run it draws the measured result;
for a sweep it draws the best cell per scenario and profile, taken from
scaling[].best, since a sweep has hundreds of cells and only the one that won
is worth taking apart.

What it says about the current numbers

Nothing here changes a measurement, but a few things the JSON already contained
are now visible at a glance:

  • The auto policy is the open question. Against the newest sweep, the
    regret of auto is 130% to 190% on calib-100x64k and small, on every
    profile, and under 15% only on single and sync. That is issue "concurrency: auto" is a hardcoded 4, not adaptive; the name promises something it does not do #156's
    ground truth, and the plot puts the picked coordinate (read back from the
    run's own counters) on every bar.
  • The best cell keeps landing on the largest swept concurrency, so those
    optima are bounded from below, not measured. The heatmap boxes it and both it
    and scaling say so in words.
  • Deletion does not scale with either axis (roughly 44 to 53 deletes/s
    across the whole grid), which is the expected shape: everything outside the
    per-file upload path runs over one connection through session.do.
  • The best cells reach 62% to 82% of the link's own N-stream control, and
    small reaches 8% of it on the release run, which separates the line from
    easySFTP the way the probe was added to.

Structure and safety rails

  • Schema reading moved into benchdata.py, so a plot no longer relearns it.
    Every field it reads is optional: the stored results span several schema
    versions (v1 has no phases and no operations, the pre-Benchmarks: record the link profile, keep the metrics we already collect, and cover redeploy/sync/delete before deriving an auto-config policy #184 sweeps have no
    link profile, no request_concurrency and no auto), and all of them still
    load.
  • Every figure carries its provenance and, in red, the caveats the stored file
    justifies. The load-bearing one: a run measured with tc unavailable has
    profile names that say what was asked for, not what happened, and the
    newest sweep is exactly that run.
  • scaling picks its axes from what actually varied rather than assuming
    concurrency, because scenario.AxisFor caps both axes at the file count,
    so single only sweeps request_concurrency.
  • The layer stays one way: canonical files in, pictures out, nothing in
    benchmarks/ written outside the output directory and nothing reaching into
    internal/ or the harness.

Tests

benchmarks/analysis/test_plot.py, 20 cases, standard-library unittest, no
new dependency and no benchmark run:

python -m unittest discover -s benchmarks/analysis

It loads every result committed under benchmarks/, so a reader that only
understands the newest schema fails instead of quietly dropping the rest, and
it draws every command into a temporary directory. Beyond that, every command
was run against every stored file (172 figures, no failures).

Go was not touched, so go test ./... is unaffected by this change.

Committed visuals

out/ stays ignored except for the gallery benchmarks/analysis/README.md
refers to, and the README now prints the exact command under each image. The
gallery is regenerated from the newest sweep (matrix-20260816T125322Z-main)
and v3.5.0; the five stale heatmaps of the 2026-07-30 sweep, which no document
referred to any more, are dropped.

🤖 Generated with Claude Code

…the link

The optional Python layer read two of the stored files and drew two kinds of
picture. Everything else the harness measures (the phase breakdown, the per
round-trip costs, the delete sweeps, the link probe, the auto policy, the drift
canary) was only readable as JSON.

Eight commands now, plus a report that draws every one the given files support
and writes a Markdown index of it:

  heatmap     the grid, with the best cell boxed and edge-of-grid called out
  scaling     the same cells as curves against a perfect-scaling reference
  auto        what easySFTP picks for itself against the fastest cell (#156)
  canary      whether the line held still for the whole sweep
  phases      where a deployment's wall clock goes
  operations  per round-trip cost and share of the work
  deletes     the clean deployment, the only measurement of deletion
  link        measured throughput against the link probe's own control
  trend       across releases, one marker per runner
  report      all of the above plus report.md

Schema reading moved into benchdata.py, so a plot no longer relearns it, and
every field it reads is optional: the stored results span several schema
versions and the older ones must keep loading. Every figure carries its
provenance and, in red, the caveats the stored file justifies, the load-bearing
one being a run measured with tc unavailable, whose profile names say what was
asked for and not what happened.

test_plot.py checks both halves offline: every result committed under
benchmarks/ is loaded, so a reader that only understands the newest schema
fails instead of quietly dropping the rest, and every command is drawn into a
temporary directory.

The committed gallery is regenerated from the newest sweep and v3.5.0; the
stale heatmaps of the 2026-07-30 sweep, which no document referred to any more,
are dropped.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@eiserv
eiserv merged commit 359421d into main Aug 17, 2026
13 checks passed
@eiserv
eiserv deleted the feat/benchmark-analysis-plots branch August 17, 2026 23:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant