feat(benchmark): add analysis plots for phases, operations, auto and the link - #204
Merged
Conversation
…the link The optional Python layer read two of the stored files and drew two kinds of picture. Everything else the harness measures (the phase breakdown, the per round-trip costs, the delete sweeps, the link probe, the auto policy, the drift canary) was only readable as JSON. Eight commands now, plus a report that draws every one the given files support and writes a Markdown index of it: heatmap the grid, with the best cell boxed and edge-of-grid called out scaling the same cells as curves against a perfect-scaling reference auto what easySFTP picks for itself against the fastest cell (#156) canary whether the line held still for the whole sweep phases where a deployment's wall clock goes operations per round-trip cost and share of the work deletes the clean deployment, the only measurement of deletion link measured throughput against the link probe's own control trend across releases, one marker per runner report all of the above plus report.md Schema reading moved into benchdata.py, so a plot no longer relearns it, and every field it reads is optional: the stored results span several schema versions and the older ones must keep loading. Every figure carries its provenance and, in red, the caveats the stored file justifies, the load-bearing one being a run measured with tc unavailable, whose profile names say what was asked for and not what happened. test_plot.py checks both halves offline: every result committed under benchmarks/ is loaded, so a reader that only understands the newest schema fails instead of quietly dropping the rest, and every command is drawn into a temporary directory. The committed gallery is regenerated from the newest sweep and v3.5.0; the stale heatmaps of the 2026-07-30 sweep, which no document referred to any more, are dropped. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The optional Python layer under
benchmarks/analysis/read two of the storedfiles and drew two kinds of picture. Everything else the harness measures (the
phase breakdown, the per round-trip costs, the delete sweeps, the link probe,
the auto policy, the drift canary) was only readable as JSON. This adds the
plots for those, plus the structure to keep adding them.
What is new
heatmapmatrix.csvscalingmatrix.jsonautomatrix.jsoncanarymatrix.jsonphasesoperationsdeleteslinktrendtrend.csvreportreport.mdindexCommon options on every command:
--scenario,--profile,--out,--format png|svg|pdf,--dpi, plus--metricfor the heatmap and--include-deletesfor the phases."either" means a stored run or a sweep. For a run it draws the measured result;
for a sweep it draws the best cell per scenario and profile, taken from
scaling[].best, since a sweep has hundreds of cells and only the one that wonis worth taking apart.
What it says about the current numbers
Nothing here changes a measurement, but a few things the JSON already contained
are now visible at a glance:
regret of
autois 130% to 190% oncalib-100x64kandsmall, on everyprofile, and under 15% only on
singleandsync. That is issue "concurrency: auto" is a hardcoded 4, not adaptive; the name promises something it does not do #156'sground truth, and the plot puts the picked coordinate (read back from the
run's own counters) on every bar.
concurrency, so thoseoptima are bounded from below, not measured. The heatmap boxes it and both it
and
scalingsay so in words.across the whole grid), which is the expected shape: everything outside the
per-file upload path runs over one connection through
session.do.smallreaches 8% of it on the release run, which separates the line fromeasySFTP the way the probe was added to.
Structure and safety rails
benchdata.py, so a plot no longer relearns it.Every field it reads is optional: the stored results span several schema
versions (v1 has no phases and no operations, the pre-Benchmarks: record the link profile, keep the metrics we already collect, and cover redeploy/sync/delete before deriving an auto-config policy #184 sweeps have no
link profile, no
request_concurrencyand noauto), and all of them stillload.
justifies. The load-bearing one: a run measured with
tcunavailable hasprofile names that say what was asked for, not what happened, and the
newest sweep is exactly that run.
scalingpicks its axes from what actually varied rather than assumingconcurrency, becausescenario.AxisForcaps both axes at the file count,so
singleonly sweepsrequest_concurrency.benchmarks/written outside the output directory and nothing reaching intointernal/or the harness.Tests
benchmarks/analysis/test_plot.py, 20 cases, standard-libraryunittest, nonew dependency and no benchmark run:
It loads every result committed under
benchmarks/, so a reader that onlyunderstands the newest schema fails instead of quietly dropping the rest, and
it draws every command into a temporary directory. Beyond that, every command
was run against every stored file (172 figures, no failures).
Go was not touched, so
go test ./...is unaffected by this change.Committed visuals
out/stays ignored except for the gallerybenchmarks/analysis/README.mdrefers to, and the README now prints the exact command under each image. The
gallery is regenerated from the newest sweep (
matrix-20260816T125322Z-main)and v3.5.0; the five stale heatmaps of the 2026-07-30 sweep, which no document
referred to any more, are dropped.
🤖 Generated with Claude Code