feat(compare): compare load-test reports and enforce regression budgets (#37) + release v1.6.0

[ragilhadi]

Closes #37.

Flux produced a report per run, but nothing could say whether a change made things worse. This adds a compare subcommand that reads a baseline and a candidate JSON report and decides against configured budgets, so a performance regression can fail a CI job.

flux --config perf.yaml --output-json artifacts/current.json

flux compare artifacts/baseline.json artifacts/current.json \
  --max-p95-regression 10% --max-error-rate-increase 0.5

What changed

• Versioned report schema. Reports now carry schema_version: 1 and a status_codes distribution in the summary. Comparison reads reports written before either existed as schema v0, and rejects a report from a newer schema with a message telling the user to upgrade Flux. When one side predates status recording, the status table is skipped with a note instead of reporting every status as new.
• Deltas. Aggregate and per-scenario: total/successful/failed requests, throughput, error rate, mean and p50/p90/p95/p99 latency, plus the status distribution with each report's own share as the denominator. Scenarios are matched by name; scenarios added or removed between runs are listed explicitly and excluded from per-scenario budgets.
• Budgets. --max-{mean,p50,p90,p95,p99}-regression and --max-throughput-drop take either a percentage of the baseline (10%) or an absolute movement in the metric's own unit (25ms). --max-error-rate-increase is in percentage points. --per-scenario-budgets extends the same budgets to every scenario present in both reports.
• Exit codes. 0 every configured budget met (or none configured), 1 a budget was exceeded, 2 the comparison could not be made at all (missing or unreadable report, unparseable budget, unknown schema).
• Denominator handling. Percentage change always divides by the baseline. A zero baseline has no denominator, so the change prints as n/a (baseline 0) rather than a fabricated number, and a percentage budget treats any movement in the bad direction from zero as exceeded. Status shares print n/a when a run made no requests.
• CI artifacts. --output-json writes the full comparison, --output-markdown writes tables ready for a job summary or PR comment. Both are written before the exit code is decided, so they exist even when the run fails.
• Honest reporting. Every rendering carries the caveat that a comparison of two runs is an operational budget check, not a statistical test.

Example terminal output:

Aggregate Deltas:
  Metric                       Baseline      Candidate          Delta           Change       Budget
  throughput                33.30 req/s    30.00 req/s    -3.30 req/s            -9.9%            -
  error rate                      0.20%          1.10%        +0.90pp          +450.0%        0.5pp
  p95 latency                     110ms          140ms          +30ms           +27.3%          10%

❌ FAIL — regression budgets exceeded:
  • aggregate error rate: +0.90pp exceeds the 0.5pp increase budget
  • aggregate p95 latency: +30ms (+27.3%) exceeds the 10% increase budget

Release

Bumps vars/version to v1.6.0. Cargo.toml moves from 0.1.0 to 1.6.0 as well, so flux --version reports something true — earlier releases left it at the default.

Docs

README gains a "Comparing Reports and Regression Budgets" section: the workflow, a budget flag table, denominator and added/removed-scenario semantics, exit codes, a GitHub Actions example that fails the job and publishes the Markdown comparison to the step summary, and guidance on picking budgets wide enough to clear observed run-to-run noise. The JSON report example now shows schema_version and status_codes.

Testing

• cargo fmt --check, cargo clippy --all-targets --all-features -- -D warnings clean.
• cargo test: 111 passing, up from 90. New coverage for schema parsing (legacy, current, newer-than-supported, malformed), percentage calculation, threshold decisions in both directions, zero and empty baselines, added/removed scenarios, per-scenario budget opt-in, status-code deltas, and the JSON/Markdown renderings.
• Ran the built binary end to end against hand-written baseline and candidate reports, including a legacy report with no schema_version and no status_codes, confirming exit code 1 on a breach and 0 on a pass.

IMPLEMENTATION.md is untouched — it was already stale before this change (missing dashboard.rs, prometheus.rs, redact.rs, cancel.rs) and refreshing it belongs in its own change.

🤖 Generated with Claude Code

https://claude.ai/code/session_012ATPLmE2H9oRipANby2d9H

---

Generated by Claude Code