Part of #33. Makes A/B (PR-vs-main) comparisons meaningful.
Problem
Run summary emits ~3 gauges today. Two runs aren't comparable without pinning the SUT build (the variable) and every load/workload knob (the things held equal).
Approach
Schema-versioned JSON artifact with:
- Identity:
run_id, seiload_git_sha, config_sha256, seed (settable), sut_build (chain web3_clientVersion/commit), methodology_version (bump when latency semantics change, so pre/post runs aren't silently A/B'd).
- Load model:
arrival_model, offered_tps_target, offered_tps_achieved, duration, actual_duration, worker_pool_size.
- Workload:
num_accounts, recordcount/num_hot_keys, zipf_theta, size_buckets.
- Results: offered/accepted/committed/reverted/dropped,
goodput_ratio, submit_to_inclusion_latency HdrHistogram (p50/p90/p99/p99.9/max), schedule_lag_p99.
Acceptance criteria
- One JSON object per run, schema-versioned; A/B diff is a field-by-field compare keyed on
(zipf_theta, offered_tps_target).
- HdrHistogram lives in the JSON (not Prometheus — downsampling erodes the tail).
Design: https://github.com/sei-protocol/platform/blob/main/docs/designs/sei-load-workload-modeler.md
Part of #33. Makes A/B (PR-vs-main) comparisons meaningful.
Problem
Run summary emits ~3 gauges today. Two runs aren't comparable without pinning the SUT build (the variable) and every load/workload knob (the things held equal).
Approach
Schema-versioned JSON artifact with:
run_id,seiload_git_sha,config_sha256,seed(settable),sut_build(chainweb3_clientVersion/commit),methodology_version(bump when latency semantics change, so pre/post runs aren't silently A/B'd).arrival_model,offered_tps_target,offered_tps_achieved,duration,actual_duration,worker_pool_size.num_accounts,recordcount/num_hot_keys,zipf_theta,size_buckets.goodput_ratio,submit_to_inclusion_latencyHdrHistogram (p50/p90/p99/p99.9/max),schedule_lag_p99.Acceptance criteria
(zipf_theta, offered_tps_target).Design: https://github.com/sei-protocol/platform/blob/main/docs/designs/sei-load-workload-modeler.md