diff --git a/.grok/skills/implement-experiment/SKILL.md b/.grok/skills/implement-experiment/SKILL.md index db243a0..1444675 100644 --- a/.grok/skills/implement-experiment/SKILL.md +++ b/.grok/skills/implement-experiment/SKILL.md @@ -40,7 +40,7 @@ export PATH="${HOME}/.local/go/bin:${HOME}/.cargo/bin:${HOME}/.dotnet:${HOME}/.l 2. **One question per folder.** Number it `NN-short-kebab` (next free `NN` after existing dirs). 3. **Edit `experiment.yaml` only** for sample, languages, libraries, run mode, grouping cut-offs. `run.yaml` is generated. Do not keep a second library list. 4. **Shared sample** at the experiment root (`sample.json`). Language folders hold **results** only (`results.md`, `results.json`). **Do not commit experiment logs** (CSVs under `/logs/`). Those stay on the machine that ran the timing. Saved results are enough. -5. **Do not compare write times across languages.** Size is the only roughly fair cross-language number, and only when both sides write the same field description. +5. **Do not compare times across languages.** Different runtimes. Directional only. Do not invent a size contest — payload bytes are the **data type**, not a library score. 6. **Do not crown a single winner.** Use `top_group` (similar / close / slower via Cliff’s delta vs the fastest library in the comparison set). Not “top 5%.” 7. **Textbook language** in `experiment.yaml` (`story.example`, `story.tradeoff`, `story.why`), `README.md`, `results.md`, the Dashboard, and the PLAN update. Write for a high-school student: complete sentences, no slang, no telegraphic fragments, no unexplained jargon. The Dashboard copies `story` from `experiment.yaml`. 8. **Do not overwrite** published site tables (`docs//results.md`) or dashboard `*_latest.json.gz` unless the user asked to publish suite numbers. diff --git a/.grok/skills/improve-docs/SKILL.md b/.grok/skills/improve-docs/SKILL.md index 10c77c5..12de277 100644 --- a/.grok/skills/improve-docs/SKILL.md +++ b/.grok/skills/improve-docs/SKILL.md @@ -87,7 +87,7 @@ Cut scope creep, ensure dismissible UI is not permanent noise, prefer small diff - Edit sources under `dashboard/` then **`npm run build`** in `dashboard/` so `docs/dashboard/` updates. - **README:** follow `README_EDITING.md` — short lede, compact audience table, section order Who → languages → Try it → Quick start; no slogan/meta CTA prose; no unsolicited role expansion. - MkDocs storefront (`docs/index.md`) only if asked or if it clearly contradicts a user-approved README fact. -- Follow `STYLE.md` (simple, no new frameworks, terminology: **data type** not fixture). +- Follow `STYLE.md` (simple, no new frameworks). **Terminology: always “data type”, never “fixture”** — in user copy, comments, identifiers, and JSON keys. ### 2.5 Critique the implementation diff --git a/.grok/skills/improve-docs/references/STYLE.md b/.grok/skills/improve-docs/references/STYLE.md index 405b454..9e09833 100644 --- a/.grok/skills/improve-docs/references/STYLE.md +++ b/.grok/skills/improve-docs/references/STYLE.md @@ -2,6 +2,16 @@ Keep it simple. Prefer delete and clarify over decorate. +## Terminology (binding) + +| Say | Never say | +|-----|-----------| +| **data type** (`message`, `document`, `telemetry`, `strings`, `event`) | fixture, fixtures, fixtureKey, `dataset.fixtures` | + +This is the catalog payload shape. Use **data type** in README, Dashboard copy, skills, comments, function names, and new JSON keys. Do not keep `fixture*` as an internal alias. Old `configs.json` may still contain `fixtures`; read it as a fallback, write `data_types`. + +Do not measure or display payload **size** as a library result. Size is the data type, not a score. + --- ## Scope of “docs” in this skill @@ -18,8 +28,9 @@ Keep it simple. Prefer delete and clarify over decorate. ## Content style - **One idea per paragraph.** Prefer tables for role/path matrices **on the site / Dashboard**, not by fattening the root README. -- **User terms:** data type, mode (bytes/stream), ops/s, latency, median size, Pareto, baseline. -- **Avoid in user copy:** fixture (internal OK), harness (prefer benchmark runner), unexplained IQR/P95. +- **User terms:** data type, mode (bytes/stream), ops/s, latency, Pareto, baseline. +- **Never say “fixture”.** The catalog entry is a **data type** (`message`, `document`, …). Say it in docs, Dashboard copy, skills, comments, identifiers, and JSON keys. Do not keep `fixture*` as an internal alias “until later.” +- **Avoid in user copy:** fixture, median size (we do not measure payload size as a result), harness (prefer benchmark runner), unexplained IQR/P95. - **Honesty line** when ranks appear: within one language; cross-lang directional — prefer **one** place (e.g. Statistics / Method), not a second essay block on README. - **Links:** prefer site paths that match MkDocs nav labels (Dashboard, Learn, Method). Avoid “storefront” / “CTA” wording in user-facing labels. - **No emoji spam** in product UI; README badges OK. diff --git a/README.md b/README.md index ff9403c..ac67747 100644 --- a/README.md +++ b/README.md @@ -131,9 +131,9 @@ Every language writes the same columns (nanoseconds). Domain mapping: | `TimeDeq` | Dequeue ns | | `TimeHandoff` | Handoff ns | | `Pattern` | `bytes` = **SPSC**, `stream` = **MPMC** (not I/O) | -| `Size` | Payload bytes | +| `CpuTimeNs` | Process CPU time (spin vs block) | -See [architecture](https://leo-gan.github.io/queue-benchmark/analysis/architecture/). +See [Metrics](https://leo-gan.github.io/queue-benchmark/analysis/METRICS/) and [architecture](https://leo-gan.github.io/queue-benchmark/analysis/architecture/). --- diff --git a/analysis/src/benchmark_analysis/abi.py b/analysis/src/benchmark_analysis/abi.py index db5caea..1e16c10 100644 --- a/analysis/src/benchmark_analysis/abi.py +++ b/analysis/src/benchmark_analysis/abi.py @@ -20,7 +20,6 @@ "LibraryVersion", "TimeEnq", "TimeDeq", - "Size", "TimeHandoff", "OpPerSecEnq", "OpPerSecDeq", @@ -29,8 +28,6 @@ "FidelityScore", "DataTypeInstanceCount", "TypeConfigHash", - "SizeGzip", - "SizeZstd", "NativeKind", "StreamMode", "RunOrder", @@ -133,7 +130,6 @@ def canonicalize_csv_record(row: Mapping[str, Any], language: str = "") -> Dict[ "LibraryName": str(pick(row, "LibraryName", "") or ""), "TimeEnq": _as_int(pick(row, "TimeEnq", 0)), "TimeDeq": _as_int(pick(row, "TimeDeq", 0)), - "Size": _as_int(row.get("Size", 0)), "TimeHandoff": _as_int(pick(row, "TimeHandoff", 0)), "OpPerSecEnq": _as_float(pick(row, "OpPerSecEnq", 0)), "OpPerSecDeq": _as_float(pick(row, "OpPerSecDeq", 0)), @@ -146,8 +142,6 @@ def canonicalize_csv_record(row: Mapping[str, Any], language: str = "") -> Dict[ "MemoryPeakBytes", "CpuTimeNs", "DataTypeInstanceCount", - "SizeGzip", - "SizeZstd", "RunOrder", "SchedulePosition", ) diff --git a/analysis/src/benchmark_analysis/environment.py b/analysis/src/benchmark_analysis/environment.py index 3673207..6d14353 100644 --- a/analysis/src/benchmark_analysis/environment.py +++ b/analysis/src/benchmark_analysis/environment.py @@ -3,7 +3,7 @@ Writes a ``*.configs.json`` sidecar beside the result CSV: - ``environment`` — hardware, OS, runtimes, git (preferred) -- ``dataset`` — optional: seed, fixtures, repetitions (best-effort) +- ``dataset`` — optional: seed, data_types, repetitions (best-effort) - ``queues`` — optional: names from the run (best-effort) - ``run`` — optional: mode, metrics profile, timestamp @@ -187,7 +187,7 @@ def _dataset_block() -> Dict[str, Any]: cfg = load_master_config() block["config_path"] = "config/benchmark_config.yaml" - # Catalog is normative for suite fixtures. + # Catalog is normative for suite data types. block["catalog_file"] = dig( cfg, "test_data.catalog_file", dig(cfg, "data_model_v2.catalog_file", "schemas/data_catalog_v2.yaml"), @@ -196,7 +196,7 @@ def _dataset_block() -> Dict[str, Any]: block["test_data_config"] = block["catalog_file"] types = dig(cfg, "test_data.types") or [] if isinstance(types, list) and types: - block["fixtures"] = [ + block["data_types"] = [ { "name": t.get("name"), "category": t.get("category"), diff --git a/analysis/src/benchmark_analysis/metrics_catalog.py b/analysis/src/benchmark_analysis/metrics_catalog.py index 6b349e6..244732a 100644 --- a/analysis/src/benchmark_analysis/metrics_catalog.py +++ b/analysis/src/benchmark_analysis/metrics_catalog.py @@ -17,8 +17,7 @@ "avg_time_deq_ns": "medium", "handoff_mean_ns": "medium", "avg_ops_per_sec": "high", - "median_size_bytes": "high", - "mean_fidelity": "high", + "mean_fidelity": "medium", "library_version": "high", "runs": "high", "handoff_ci_low_ns": "medium", @@ -110,8 +109,9 @@ def rank_by_field(metrics_cfg: Optional[Dict[str, Any]] = None) -> str: ("enq_median_ns", "Median enqueue (µs)", True, False), ("deq_median_ns", "Median dequeue (µs)", True, False), ("avg_ops_per_sec", "Ops/s (from mean)", False, True), - ("median_size_bytes", "Median size (B)", False, False), - # runs / mean_fidelity intentionally omitted from multi-way Summary. + ("handoff_p99_ns", "Handoff p99 (µs)", True, False), + ("msgs_per_cpu_sec", "Msgs / CPU-s", False, True), + # Fidelity is a gate, not a ranked score. ("library_version", "Version", False, None), ("effect_vs_fastest_cliffs_label", "δ vs fastest", False, None), ) diff --git a/analysis/src/benchmark_analysis/regression.py b/analysis/src/benchmark_analysis/regression.py index 841442c..9f87dad 100644 --- a/analysis/src/benchmark_analysis/regression.py +++ b/analysis/src/benchmark_analysis/regression.py @@ -210,7 +210,6 @@ def save_baseline( pick_stats(stat, "handoff_ci_high_ns") or handoff ), "avg_ops_per_sec": float(stat.get("avg_ops_per_sec") or 0.0), - "median_size_bytes": float(stat.get("median_size_bytes") or 0.0), "runs": int(stat.get("runs") or 0), } if store_samples: diff --git a/analysis/src/benchmark_analysis/reports.py b/analysis/src/benchmark_analysis/reports.py index 3bb54d1..5b8340c 100644 --- a/analysis/src/benchmark_analysis/reports.py +++ b/analysis/src/benchmark_analysis/reports.py @@ -56,7 +56,7 @@ def _stat_get(entry: Any, field_id: str) -> Any: _FIXTURE_KEY_RE = re.compile(r"^(.*?)(?:@n=(\d+))+$", re.IGNORECASE) -def _format_fixture_display(label: str) -> str: +def _format_data_type_display(label: str) -> str: """Decode cryptic ``message@n=100`` keys for titles and table headers. Examples: @@ -77,7 +77,7 @@ def _format_fixture_display(label: str) -> str: else: base, n = s, None - # Suite type_ids are lowercase words; legacy fixtures are already Title/Pascal. + # Suite type_ids are lowercase words; older labels may already be Title/Pascal. if base and base == base.lower() and re.fullmatch(r"[a-z][a-z0-9_]*", base): pretty = base.replace("_", " ").title() else: @@ -206,13 +206,13 @@ def _generate_violin_plot( top_n: Optional[int] = None, data_source: str = "", ) -> Optional[str]: - """Generate combined mean-bar + violin figure for one fixture. + """Generate combined mean-bar + violin figure for one data type. Layout (shared Y = queue rank, both linear µs from 0): left — horizontal bars at **mean** enqueue / dequeue (easy ranking; aligns with ops/s) right — split violins of full sample density (spread / shape) - Embeds mapping metadata (fixture, language id, log path, modes, n) in the + Embeds mapping metadata (data type, language id, log path, modes, n) in the title/footer so plots can be tied back to CSV results. """ if melted_df.empty or data_type not in melted_df['TestDataName'].values: @@ -333,8 +333,8 @@ def _generate_violin_plot( leg_v.remove() lang_key = (lang_id or _lang_file_key("", language)).lower().replace("#", "sharp") - safe_fixture = data_type.replace(" ", "_") - img_name = f"{lang_key}_{safe_fixture}.png" + safe_type = data_type.replace(" ", "_") + img_name = f"{lang_key}_{safe_type}.png" src = data_source or f"logs/{lang_key}/benchmark-log.csv" modes = sorted( {str(m) for m in subset.get("Pattern", pd.Series(dtype=str)).dropna().unique()} @@ -344,7 +344,7 @@ def _generate_violin_plot( top_note = f" · Top {int(top_n)}" if top_n and int(top_n) > 0 else "" fig.suptitle( - f"{language or lang_key} · {_format_fixture_display(data_type)}{top_note}", + f"{language or lang_key} · {_format_data_type_display(data_type)}{top_note}", fontsize=12, y=1.02, ) @@ -400,7 +400,7 @@ def _generate_violin_plot( return None -# Latency distributions: always show this many fastest serializers per fixture (all languages). +# Latency distributions: always show this many fastest queues per data type (all languages). VIOLIN_TOP_N_SERIALIZERS = 5 # CSV StringOrStream values → human labels (not "number of bytes") @@ -655,7 +655,7 @@ def _pick_entry(matching: List[Dict]) -> Optional[Dict]: else: cell[(rv, cv)] = None - # all@all-style row average: mean of per-fixture cell values (type × n). + # all@all-style row average: mean of per-data-type cell values (type × n). row_avg: Dict[str, Optional[float]] = {} if include_row_average: for rv in row_vals: @@ -875,7 +875,7 @@ def _category_pivot_md(stats: Dict, lang_id: str, title: str) -> str: "", ] for cat in sorted(by_cat.keys()): - # average ops per serializer across fixtures; rows sorted by name + # average ops per queue across data types; rows sorted by name acc: Dict[str, List[float]] = defaultdict(list) for ser, ops in by_cat[cat]: acc[ser].append(ops) @@ -952,8 +952,9 @@ def _config_section_md(lang_id: str, csv_path: Optional[str]) -> str: ) if doc: ds = doc.get("dataset") if isinstance(doc.get("dataset"), dict) else {} - if ds.get("fixtures"): - names = [f.get("name") for f in ds["fixtures"] if isinstance(f, dict) and f.get("name")] + types = ds.get("data_types") or ds.get("fixtures") or [] + if types: + names = [f.get("name") for f in types if isinstance(f, dict) and f.get("name")] if names: body.append( f"- **Data types (config):** {', '.join(str(n) for n in names)}" @@ -1074,7 +1075,7 @@ def _scientific_summary_md(stats: Dict, profile: str = "multi_way") -> str: if not cols: return "" - # One row per queue: prefer bytes mode, average medians across fixtures if needed + # One row per queue: prefer bytes mode, average medians across data types if needed by_ser: Dict[str, List[Dict]] = {} for e in stats.values(): if not isinstance(e, dict): @@ -1133,7 +1134,7 @@ def _scientific_summary_md(stats: Dict, profile: str = "multi_way") -> str: # Shared K/M scale per column (same rule as ops/s and pivot tables) col_units: Dict[str, tuple] = {} - for field_id in ("avg_ops_per_sec", "median_size_bytes"): + for field_id in ("avg_ops_per_sec", "msgs_per_cpu_sec"): vals_in_col = [cell_vals[(ser, field_id)] for ser in serializers if (ser, field_id) in cell_vals] if vals_in_col: col_units[field_id] = _pick_column_unit(vals_in_col) @@ -1180,7 +1181,7 @@ def _scientific_summary_md(stats: Dict, profile: str = "multi_way") -> str: elif is_time: # ns → µs; fixed-point only (never 1.17e+03) text = _format_sig(num / 1000.0, sig=3) - elif field_id in ("avg_ops_per_sec", "median_size_bytes"): + elif field_id in ("avg_ops_per_sec", "msgs_per_cpu_sec"): # 3 significant digits + shared column K/M (thousands / millions) div, unit = col_units.get(field_id) or _pick_column_unit([num]) text = _format_in_unit(num, div, unit, sig=3) @@ -1393,7 +1394,7 @@ def generate_language_results_pages( "| Term | Meaning |", "|------|---------|", "| **data type** | Sample shape: `message`, `document`, `telemetry`, `strings`, or `event` " - "(CSV `TestDataName`; older text may say “fixture”) |", + "(CSV `TestDataName`) |", "| **bytes mode** | In-memory buffer API (encode to bytes / decode from a buffer). " "On C# this is often the **string** path — see [Modes](../analysis/modes.md). |", "| **stream mode** | Stream-style API (write/read through a stream). " @@ -1453,11 +1454,11 @@ def generate_language_results_pages( base = str(e2.get("test_data") or "") if n not in (None, ""): try: - e2["test_data"] = _format_fixture_display(f"{base}@n={int(n)}") + e2["test_data"] = _format_data_type_display(f"{base}@n={int(n)}") except (TypeError, ValueError): - e2["test_data"] = _format_fixture_display(base) + e2["test_data"] = _format_data_type_display(base) else: - e2["test_data"] = _format_fixture_display(base) + e2["test_data"] = _format_data_type_display(base) display_stats[k] = e2 lines.append("## Summary tables") @@ -1539,7 +1540,7 @@ def generate_language_results_pages( ) lines.append("") for dtype, fname in items: - pretty = _format_fixture_display(dtype) + pretty = _format_data_type_display(dtype) lines.append(f"### {pretty}") lines.append("") lines.append(f"![{pretty}]({plot_rel_from_lang}/{fname}){{ width=\"80%\" }}") @@ -1624,7 +1625,7 @@ def generate_violin_plots( ) violin_images: Dict[str, str] = {} - # lang_id -> {fixture -> plot filename} plus source path for results.md + # lang_id -> {data type -> plot filename} plus source path for results.md plot_meta: Dict[str, Dict] = {} for lang_id in lang_ids: diff --git a/analysis/src/benchmark_analysis/stats.py b/analysis/src/benchmark_analysis/stats.py index e39108d..8c16e67 100644 --- a/analysis/src/benchmark_analysis/stats.py +++ b/analysis/src/benchmark_analysis/stats.py @@ -923,7 +923,6 @@ def compute_statistics( "times_enq": [], "times_deq": [], "times_handoff": [], - "sizes": [], "fidelity": [], "memory_peak": [], "cpu_time_ns": [], @@ -942,7 +941,6 @@ def compute_statistics( if handoff in (None, ""): handoff = (pick(r, "TimeEnq", 0) or 0) + (pick(r, "TimeDeq", 0) or 0) stats[key]["times_handoff"].append(float(handoff)) - stats[key]["sizes"].append(float(r["Size"])) stats[key]["language"] = lang qv = pick(r, "LibraryVersion") if qv not in (None, ""): @@ -994,7 +992,6 @@ def compute_statistics( min_ops = 1e9 / handoff_stats["handoff_max_ns"] if handoff_stats["handoff_max_ns"] > 0 else 0.0 max_ops = 1e9 / handoff_stats["handoff_min_ns"] if handoff_stats["handoff_min_ns"] > 0 else 0.0 - sizes = data["sizes"] # key: (queue, test_data, type_config_hash, instance_count, mode, language) entry = { "library": key[0], @@ -1007,7 +1004,6 @@ def compute_statistics( "avg_time_enq_ns": enq_stats["enq_mean_ns"], "avg_time_deq_ns": deq_stats["deq_mean_ns"], "avg_time_handoff_ns": avg_time_handoff_ns, - "median_size_bytes": float(np.median(sizes)) if sizes else 0.0, "avg_ops_per_sec": avg_ops_per_sec, "min_ops_per_sec": min_ops, "max_ops_per_sec": max_ops, @@ -1201,7 +1197,11 @@ def public_stats_entry(entry: Dict[str, Any]) -> Dict[str, Any]: def compute_pareto_front(groups: List[Dict[str, Any]]) -> List[Dict[str, Any]]: - """Classical 2D Pareto: minimize avg_time_handoff_ns and median_size_bytes.""" + """2D Pareto: minimize handoff time, maximize msgs_per_cpu_sec. + + Payload size is not an axis. When CPU is missing, the front is the + fastest handoff only. + """ front: List[Dict[str, Any]] = [] workloads: Dict[Tuple[Any, Any], List[Dict[str, Any]]] = {} for g in groups: @@ -1210,18 +1210,26 @@ def compute_pareto_front(groups: List[Dict[str, Any]]) -> List[Dict[str, Any]]: for items in workloads.values(): for item in items: t = pick_stats(item, "avg_time_handoff_ns") - s = item.get("median_size_bytes") - if t is None or s is None: + cpu = item.get("msgs_per_cpu_sec") + if t is None: continue dominated = False for other in items: if other is item: continue ot = pick_stats(other, "avg_time_handoff_ns") - os_ = other.get("median_size_bytes") - if ot is None or os_ is None: + ocpu = other.get("msgs_per_cpu_sec") + if ot is None: + continue + if cpu is None or ocpu is None: + if ot < t: + dominated = True + break continue - if (ot <= t and os_ < s) or (ot < t and os_ <= s): + better_or_eq_time = ot <= t + better_or_eq_cpu = ocpu >= cpu + strictly = ot < t or ocpu > cpu + if better_or_eq_time and better_or_eq_cpu and strictly: dominated = True break if not dominated: @@ -1231,7 +1239,7 @@ def compute_pareto_front(groups: List[Dict[str, Any]]) -> List[Dict[str, Any]]: "test_data": item.get("test_data"), "mode": item.get("mode"), "time": t, - "size": s, + "msgs_per_cpu_sec": cpu, "filter_policy": (item.get("filter") or {}).get("policy"), } ) diff --git a/analysis/tests/test_abi.py b/analysis/tests/test_abi.py index b145749..816abe9 100644 --- a/analysis/tests/test_abi.py +++ b/analysis/tests/test_abi.py @@ -24,6 +24,9 @@ def test_csv_header_uses_library_and_enq(): assert "Pattern" in CSV_HEADER assert "SerializerName" not in CSV_HEADER assert "QueueName" not in CSV_HEADER + assert "Size" not in CSV_HEADER + assert "SizeGzip" not in CSV_HEADER + assert "SizeZstd" not in CSV_HEADER def test_pick_accepts_leftover_serializer_columns(): diff --git a/analysis/tests/test_parser.py b/analysis/tests/test_parser.py index 432eaf2..c629419 100644 --- a/analysis/tests/test_parser.py +++ b/analysis/tests/test_parser.py @@ -125,5 +125,8 @@ def test_parse_size_gzip_zstd(): path = f.name recs, skipped = parse_csv_file(path) assert skipped == 0 - assert recs[0]["SizeGzip"] == 229 - assert recs[0]["SizeZstd"] == 180 + assert recs[0]["LibraryName"] == "encoding/json" + assert recs[0]["TimeHandoff"] == 300 + assert "SizeGzip" not in recs[0] + assert "SizeZstd" not in recs[0] + assert "Size" not in recs[0] diff --git a/c-sharp/src/Program.cs b/c-sharp/src/Program.cs index efc6e15..66ba9b1 100644 --- a/c-sharp/src/Program.cs +++ b/c-sharp/src/Program.cs @@ -41,7 +41,7 @@ var stamp = Environment.GetEnvironmentVariable("BENCHMARK_TS") ?? "run"; var csv = Path.Combine(logDir, stamp + ".csv"); var sb = new StringBuilder(); -sb.AppendLine("Language,Pattern,TestDataName,Repetitions,RepetitionIndex,LibraryName,LibraryVersion,TimeEnq,TimeDeq,Size,TimeHandoff,OpPerSecEnq,OpPerSecDeq,OpPerSecHandoff,MemoryPeakBytes,FidelityScore,DataTypeInstanceCount,TypeConfigHash,SizeGzip,SizeZstd,NativeKind,StreamMode,RunOrder,SchedulePosition,CpuTimeNs"); +sb.AppendLine("Language,Pattern,TestDataName,Repetitions,RepetitionIndex,LibraryName,LibraryVersion,TimeEnq,TimeDeq,TimeHandoff,OpPerSecEnq,OpPerSecDeq,OpPerSecHandoff,MemoryPeakBytes,FidelityScore,DataTypeInstanceCount,TypeConfigHash,NativeKind,StreamMode,RunOrder,SchedulePosition,CpuTimeNs"); string Ver = Environment.Version.ToString(); double Ops(long ns) => ns > 0 ? 1_000_000_000.0 / ns : 0; @@ -76,7 +76,7 @@ var item = new byte[cell.Payload]; for (int i = 0; i < item.Length; i++) item[i] = (byte)(i % 251); var items = Enumerable.Repeat(item, cell.N).ToArray(); - var size = cell.Payload * cell.N; + foreach (var q in queues) { if (qf.Length > 0 && !q.name.Contains(qf, StringComparison.OrdinalIgnoreCase)) @@ -126,9 +126,9 @@ var tot = enq + deq; sb.AppendLine(string.Join(",", "csharp", cell.Mode, cell.Type, reps, i, q.name, Ver, - enq, deq, size, tot, + enq, deq, tot, Ops(enq).ToString("F6"), Ops(deq).ToString("F6"), Ops(tot).ToString("F6"), - rss, "1.0000", cell.N, cell.Hash, 0, 0, q.kind, + rss, "1.0000", cell.N, cell.Hash, q.kind, cell.Mode == "stream" ? "native" : "", order, order, cpuNs)); order++; } diff --git a/c/src/main.c b/c/src/main.c index ff5eee2..ef2fc8a 100644 --- a/c/src/main.c +++ b/c/src/main.c @@ -333,13 +333,13 @@ static double ops(uint64_t ns) { static void write_row(FILE *f, const char *mode, const char *ty, int reps, int idx, const char *name, const char *ver, uint64_t enq, uint64_t deq, - size_t size, int n, const char *hash, const char *kind, int order, + int n, const char *hash, const char *kind, int order, uint64_t cpu, uint64_t rss) { uint64_t tot = enq + deq; fprintf(f, - "c,%s,%s,%d,%d,%s,%s,%llu,%llu,%zu,%llu,%.6f,%.6f,%.6f,%llu,1.0000,%d,%s,0,0,%s,%s,%d,%d,%llu\n", + "c,%s,%s,%d,%d,%s,%s,%llu,%llu,%llu,%.6f,%.6f,%.6f,%llu,1.0000,%d,%s,%s,%s,%d,%d,%llu\n", mode, ty, reps, idx, name, ver, - (unsigned long long)enq, (unsigned long long)deq, size, (unsigned long long)tot, + (unsigned long long)enq, (unsigned long long)deq, (unsigned long long)tot, ops(enq), ops(deq), ops(tot), (unsigned long long)rss, n, hash, kind, strcmp(mode, "stream") == 0 ? "native" : "", order, order, (unsigned long long)cpu); @@ -380,7 +380,7 @@ int main(int argc, char **argv) { int include_psd = env_on("BENCHMARK_INCLUDE_PSD"); const char *psd_names = getenv("BENCHMARK_PSD_NAMES"); if (!psd_names) psd_names = ""; - fprintf(out, "Language,Pattern,TestDataName,Repetitions,RepetitionIndex,LibraryName,LibraryVersion,TimeEnq,TimeDeq,Size,TimeHandoff,OpPerSecEnq,OpPerSecDeq,OpPerSecHandoff,MemoryPeakBytes,FidelityScore,DataTypeInstanceCount,TypeConfigHash,SizeGzip,SizeZstd,NativeKind,StreamMode,RunOrder,SchedulePosition,CpuTimeNs\n"); + fprintf(out, "Language,Pattern,TestDataName,Repetitions,RepetitionIndex,LibraryName,LibraryVersion,TimeEnq,TimeDeq,TimeHandoff,OpPerSecEnq,OpPerSecDeq,OpPerSecHandoff,MemoryPeakBytes,FidelityScore,DataTypeInstanceCount,TypeConfigHash,NativeKind,StreamMode,RunOrder,SchedulePosition,CpuTimeNs\n"); FILE *cf = fopen(cells, "r"); if (!cf) { @@ -401,7 +401,7 @@ int main(int argc, char **argv) { memset(item, 'a', (size_t)payload); void **items = calloc((size_t)n, sizeof(void *)); for (int i = 0; i < n; i++) items[i] = item; - size_t size = (size_t)payload * (size_t)n; + const char *names[] = {"mutex-queue", "lfqueue", "spsc-ring", "steal-deque", "pipe-ipc", "shared-ring", "sqlite-queue"}; const char *kinds[] = {"locked", "concurrent", "spsc", "work-stealing", "concurrent", "spsc", "durable"}; const int opt_in[] = {0, 0, 0, 0, 1, 1, 1}; @@ -563,7 +563,7 @@ int main(int argc, char **argv) { free(q.buf); } write_row(out, mode, type_id, reps, i, names[qi], "0.1.0", - enq, deq, size, n, hash, kinds[qi], order, + enq, deq, n, hash, kinds[qi], order, cpu_ns() - cpu0, rss_bytes()); order++; } diff --git a/config/benchmark_config.yaml b/config/benchmark_config.yaml index b9906ef..ddd98dc 100644 --- a/config/benchmark_config.yaml +++ b/config/benchmark_config.yaml @@ -114,7 +114,6 @@ csv_schema: - SerializerVersion - TimeSer - TimeDeser - - Size - TimeSerAndDeser - OpPerSecSer - OpPerSecDeser @@ -190,7 +189,7 @@ metrics: multi_way: include_importance: [high] rank_by: handoff_median_ns - secondary_rank_by: median_size_bytes + secondary_rank_by: msgs_per_cpu_sec pairwise: include_importance: [high, medium, low] importance: @@ -204,8 +203,7 @@ metrics: enq_mean_ns: medium deq_mean_ns: medium avg_ops_per_sec: high - median_size_bytes: high - mean_fidelity: high + mean_fidelity: medium library_version: high runs: high runs_raw: medium diff --git a/dashboard/charts.js b/dashboard/charts.js index 815f555..ab99fbf 100644 --- a/dashboard/charts.js +++ b/dashboard/charts.js @@ -17,7 +17,7 @@ const fontStyle = { const gridColor = 'rgba(0, 0, 0, 0.06)'; const tickColor = '#5f6368'; -/** @type {{ logScale: boolean, rankSort: 'speed' | 'size' }} */ +/** @type {{ logScale: boolean, rankSort: 'speed' | 'cpu' }} */ let chartOptions = { logScale: false, rankSort: 'speed' }; export function initCharts() { @@ -32,9 +32,9 @@ export function getChartLogScale() { return chartOptions.logScale; } -/** Ranking chart row order: 'speed' (ops/latency) or 'size' (median bytes, compact first). */ +/** Ranking chart row order: 'speed' (ops/latency) or 'cpu' (msgs per CPU-s). */ export function setRankSort(sort) { - chartOptions.rankSort = sort === 'size' ? 'size' : 'speed'; + chartOptions.rankSort = sort === 'cpu' || sort === 'size' ? 'cpu' : 'speed'; } export function getRankSort() { @@ -47,20 +47,20 @@ export function updateCharts(groups, paretoNames, metric) { const title = document.getElementById('bar-chart-title'); if (title) { title.textContent = - metric === 'ops' ? 'Throughput & Size Ranking' : 'Latency & Size Ranking'; + metric === 'ops' ? 'Throughput & CPU ranking' : 'Latency & CPU ranking'; } const help = document.getElementById('ranking-help'); if (help) { const primaryRight = metric === 'ops' ? 'ops/s' : 'latency'; const sortLabel = - chartOptions.rankSort === 'size' - ? 'sorted by size (most compact first)' + chartOptions.rankSort === 'cpu' + ? 'sorted by messages per CPU-second (highest first)' : metric === 'ops' ? 'sorted by ops/s (highest first)' : 'sorted by latency (lowest first)'; help.innerHTML = `Single diverging chart, ${sortLabel}: ` + - `◀ size left · ` + + `◀ CPU left · ` + `${primaryRight} ▶ right. ` + `Each side is normalized to the chart max (100); hover for absolute values.`; } @@ -89,7 +89,7 @@ function updateScatterChart(groups, paretoNames, metric) { groups.forEach((g) => { const xVal = isOps ? g.avg_ops_per_sec : g.avg_time_handoff_ns; - const yVal = g.median_size_bytes; + const yVal = g.msgs_per_cpu_sec; if (xVal == null || yVal == null || !Number.isFinite(xVal) || !Number.isFinite(yVal)) return; if (chartOptions.logScale && xVal <= 0) return; const point = { @@ -118,7 +118,7 @@ function updateScatterChart(groups, paretoNames, metric) { return formatSig(value / latencyScale.divisor); }; - const sortedPareto = [...paretoPoints].sort((a, b) => a.y - b.y); + const sortedPareto = [...paretoPoints].sort((a, b) => b.y - a.y); const frontierLineData = []; for (let i = 0; i < sortedPareto.length; i++) { frontierLineData.push({ x: sortedPareto[i].x, y: sortedPareto[i].y }); @@ -197,8 +197,8 @@ function updateScatterChart(groups, paretoNames, metric) { `Queue: ${p.label}`, `Throughput: ${formatOpsCompact(p.ops)}`, `Latency: ${formatTimeCompact(p.time)}`, - `Size: ${formatIntGrouped(p.y)} bytes`, - p.onFrontier ? 'On Pareto frontier' : 'Dominated on speed/size', + `Msgs / CPU-s: ${formatSig(p.y)}`, + p.onFrontier ? 'On Pareto frontier' : 'Dominated on speed/CPU', ]; }, }, @@ -223,7 +223,7 @@ function updateScatterChart(groups, paretoNames, metric) { y: { title: { display: true, - text: 'Payload size (bytes)', + text: 'Messages per CPU-second', color: tickColor, font: { ...fontStyle, weight: 'bold' }, }, @@ -231,7 +231,7 @@ function updateScatterChart(groups, paretoNames, metric) { ticks: { color: tickColor, font: fontStyle, - callback: (v) => formatIntGrouped(v), + callback: (v) => formatSig(v), }, }, }, @@ -267,7 +267,7 @@ function updateScatterChart(groups, paretoNames, metric) { /** * Single diverging (butterfly) horizontal bar chart: * - Right (blue): throughput or latency, normalized 0..100 vs chart max - * - Left (green): median size, normalized 0..-100 vs chart max + * - Left (green): messages per CPU-second, normalized 0..-100 vs chart max * Absolute values only in tooltips — avoids dual-axis collisions. */ function updateBarChart(groups, paretoNames, metric) { @@ -276,23 +276,22 @@ function updateBarChart(groups, paretoNames, metric) { if (barChartInstance) barChartInstance.destroy(); const isOps = metric === 'ops'; - const sortBySize = chartOptions.rankSort === 'size'; + const sortByCpu = chartOptions.rankSort === 'cpu'; const sortedGroups = [...groups] .filter((g) => { if (!g) return false; - if (g.median_size_bytes == null && sortBySize) return false; + if (g.msgs_per_cpu_sec == null && sortByCpu) return false; return isOps ? g.avg_ops_per_sec != null : g.avg_time_handoff_ns != null; }) .sort((a, b) => { - if (sortBySize) { - // Compact first (ascending size); tie-break by speed - const ds = (a.median_size_bytes ?? 0) - (b.median_size_bytes ?? 0); - if (ds !== 0) return ds; + if (sortByCpu) { + const dc = (b.msgs_per_cpu_sec ?? 0) - (a.msgs_per_cpu_sec ?? 0); + if (dc !== 0) return dc; } return isOps ? b.avg_ops_per_sec - a.avg_ops_per_sec - : a.avg_time_total_ns - b.avg_time_total_ns; + : (a.avg_time_handoff_ns ?? 0) - (b.avg_time_handoff_ns ?? 0); }) .slice(0, 15); @@ -300,17 +299,17 @@ function updateBarChart(groups, paretoNames, metric) { const primaryRaw = sortedGroups.map((g) => isOps ? g.avg_ops_per_sec : g.avg_time_handoff_ns ); - const sizeRaw = sortedGroups.map((g) => Number(g.median_size_bytes) || 0); + const cpuRaw = sortedGroups.map((g) => Number(g.msgs_per_cpu_sec) || 0); const maxPrimary = Math.max(...primaryRaw.filter((v) => Number.isFinite(v) && v > 0), 1); - const maxSize = Math.max(...sizeRaw.filter((v) => Number.isFinite(v) && v > 0), 1); + const maxCpu = Math.max(...cpuRaw.filter((v) => Number.isFinite(v) && v > 0), 1); - // Normalized: speed → +0..100, size → -0..-100 (diverging from center) + // Normalized: speed → +0..100, CPU → -0..-100 (diverging from center) const speedNorm = primaryRaw.map((v) => Number.isFinite(v) && maxPrimary > 0 ? (v / maxPrimary) * 100 : 0 ); - const sizeNorm = sizeRaw.map((v) => - Number.isFinite(v) && maxSize > 0 ? -(v / maxSize) * 100 : 0 + const sizeNorm = cpuRaw.map((v) => + Number.isFinite(v) && maxCpu > 0 ? -(v / maxCpu) * 100 : 0 ); const speedColors = sortedGroups.map((g) => @@ -324,10 +323,10 @@ function updateBarChart(groups, paretoNames, metric) { type: 'bar', data: { labels, - // Legend order matches plot: size left (←), then speed/latency right (→). + // Legend order matches plot: CPU left (←), then speed/latency right (→). datasets: [ { - label: 'Size (←)', + label: 'CPU (←)', data: sizeNorm, backgroundColor: sizeColors, borderColor: sortedGroups.map((g) => @@ -383,9 +382,9 @@ function updateBarChart(groups, paretoNames, metric) { const g = sortedGroups[context.dataIndex]; if (!g) return ''; const pct = Math.abs(context.raw).toFixed(0); - // dataset 0 = size (left), 1 = speed/latency (right) + // dataset 0 = CPU (left), 1 = speed/latency (right) if (context.datasetIndex === 0) { - return `Size: ${formatIntGrouped(g.median_size_bytes)} bytes (${pct}% of max in chart)`; + return `Msgs / CPU-s: ${formatSig(g.msgs_per_cpu_sec)} (${pct}% of max in chart)`; } const abs = isOps ? formatOpsCompact(g.avg_ops_per_sec) @@ -398,7 +397,7 @@ function updateBarChart(groups, paretoNames, metric) { const g = sortedGroups[items[0]?.dataIndex]; if (!g) return []; return [ - paretoNames.includes(g.library) ? 'Pareto optimal' : 'Dominated on speed/size', + paretoNames.includes(g.library) ? 'Pareto optimal' : 'Dominated on speed/CPU', ]; }, }, @@ -417,8 +416,8 @@ function updateBarChart(groups, paretoNames, metric) { title: { display: true, text: isOps - ? '◀ larger size normalized % of chart max higher ops/s ▶' - : '◀ larger size normalized % of chart max higher latency ▶', + ? '◀ more msgs / CPU-s normalized % of chart max higher ops/s ▶' + : '◀ more msgs / CPU-s normalized % of chart max higher latency ▶', color: tickColor, font: { ...fontStyle, size: 10 }, }, diff --git a/dashboard/exp-export.js b/dashboard/exp-export.js index 6d4ff90..c55eb68 100644 --- a/dashboard/exp-export.js +++ b/dashboard/exp-export.js @@ -36,13 +36,12 @@ function hasFiniteMetric(rows, key) { return (rows || []).some((r) => Number.isFinite(Number(r?.[key]))); } -/** Which optional experiment-table columns have at least one real value. */ +/** Optional experiment-table columns. Payload size is not a result. */ export function experimentTableFlags(rows) { const list = rows || []; return { write: hasFiniteMetric(list, 'write_median_ns'), read: hasFiniteMetric(list, 'read_median_ns'), - size: hasFiniteMetric(list, 'size_bytes'), spread: list.some((r) => totalStdUs(r) != null), }; } @@ -55,8 +54,6 @@ const CSV_COLUMNS = [ 'read_us', 'total_us', 'spread_std_us', - 'size_bytes', - 'size_gzip_bytes', 'trials', 'trials_raw', 'vs_fastest', @@ -237,8 +234,6 @@ function rowFields(row, rows) { read_us: nsToUs(row.read_median_ns), total_us: nsToUs(row.total_median_ns), spread_std_us: std == null ? '' : std, - size_bytes: row.size_bytes == null ? '' : row.size_bytes, - size_gzip_bytes: row.size_gzip_bytes == null ? '' : row.size_gzip_bytes, trials: row.runs == null ? '' : row.runs, trials_raw: row.runs_raw == null ? '' : row.runs_raw, vs_fastest: compareLabel(row, rows), diff --git a/dashboard/experiments.js b/dashboard/experiments.js index 6a43713..23edf1f 100644 --- a/dashboard/experiments.js +++ b/dashboard/experiments.js @@ -83,14 +83,6 @@ function ratioCell(valueNs, bestNs, key) { return `${escapeHtml(text)}`; } -function sizeCell(value, best) { - if (value == null || best == null) { - return `${value == null ? '—' : formatIntGrouped(value)}`; - } - const { text, className } = formatRelativeCell(value, best, false, {}, 'size_bytes'); - return `${escapeHtml(text)}`; -} - async function fetchJson(url) { const res = await fetch(url); if (!res.ok) throw new Error(`${url}: ${res.status}`); @@ -451,10 +443,7 @@ function renderTable(rows) { const flags = experimentTableFlags(rows); const showWrite = flags.write; const showRead = flags.read; - const showSize = flags.size; const showSpread = flags.spread; - const showGzip = rows.some((r) => r.size_gzip_bytes != null); - const showZstd = rows.some((r) => r.size_zstd_bytes != null); const sorted = [...rows].sort((a, b) => { if (showLang && a.language !== b.language) { const langs = unique(rows, 'language').map(String); @@ -477,9 +466,6 @@ function renderTable(rows) { ${showRead ? 'Read (µs)' : ''} Total (µs) ${showSpread ? 'Spread (std)' : ''} - ${showSize ? 'Size' : ''} - ${showGzip ? 'After gzip' : ''} - ${showZstd ? 'After zstd' : ''} Trials Vs fastest `; @@ -494,14 +480,6 @@ function renderTable(rows) { const bestWrite = bestAmong(row, sorted, 'write_median_ns'); const bestRead = bestAmong(row, sorted, 'read_median_ns'); const bestTotal = bestAmong(row, sorted, 'total_median_ns'); - const sizePool = showLang ? competingRows(sorted) : competingRows(peerRows(row, sorted)); - const minSize = (key) => { - const nums = sizePool.map((r) => Number(r[key])).filter(Number.isFinite); - return nums.length ? Math.min(...nums) : null; - }; - const bestSize = minSize('size_bytes'); - const bestGzip = minSize('size_gzip_bytes'); - const bestZstd = minSize('size_zstd_bytes'); return ` ${showLang ? `${escapeHtml(langLabel(row.language))}` : ''} @@ -513,9 +491,6 @@ function renderTable(rows) { ${showRead ? (skipped ? `${formatSig(Number(row.read_median_ns) / 1000)}` : ratioCell(row.read_median_ns, bestRead, 'read_median_ns')) : ''} ${skipped ? `${formatSig(Number(row.total_median_ns) / 1000)}` : ratioCell(row.total_median_ns, bestTotal, 'total_median_ns')} ${showSpread ? `${std == null ? '—' : formatSig(std)}` : ''} - ${showSize ? (skipped ? `${formatIntGrouped(row.size_bytes)}` : sizeCell(row.size_bytes, bestSize)) : ''} - ${showGzip ? (skipped ? `${formatIntGrouped(row.size_gzip_bytes)}` : sizeCell(row.size_gzip_bytes, bestGzip)) : ''} - ${showZstd ? (skipped ? `${formatIntGrouped(row.size_zstd_bytes)}` : sizeCell(row.size_zstd_bytes, bestZstd)) : ''} ${trials} ${escapeHtml(compareLabel(row, sorted))} `; diff --git a/dashboard/format.js b/dashboard/format.js index 5d10081..e0f4f78 100644 --- a/dashboard/format.js +++ b/dashboard/format.js @@ -158,6 +158,7 @@ export function metricKind(key) { ) { return 'ops'; } + if (key === 'msgs_per_cpu_sec') return 'rate'; if (key.endsWith('_bytes') || key.startsWith('size_') || key === 'median_size_bytes') return 'bytes'; if ( key === 'runs' || @@ -187,7 +188,7 @@ export function formatMetricCell(key, value, scales = {}) { if (kind === 'latency') return formatLatencyCell(value, scales.latency); if (kind === 'ops') return formatOpsCell(value, scales.ops); if (kind === 'bytes' || kind === 'count') return formatIntGrouped(value); - if (kind === 'ratio') return formatSig(value); + if (kind === 'ratio' || kind === 'rate') return formatSig(value); if (Number.isInteger(value) && Math.abs(value) < 1e9) return formatIntGrouped(value); return formatSig(value); } diff --git a/dashboard/index.html b/dashboard/index.html index 4187459..784871e 100644 --- a/dashboard/index.html +++ b/dashboard/index.html @@ -151,11 +151,11 @@