Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions dashboard/experiments.css
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,11 @@ body.dash-view-experiments #filter-policy-meta {
margin: 0.5rem 0 0;
}

.exp-why[open] {
max-height: min(28rem, 55vh);
overflow: auto;
}

.exp-story p {
margin: 0.35rem 0;
}
Expand Down
41 changes: 21 additions & 20 deletions dashboard/experiments.js
Original file line number Diff line number Diff line change
Expand Up @@ -244,43 +244,44 @@ function selectHtml(id, label, values, current, labelFn) {

function renderWhyExperiments() {
return `
<details class="exp-howto exp-why glass-panel" open>
<details class="exp-howto exp-why glass-panel">
<summary>Why experiments, if the Dashboard already has every number?</summary>
<p>
The main Dashboard is the <strong>full warehouse</strong>: every library, every record shape.
That is the right place to browse. It is the wrong place to pick a library for a real service
by sorting on speed.
The main Dashboard is the <strong>full warehouse</strong>: every library, every payload size,
every pattern. That is the right place to browse. It is the wrong place to pick a queue
for a real service by sorting on speed.
</p>
<p>
The fastest row in the big table may be a library you <strong>cannot use</strong>.
It might write bytes that only one language can read.
It might drop field names so partners cannot parse the text.
It might be built for a file you write once, not for a web request you write on every click.
The Dashboard will still put it first, because it is fast.
The fastest row in the big table may be a queue you <strong>cannot use</strong>.
It might only work with one writer and one reader. It might be an async queue
while your workers are operating-system threads. It might be a scheduler that
does not hand a payload from a producer to a consumer. The Dashboard will still
put it first, because it is fast on that slice.
</p>
<p>
An experiment starts with a <strong>decision</strong>, not a sort.
Example: “We must keep JSON on the public website. Which JSON library is fast enough?”
We keep only the libraries that could be the answer. We time one shared record.
Example: “One request thread puts work; one worker takes it. The ticket is
256 bytes. Which in-process queue is fast enough?”
We keep only the libraries that could be the answer. We time one shared payload.
We write down what we would give up.
</p>
<p>
Think of a grocery store and a recipe. The store has every product — that is the main Dashboard.
The recipe lists only what you need for dinner, and why — that is an experiment.
You need both. The store does not tell you what to cook.
Think of a grocery store and a recipe. The store has every product — that is the
main Dashboard. The recipe lists only what you need for dinner, and why — that
is an experiment. You need both. The store does not tell you what to cook.
</p>
<p>
Concrete traps the big table does not prevent:
</p>
<ul>
<li><strong>Different jobs in one list.</strong> JSON, compact bytes, and a Python-only format can share a table. “What is fastest?” is not one question.</li>
<li><strong>A tiny gap looks like a winner.</strong> 1.7 µs vs 1.8 µs is not a reason to rewrite a service.</li>
<li><strong>Your constraint is invisible.</strong> If browsers must keep JSON, the big table still shows every format.</li>
<li><strong>One record is not one hundred.</strong> The winner at 1 can lose at 100.</li>
<li><strong>Numbers without a story.</strong> “Binary is faster” is how teams break a public website.</li>
<li><strong>Different jobs in one list.</strong> A 1P1C ring and a 4P4C channel can share a table. “What is fastest?” is not one question.</li>
<li><strong>A tiny gap looks like a winner.</strong> A few percent is not a reason to rewrite a service.</li>
<li><strong>Your constraint is invisible.</strong> If you must use threads, the big table still shows async queues.</li>
<li><strong>One size is not another.</strong> The winner at 256 bytes can lose at 4 KiB.</li>
<li><strong>One pattern is not another.</strong> The winner at 1P1C can lose at 4P4C.</li>
</ul>
<p>
We do not run a second benchmark. We cut the same numbers down to a fair contest
We do not run a second clock. We cut the same numbers down to a fair contest
and put the story next to them.
<a href="../experiments/">Longer notes</a>
</p>
Expand Down
22 changes: 11 additions & 11 deletions dashboard/main.js
Original file line number Diff line number Diff line change
Expand Up @@ -1878,7 +1878,7 @@ function populateDataTypeSelect(options, cfg = {}) {
batchCompound.forEach(addOpt);
}
if (allTypes.length) {
addSep('── compounded data types ──');
addSep('── compounded data sizes ──');
allTypes.forEach(addOpt);
}
if (allAll.length) {
Expand Down Expand Up @@ -2231,20 +2231,20 @@ function discoverDataTypeOptions(allGroups) {
}
batchCompound.sort();

// Cross-type at fixed n
// Both published sizes at one item count: all@100, all@1000.
const allTypes = [];
if (nsGlobal.has(1)) allTypes.push('all@1');
if (nsGlobal.has(100)) allTypes.push('all@100');

// Everything
const allAll = natural.length ? ['all@all'] : [];
if (byBase.size >= 2) {
for (const n of [...nsGlobal].sort((a, b) => a - b)) {
allTypes.push(`all@${n}`);
}
}

return {
natural,
batchCompound,
allTypes,
allAll,
all: [...natural, ...batchCompound, ...allTypes, ...allAll],
allAll: [],
all: [...natural, ...batchCompound, ...allTypes],
};
}

Expand Down Expand Up @@ -3420,7 +3420,7 @@ function renderTable() {
scopeNote =
` <strong>Compounded batch</strong>: mean of <code>${escapeHtml(scope.base)}@n=${scope.nA}</code> and <code>${escapeHtml(scope.base)}@n=${scope.nB}</code>.`;
} else if (scope.kind === 'all_n') {
scopeNote = ` <strong>Compounded data types</strong> at <code>n=${scope.n}</code>.`;
scopeNote = ` <strong>Both sizes</strong> at <code>${scope.n}</code> items.`;
} else if (scope.kind === 'all_all') {
scopeNote = ` <strong>Compounded all</strong> (<code>all@all</code>).`;
}
Expand Down Expand Up @@ -3730,7 +3730,7 @@ function copyRosterMarkdown() {
return `Scope: compounded batch ${state.currentTestData} · mode ${state.currentMode}`;
}
if (s.kind === 'all_n') {
return `Scope: all data types @ n=${s.n} · mode ${state.currentMode}`;
return `Scope: both sizes · ${s.n} items · ${modeDisplayLabel(normalizeMode(state.currentMode))}`;
}
if (s.kind === 'all_all') {
return `Scope: all@all (all types × all n) · mode ${state.currentMode}`;
Expand Down
Binary file modified dashboard/public/data/experiments/13-payload-size-sweep.json.gz
Binary file not shown.
6 changes: 3 additions & 3 deletions dashboard/public/data/experiments/index.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"schema": "gld.dashboard.experiments/1",
"generated_at": "2026-09-01T22:23:00Z",
"generated_at": "2026-09-02T00:00:42Z",
"count": 13,
"with_results": 13,
"experiments": [
Expand Down Expand Up @@ -598,9 +598,9 @@
"number": 13,
"slug": "payload-size-sweep",
"title": "Which payload sizes change the ranking?",
"question": "Same SPSC handoff, five payload sizes (1 B, 64 B, 256 B, 4 KiB, 64 KiB). Which sizes produce a different ranking, and which are just more of the same?\n",
"question": "Same 1P1C handoff, five payload sizes (1 B, 64 B, 256 B, 4 KiB, 64 KiB). Which sizes produce a different ranking, and which are just more of the same?\n",
"story": {
"why": "The default suite names five data types, but every runner builds an opaque byte string of a chosen length. If ranking only moves at a couple of lengths, the published matrix should keep those lengths and drop the rest.\n",
"why": "Every runner builds an opaque byte string of a chosen length. If ranking only moves at a couple of lengths, the published matrix should keep those lengths and drop the rest.\n",
"example": "A ticket is a few hundred bytes. An HTTP body is a few kilobytes. A file chunk can be tens of kilobytes. We need to know which of those worlds actually changes which queue wins.\n",
"tradeoff": "Extra sizes cost machine time and make the dashboard harder to read. One size is enough if copy cost never changes the order. Two sizes are enough if a small payload and a large payload tell different stories.\n"
},
Expand Down
12 changes: 5 additions & 7 deletions experiments/13-payload-size-sweep/README.md
Original file line number Diff line number Diff line change
@@ -1,16 +1,14 @@
# 13 — Payload size sweep

**Question:** which payload sizes change the SPSC ranking, and which are
**Question:** which payload sizes change the 1P1C ranking, and which are
redundant?

This is not another named data type. Every runner already builds an opaque
byte string. The five catalog names (`message`, `event`, `telemetry`,
`strings`, `document`) are five lengths between 256 B and 4 KiB. This
folder asks whether the published suite needs those names at all, or only
one or two lengths.
Every runner already builds an opaque byte string of a chosen length.
This folder asked which lengths change ranking, and which are just more
of the same.

Sizes in this run: **1 B, 64 B, 256 B, 4 KiB, 64 KiB**. One hundred items
per repetition. SPSC only. One hundred repetitions (`full`).
per repetition. 1P1C only. One hundred repetitions (`full`).

```bash
./experiments/13-payload-size-sweep/run.sh python
Expand Down
9 changes: 4 additions & 5 deletions experiments/13-payload-size-sweep/experiment.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,15 +2,14 @@ schema: gld.experiment.config/1
id: 13-payload-size-sweep
title: Which payload sizes change the ranking?
question: >
Same SPSC handoff, five payload sizes (1 B, 64 B, 256 B, 4 KiB, 64 KiB).
Same 1P1C handoff, five payload sizes (1 B, 64 B, 256 B, 4 KiB, 64 KiB).
Which sizes produce a different ranking, and which are just more of the
same?
story:
why: >
The default suite names five data types, but every runner builds an
opaque byte string of a chosen length. If ranking only moves at a
couple of lengths, the published matrix should keep those lengths and
drop the rest.
Every runner builds an opaque byte string of a chosen length. If ranking
only moves at a couple of lengths, the published matrix should keep those
lengths and drop the rest.
example: >
A ticket is a few hundred bytes. An HTTP body is a few kilobytes. A
file chunk can be tens of kilobytes. We need to know which of those
Expand Down
2 changes: 1 addition & 1 deletion experiments/13-payload-size-sweep/results.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"generated_at": "2026-09-01T22:23:00Z",
"question": "Which SPSC payload sizes change the ranking?",
"question": "Which 1P1C payload sizes change the ranking?",
"sizes_bytes": [
1,
64,
Expand Down
6 changes: 3 additions & 3 deletions experiments/13-payload-size-sweep/results.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

Times are not comparable across languages.

Question: which SPSC payload sizes change the ranking?
Question: which 1P1C payload sizes change the ranking?

## Recommendation

Expand All @@ -24,9 +24,9 @@ Do not confuse **payload bytes per item** with **how many items** move in one re
- `size_1` (1 B): same first place as 256 B, no copy-bound pack collapse, no inversion.
- `size_64` (64 B): same first place as 256 B, no copy-bound pack collapse, no inversion.

## What this means for the named types
## What this means for the published matrix

The published catalog names `message`, `event`, `telemetry`, `strings`, and `document` are already five lengths: 256 B, 512 B, 1 KiB, 2 KiB, and 4 KiB. Runners do not serialize those object graphs. The warehouse SPSC ranks on those names match this sweep: Python and C never change first place; JavaScript and C# only shuffle a mid-pack pair; Rust first becomes copy-bound at 4 KiB. 512 B, 1 KiB, and 2 KiB do not add a ranking question. Keep `message` (256 B) and `document` (4 KiB). Drop the three names in between from the default matrix when that change is applied.
Every runner builds an opaque byte string. This sweep asked which lengths change 1P1C ranking. Python and C never change first place. JavaScript and C# only shuffle a close pair. Rust first changes at 4 KiB, when the queue copies every byte. Keep 256 B and 4 KiB on the default matrix. 1 B and 64 B match 256 B. 64 KiB is a later research cell, not a third published size.

## Per language vs 256 B

Expand Down
21 changes: 9 additions & 12 deletions experiments/13-payload-size-sweep/summarize.py
Original file line number Diff line number Diff line change
Expand Up @@ -371,7 +371,7 @@ def write_combined_md(
"",
"Times are not comparable across languages.",
"",
"Question: which SPSC payload sizes change the ranking?",
"Question: which 1P1C payload sizes change the ranking?",
"",
"## Recommendation",
"",
Expand Down Expand Up @@ -408,17 +408,14 @@ def write_combined_md(
lines.append("")
lines.extend(
[
"## What this means for the named types",
"## What this means for the published matrix",
"",
"The published catalog names `message`, `event`, `telemetry`, "
"`strings`, and `document` are already five lengths: 256 B, 512 B, "
"1 KiB, 2 KiB, and 4 KiB. Runners do not serialize those object "
"graphs. The warehouse SPSC ranks on those names match this sweep: "
"Python and C never change first place; JavaScript and C# only "
"shuffle a mid-pack pair; Rust first becomes copy-bound at 4 KiB. "
"512 B, 1 KiB, and 2 KiB do not add a ranking question. Keep "
"`message` (256 B) and `document` (4 KiB). Drop the three names "
"in between from the default matrix when that change is applied.",
"Every runner builds an opaque byte string. This sweep asked which "
"lengths change 1P1C ranking. Python and C never change first "
"place. JavaScript and C# only shuffle a close pair. Rust first "
"changes at 4 KiB, when the queue copies every byte. Keep 256 B "
"and 4 KiB on the default matrix. 1 B and 64 B match 256 B. "
"64 KiB is a later research cell, not a third published size.",
"",
"## Per language vs 256 B",
"",
Expand Down Expand Up @@ -470,7 +467,7 @@ def main() -> int:
recommendation = recommend(lang_reports)
payload = {
"generated_at": datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"),
"question": "Which SPSC payload sizes change the ranking?",
"question": "Which 1P1C payload sizes change the ranking?",
"sizes_bytes": [SIZE_BYTES[t] for t in SIZE_ORDER],
"baseline_bytes": SIZE_BYTES[BASELINE],
"recommendation": recommendation,
Expand Down
Loading