Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .grok/skills/implement-experiment/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,9 +40,9 @@ export PATH="${HOME}/.local/go/bin:${HOME}/.cargo/bin:${HOME}/.dotnet:${HOME}/.l
2. **One question per folder.** Number it `NN-short-kebab` (next free `NN` after existing dirs).
3. **Edit `experiment.yaml` only** for sample, languages, libraries, run mode, grouping cut-offs. `run.yaml` is generated. Do not keep a second library list.
4. **Shared sample** at the experiment root (`sample.json`). Language folders hold **results** only (`results.md`, `results.json`). **Do not commit experiment logs** (CSVs under `<lang>/logs/`). Those stay on the machine that ran the timing. Saved results are enough.
5. **Do not compare times across languages.** Different runtimes. Directional only. Do not invent a size contest — payload bytes are the **data type**, not a library score.
5. **Do not compare times across languages.** Different runtimes. Directional only. Do not invent a size contest — payload size is the sample, not a library score.
6. **Do not crown a single winner.** Use `top_group` (similar / close / slower via Cliff’s delta vs the fastest library in the comparison set). Not “top 5%.”
7. **Textbook language** in `experiment.yaml` (`story.example`, `story.tradeoff`, `story.why`), `README.md`, `results.md`, the Dashboard, and the PLAN update. Write for a high-school student: complete sentences, no slang, no telegraphic fragments, no unexplained jargon. The Dashboard copies `story` from `experiment.yaml`.
7. **Textbook language** — follow `.grok/skills/improve-docs/references/STYLE.md` § Voice. That includes the finding you tell the user, not only `experiment.yaml` (`story.example`, `story.tradeoff`, `story.why`), `README.md`, `results.md`, the Dashboard, and the PLAN update. The Dashboard copies `story` from `experiment.yaml`.
8. **Do not overwrite** published site tables (`docs/<lang>/results.md`) or dashboard `*_latest.json.gz` unless the user asked to publish suite numbers.
9. After the run, **update `experiments/PLAN.md`**: mark the item done, add an **After Experiment *n*** block, change later questions only if the finding requires it. Do not rewrite old findings.

Expand Down Expand Up @@ -97,7 +97,7 @@ Copy Experiment 1’s file and change:
|-----|-------------|
| `id` | Folder name (`NN-short-kebab`) |
| `title` / `question` | From the PLAN section (ordinary words) |
| `sample.kind` | `message` \| `document` \| `telemetry` \| `strings` \| `event` |
| `sample.kind` | `size_256` \| `size_4096` \| `message` \| `document` |
| `sample.records_per_write` | `1` or `[1, 100]` |
| `sample.settings` | Only extra knobs (e.g. `points: 128`). `{}` = catalog defaults |
| `sample.seed` | `42` unless the PLAN says otherwise |
Expand Down
2 changes: 1 addition & 1 deletion .grok/skills/improve-docs/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ Cut scope creep, ensure dismissible UI is not permanent noise, prefer small diff
- Edit sources under `dashboard/` then **`npm run build`** in `dashboard/` so `docs/dashboard/` updates.
- **README:** follow `README_EDITING.md` — short lede, compact audience table, section order Who → languages → Try it → Quick start; no slogan/meta CTA prose; no unsolicited role expansion.
- MkDocs storefront (`docs/index.md`) only if asked or if it clearly contradicts a user-approved README fact.
- Follow `STYLE.md` (simple, no new frameworks). **Terminology: always “data type”, never “fixture”** — in user copy, comments, identifiers, and JSON keys.
- Follow `STYLE.md` (simple, no new frameworks). **Terminology:** published filter is **payload size**; never say “fixture”. Catalog ids stay type_id values.

### 2.5 Critique the implementation

Expand Down
15 changes: 9 additions & 6 deletions .grok/skills/improve-docs/references/STYLE.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,11 +6,12 @@ Keep it simple. Prefer delete and clarify over decorate.

| Say | Never say |
|-----|-----------|
| **data type** (`message`, `document`, `telemetry`, `strings`, `event`) | fixture, fixtures, fixtureKey, `dataset.fixtures` |
| **payload size** (256 B, 4 KiB) for the published filter | fixture; five leftover type names as if they were different shapes |
| **data type** only as the catalog id (`size_256`, `size_4096`) | fixture, fixtures, fixtureKey, `dataset.fixtures` |

This is the catalog payload shape. Use **data type** in README, Dashboard copy, skills, comments, function names, and new JSON keys. Do not keep `fixture*` as an internal alias. Old `configs.json` may still contain `fixtures`; read it as a fallback, write `data_types`.
The published axis is how many bytes each queued item is. Experiment 13 showed that 256 B and 4 KiB tell different stories; 512 B / 1 KiB / 2 KiB do not. Use **payload size** in README, Dashboard copy, and findings. Catalog ids stay `type_id` values (`size_256`, `size_4096`). Older folders may still say `message` / `document` (same lengths). Do not keep `fixture*` as an internal alias. Old `configs.json` may still contain `fixtures`; read it as a fallback, write `data_types`.

Do not measure or display payload **size** as a library result. Size is the data type, not a score.
Do not rank a library on payload size. Every queue in a cell moves the same bytes. Size is the sample, not a score.

---

Expand All @@ -28,8 +29,8 @@ Do not measure or display payload **size** as a library result. Size is the data
## Content style

- **One idea per paragraph.** Prefer tables for role/path matrices **on the site / Dashboard**, not by fattening the root README.
- **User terms:** data type, mode (bytes/stream), ops/s, latency, Pareto, baseline.
- **Never say “fixture”.** The catalog entry is a **data type** (`message`, `document`, …). Say it in docs, Dashboard copy, skills, comments, identifiers, and JSON keys. Do not keep `fixture*` as an internal alias “until later.”
- **User terms:** payload size (256 B, 4 KiB), mode (bytes/stream), ops/s, latency, Pareto, baseline.
- **Never say “fixture”.** The published sample is a **payload size**. The catalog id is a **data type** (`size_256`, `size_4096`). Say that in docs, Dashboard copy, skills, comments, identifiers, and JSON keys. Do not keep `fixture*` as an internal alias “until later.”
- **Avoid in user copy:** fixture, median size (we do not measure payload size as a result), harness (prefer benchmark runner), unexplained IQR/P95.
- **Honesty line** when ranks appear: within one language; cross-lang directional — prefer **one** place (e.g. Statistics / Method), not a second essay block on README.
- **Links:** prefer site paths that match MkDocs nav labels (Dashboard, Learn, Method). Avoid “storefront” / “CTA” wording in user-facing labels.
Expand All @@ -38,15 +39,17 @@ Do not measure or display payload **size** as a library result. Size is the data
### Voice

- **Textbook quality for a high-school student.** Complete sentences. One idea, then the reason. Define a term the first time it is not everyday English.
- This Voice section applies to **replies to the user** as well as published copy. A chat explanation is reader-facing.
- No slang (“at 3 a.m.”, “chatty ping”, “get user”).
- No telegraphic fragments (“Keep YAML on disk. Convert once.”). Write the sentence out.
- No unexplained jargon (stream, socket, pickle, gzip) unless the next sentence says what it is.
- **No clipped systems metaphors.** Do not write “pass a handle”, “pointer-bound”, “copy-bound”, “pack collapse”, “size barely moves them”, or “the knee” as if those were ordinary English. First say what the computer does, then what the clocks did. Example: the queue either copies every byte of the message, or it stores a short reference to a message that is already in memory. A larger message then either takes longer (because every byte is copied) or it does not (because only the reference is moved).
- Direct, calm. Not marketing hype (“blazing”, “crush”).
- “We measure …” not “We revolutionize …”.
- **No slogan stacks** under the title (“Same A. Same B. Same C.”).
- **No lede hedges** that argue with imaginary critics (“not marketing microbenchmarks”).

This rule applies to **every** reader-facing surface: Dashboard story cards (Why / Example / Trade-off), experiment YAML, `docs/experiments/`, Learn pages, README. The Dashboard reads Example and Trade-off from `experiments/*/experiment.yaml` via `sync-experiments.py`.
This rule applies to **every** reader-facing surface: replies to the user, Dashboard story cards (Why / Example / Trade-off), experiment YAML, `results.md`, `experiments/PLAN.md`, `docs/experiments/`, Learn pages, README. The Dashboard reads Example and Trade-off from `experiments/*/experiment.yaml` via `sync-experiments.py`.

### README-specific (authoritative)

Expand Down
1 change: 1 addition & 0 deletions .grok/skills/prepare-pr/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -137,6 +137,7 @@ export PREPARE_PR_LANGS="${PREPARE_PR_LANGS:-$CHANGED_LANGS}"
| `schemas/**` **except** prose (see below) | → **all** enabled languages |
| `scripts/run-all-benchmarks.sh`, `scripts/lib/**`, `scripts/read-config.py`, `scripts/resolve_run_config.py` | → **all** |
| `config/benchmark_config.yaml` | → **all** |
| `config/library/*` (default / smoke run matrix) | → **all** |
| **Always ignored (never select langs, never force-all)** | |
| `docs/**`, `dashboard/public/data/**`, `logs/**`, `reports/**`, `site/**`, `.grok/**` | meta / published artifacts |
| `**/*.md`, `**/*.mdx`, `**/README*`, `LICENSE*` **anywhere** (including under `schemas/`, `scripts/`, or a language tree) | prose-only; e.g. terminology edits in `schemas/v2/README.md` must **not** re-bench the world |
Expand Down
2 changes: 1 addition & 1 deletion .grok/skills/prepare-pr/scripts/detect-changed-langs.sh
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@ is_shared_force_all() {
scripts/run-all-benchmarks.sh|scripts/lib/*|scripts/read-config.py|scripts/resolve_run_config.py|scripts/emit-cells-tsv.py)
return 0
;;
config/benchmark_config.yaml)
config/benchmark_config.yaml|config/library/*)
return 0
;;
esac
Expand Down
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,8 @@ commit before `publish-docs` deploys the site.

## Test data

Shared **payload types**: `message`, `document`, `telemetry`, `strings`, and `event`.
Shared **payload sizes**: 256 B (`size_256`) and 4 KiB (`size_4096`). Every
runner builds an opaque byte string of that length.

Catalog and defaults: `schemas/data_catalog_v2.yaml`. Run matrices: `config/library/`.

Expand Down
2 changes: 1 addition & 1 deletion analysis/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ uv pip install -e .

Each run writes a timestamped `YYYY-MM-DD-HHMMSS.csv` (never overwritten). Analysis picks the **latest** per language by default.

Suite type ids: `message`, `document`, `telemetry`, `strings`, `event`.
Suite type ids: `size_256` (256 B), `size_4096` (4 KiB).

```bash
analyze-benchmarks
Expand Down
37 changes: 26 additions & 11 deletions analysis/src/benchmark_analysis/reports.py
Original file line number Diff line number Diff line change
Expand Up @@ -56,13 +56,27 @@ def _stat_get(entry: Any, field_id: str) -> Any:
_FIXTURE_KEY_RE = re.compile(r"^(.*?)(?:@n=(\d+))+$", re.IGNORECASE)


_PAYLOAD_SIZE_LABELS = {
"size_1": "1 B",
"size_64": "64 B",
"size_256": "256 B",
"size_4096": "4 KiB",
"size_65536": "64 KiB",
"message": "256 B",
"document": "4 KiB",
"event": "512 B",
"telemetry": "1 KiB",
"strings": "2 KiB",
}


def _format_data_type_display(label: str) -> str:
"""Decode cryptic ``message@n=100`` keys for titles and table headers.
"""Decode ``size_256@n=100`` keys for titles and table headers.

Examples:
``message@n=1`` → ``Message · 1 instance``
``message@n=100`` → ``Message · 100 instances``
``document`` → ``Document``
``size_256@n=1`` → ``256 B · 1 item``
``size_4096@n=100`` → ``4 KiB · 100 items``
``message`` → ``256 B``
"""
s = str(label or "").strip()
if not s:
Expand All @@ -77,15 +91,16 @@ def _format_data_type_display(label: str) -> str:
else:
base, n = s, None

# Suite type_ids are lowercase words; older labels may already be Title/Pascal.
if base and base == base.lower() and re.fullmatch(r"[a-z][a-z0-9_]*", base):
pretty = base.replace("_", " ").title()
else:
pretty = base
pretty = _PAYLOAD_SIZE_LABELS.get(base)
if pretty is None:
if base and base == base.lower() and re.fullmatch(r"[a-z][a-z0-9_]*", base):
pretty = base.replace("_", " ").title()
else:
pretty = base

if n is None:
return pretty
unit = "instance" if n == 1 else "instances"
unit = "item" if n == 1 else "items"
return f"{pretty} · {n} {unit}"


Expand Down Expand Up @@ -1393,7 +1408,7 @@ def generate_language_results_pages(
"",
"| Term | Meaning |",
"|------|---------|",
"| **data type** | Sample shape: `message`, `document`, `telemetry`, `strings`, or `event` "
"| **payload size** | How many bytes each item is: `size_256` (256 B) or `size_4096` (4 KiB) "
"(CSV `TestDataName`) |",
"| **bytes mode** | In-memory buffer API (encode to bytes / decode from a buffer). "
"On C# this is often the **string** path — see [Modes](../analysis/modes.md). |",
Expand Down
30 changes: 25 additions & 5 deletions analysis/tests/test_run_config_v2.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,10 +21,28 @@
_CATALOG = _REPO / "schemas" / "data_catalog_v2.yaml"


_CORE_TYPES = {"message", "document", "telemetry", "strings", "event"}
_SIZE_SWEEP_TYPES = {
"size_1": 1,
"size_64": 64,
"size_256": 256,
"size_4096": 4096,
"size_65536": 65536,
}


def test_catalog_loads_five_types():
cat = load_catalog(_CATALOG)
assert cat["data_model_version"] == 2
assert set(cat["types"]) == {"message", "document", "telemetry", "strings", "event"}
assert _CORE_TYPES <= set(cat["types"])


def test_catalog_size_sweep_types():
cat = load_catalog(_CATALOG)
assert set(_SIZE_SWEEP_TYPES) <= set(cat["types"])
for type_id, nbytes in _SIZE_SWEEP_TYPES.items():
resolved = resolve_type_config(type_id, {}, cat)
assert resolved["payload_bytes"] == nbytes


def test_resolve_empty_type_config_fills_defaults():
Expand Down Expand Up @@ -63,17 +81,19 @@ def test_type_config_hash_stable():

def test_smoke_expand_cell_count():
resolved = resolve_run_config(_LIBRARY / "smoke.yaml", catalog_path=_CATALOG, seed=42)
assert resolved["cell_count"] == 1 # message × [1]
assert resolved["cell_count"] == 1 # size_256 × [1]
assert resolved["seed"] == 42
assert resolved["run_config"]["content_sha256"]
ids = {(c["type_id"], c["data_type_instance_count"]) for c in resolved["cells"]}
assert ids == {("message", 1)}
assert ids == {("size_256", 1)}


def test_default_expand_cell_count():
resolved = resolve_run_config(_LIBRARY / "default.yaml", catalog_path=_CATALOG)
# 5 types × 2 counts
assert resolved["cell_count"] == 10
# 2 sizes × 2 counts
assert resolved["cell_count"] == 4
ids = {c["type_id"] for c in resolved["cells"]}
assert ids == {"size_256", "size_4096"}
counts = {c["data_type_instance_count"] for c in resolved["cells"]}
assert counts == {100, 1000}

Expand Down
2 changes: 1 addition & 1 deletion c-sharp/scripts/run-benchmarks.sh
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ else
REPS="$(bench_mode_reps "$MODE")"
if [[ "$MODE" == "smoke" ]]; then
FILTER_SER="${FILTER_SER:-Queue+lock}"
FILTER_DATA="${FILTER_DATA:-message}"
FILTER_DATA="${FILTER_DATA:-size_256}"
fi
fi

Expand Down
2 changes: 1 addition & 1 deletion c/scripts/run-benchmarks.sh
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ else
REPS="$(bench_mode_reps "$MODE")"
if [[ "$MODE" == "smoke" ]]; then
FILTER_SER="${FILTER_SER:-mutex-queue}"
FILTER_DATA="${FILTER_DATA:-message}"
FILTER_DATA="${FILTER_DATA:-size_256}"
fi
fi

Expand Down
28 changes: 8 additions & 20 deletions config/benchmark_config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -53,30 +53,18 @@ data_model_v2:
schemas_root: "schemas/v2"
generate_script: "scripts/schemas/generate-all.sh"

# Payload shapes. Catalog is normative; names are type_id values.
# For queues these are payload sizes / shapes, not object graphs.
# Payload sizes. Catalog is normative; names are type_id values.
# For queues these are lengths of an opaque byte string, not object graphs.
test_data:
catalog_file: "schemas/data_catalog_v2.yaml"
types:
- name: message
category: flat_mixed
description: "Small message payload (~256 B)"
- name: size_256
category: opaque_bytes
description: "256-byte payload (small message)"
supports_circular: false
- name: document
category: nested
description: "Larger document payload (~4 KiB)"
supports_circular: false
- name: telemetry
category: numeric_bulk
description: "Numeric bulk payload (~1 KiB)"
supports_circular: false
- name: strings
category: text_bulk
description: "Text bulk payload (~2 KiB)"
supports_circular: false
- name: event
category: envelope
description: "Event envelope payload (~512 B)"
- name: size_4096
category: opaque_bytes
description: "4 KiB payload (copy cost starts to matter)"
supports_circular: false

# Pattern axis (CSV StringOrStream). Logged names are serializer ABI leftovers:
Expand Down
9 changes: 5 additions & 4 deletions config/library/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,15 +6,16 @@ Named **run configs** select the measurement matrix:
- `data_type_instance_count` (axis C): items moved per repetition
- `execution.io_modes`: `bytes` = **SPSC**, `stream` = **MPMC** (legacy name; not I/O)

Type ids: `message` · `document` · `telemetry` · `strings` · `event`
(catalog: `schemas/data_catalog_v2.yaml`).
Published type ids: `size_256` (256 B) · `size_4096` (4 KiB).
Older experiment folders still use `message` / `document` (same lengths).
Catalog: `schemas/data_catalog_v2.yaml`.

## Files

| File | Purpose |
|------|---------|
| `smoke.yaml` | CI / quick sanity (`message`, n=1, SPSC) |
| `default.yaml` | Publication matrix (all five types × [100, 1000], SPSC+MPMC) |
| `smoke.yaml` | CI / quick sanity (`size_256`, n=1, SPSC) |
| `default.yaml` | Publication matrix (256 B and 4 KiB × [100, 1000], SPSC+MPMC) |

## Usage

Expand Down
12 changes: 3 additions & 9 deletions config/library/default.yaml
Original file line number Diff line number Diff line change
@@ -1,18 +1,12 @@
# Run config: default (publication matrix)
id: default
description: "Five payload types × [100, 1000] items; SPSC, 2P2C, 1P4C, 4P1C, 4P4C"
description: "Two payload sizes (256 B, 4 KiB) × [100, 1000] items; SPSC, 2P2C, 1P4C, 4P1C, 4P4C"
data_model_version: 2

types:
- type_id: message
- type_id: size_256
type_config: {}
- type_id: document
type_config: {}
- type_id: telemetry
type_config: {}
- type_id: strings
type_config: {}
- type_id: event
- type_id: size_4096
type_config: {}

data_type_instance_count: [100, 1000]
Expand Down
4 changes: 2 additions & 2 deletions config/library/smoke.yaml
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
# Run config: smoke (CI / fast path)
id: smoke
description: "Minimal matrix — message payload, single instance, SPSC only"
description: "Minimal matrix — 256-byte payload, single instance, SPSC only"
data_model_version: 2

types:
- type_id: message
- type_id: size_256
type_config: {}

data_type_instance_count: [1]
Expand Down
Loading
Loading