diff --git a/.grok/skills/implement-experiment/SKILL.md b/.grok/skills/implement-experiment/SKILL.md index 1444675..3f13f3f 100644 --- a/.grok/skills/implement-experiment/SKILL.md +++ b/.grok/skills/implement-experiment/SKILL.md @@ -40,9 +40,9 @@ export PATH="${HOME}/.local/go/bin:${HOME}/.cargo/bin:${HOME}/.dotnet:${HOME}/.l 2. **One question per folder.** Number it `NN-short-kebab` (next free `NN` after existing dirs). 3. **Edit `experiment.yaml` only** for sample, languages, libraries, run mode, grouping cut-offs. `run.yaml` is generated. Do not keep a second library list. 4. **Shared sample** at the experiment root (`sample.json`). Language folders hold **results** only (`results.md`, `results.json`). **Do not commit experiment logs** (CSVs under `/logs/`). Those stay on the machine that ran the timing. Saved results are enough. -5. **Do not compare times across languages.** Different runtimes. Directional only. Do not invent a size contest — payload bytes are the **data type**, not a library score. +5. **Do not compare times across languages.** Different runtimes. Directional only. Do not invent a size contest — payload size is the sample, not a library score. 6. **Do not crown a single winner.** Use `top_group` (similar / close / slower via Cliff’s delta vs the fastest library in the comparison set). Not “top 5%.” -7. **Textbook language** in `experiment.yaml` (`story.example`, `story.tradeoff`, `story.why`), `README.md`, `results.md`, the Dashboard, and the PLAN update. Write for a high-school student: complete sentences, no slang, no telegraphic fragments, no unexplained jargon. The Dashboard copies `story` from `experiment.yaml`. +7. **Textbook language** — follow `.grok/skills/improve-docs/references/STYLE.md` § Voice. That includes the finding you tell the user, not only `experiment.yaml` (`story.example`, `story.tradeoff`, `story.why`), `README.md`, `results.md`, the Dashboard, and the PLAN update. The Dashboard copies `story` from `experiment.yaml`. 8. **Do not overwrite** published site tables (`docs//results.md`) or dashboard `*_latest.json.gz` unless the user asked to publish suite numbers. 9. After the run, **update `experiments/PLAN.md`**: mark the item done, add an **After Experiment *n*** block, change later questions only if the finding requires it. Do not rewrite old findings. @@ -97,7 +97,7 @@ Copy Experiment 1’s file and change: |-----|-------------| | `id` | Folder name (`NN-short-kebab`) | | `title` / `question` | From the PLAN section (ordinary words) | -| `sample.kind` | `message` \| `document` \| `telemetry` \| `strings` \| `event` | +| `sample.kind` | `size_256` \| `size_4096` \| `message` \| `document` | | `sample.records_per_write` | `1` or `[1, 100]` | | `sample.settings` | Only extra knobs (e.g. `points: 128`). `{}` = catalog defaults | | `sample.seed` | `42` unless the PLAN says otherwise | diff --git a/.grok/skills/improve-docs/SKILL.md b/.grok/skills/improve-docs/SKILL.md index 12de277..9f265b2 100644 --- a/.grok/skills/improve-docs/SKILL.md +++ b/.grok/skills/improve-docs/SKILL.md @@ -87,7 +87,7 @@ Cut scope creep, ensure dismissible UI is not permanent noise, prefer small diff - Edit sources under `dashboard/` then **`npm run build`** in `dashboard/` so `docs/dashboard/` updates. - **README:** follow `README_EDITING.md` — short lede, compact audience table, section order Who → languages → Try it → Quick start; no slogan/meta CTA prose; no unsolicited role expansion. - MkDocs storefront (`docs/index.md`) only if asked or if it clearly contradicts a user-approved README fact. -- Follow `STYLE.md` (simple, no new frameworks). **Terminology: always “data type”, never “fixture”** — in user copy, comments, identifiers, and JSON keys. +- Follow `STYLE.md` (simple, no new frameworks). **Terminology:** published filter is **payload size**; never say “fixture”. Catalog ids stay type_id values. ### 2.5 Critique the implementation diff --git a/.grok/skills/improve-docs/references/STYLE.md b/.grok/skills/improve-docs/references/STYLE.md index 9e09833..4ee3f18 100644 --- a/.grok/skills/improve-docs/references/STYLE.md +++ b/.grok/skills/improve-docs/references/STYLE.md @@ -6,11 +6,12 @@ Keep it simple. Prefer delete and clarify over decorate. | Say | Never say | |-----|-----------| -| **data type** (`message`, `document`, `telemetry`, `strings`, `event`) | fixture, fixtures, fixtureKey, `dataset.fixtures` | +| **payload size** (256 B, 4 KiB) for the published filter | fixture; five leftover type names as if they were different shapes | +| **data type** only as the catalog id (`size_256`, `size_4096`) | fixture, fixtures, fixtureKey, `dataset.fixtures` | -This is the catalog payload shape. Use **data type** in README, Dashboard copy, skills, comments, function names, and new JSON keys. Do not keep `fixture*` as an internal alias. Old `configs.json` may still contain `fixtures`; read it as a fallback, write `data_types`. +The published axis is how many bytes each queued item is. Experiment 13 showed that 256 B and 4 KiB tell different stories; 512 B / 1 KiB / 2 KiB do not. Use **payload size** in README, Dashboard copy, and findings. Catalog ids stay `type_id` values (`size_256`, `size_4096`). Older folders may still say `message` / `document` (same lengths). Do not keep `fixture*` as an internal alias. Old `configs.json` may still contain `fixtures`; read it as a fallback, write `data_types`. -Do not measure or display payload **size** as a library result. Size is the data type, not a score. +Do not rank a library on payload size. Every queue in a cell moves the same bytes. Size is the sample, not a score. --- @@ -28,8 +29,8 @@ Do not measure or display payload **size** as a library result. Size is the data ## Content style - **One idea per paragraph.** Prefer tables for role/path matrices **on the site / Dashboard**, not by fattening the root README. -- **User terms:** data type, mode (bytes/stream), ops/s, latency, Pareto, baseline. -- **Never say “fixture”.** The catalog entry is a **data type** (`message`, `document`, …). Say it in docs, Dashboard copy, skills, comments, identifiers, and JSON keys. Do not keep `fixture*` as an internal alias “until later.” +- **User terms:** payload size (256 B, 4 KiB), mode (bytes/stream), ops/s, latency, Pareto, baseline. +- **Never say “fixture”.** The published sample is a **payload size**. The catalog id is a **data type** (`size_256`, `size_4096`). Say that in docs, Dashboard copy, skills, comments, identifiers, and JSON keys. Do not keep `fixture*` as an internal alias “until later.” - **Avoid in user copy:** fixture, median size (we do not measure payload size as a result), harness (prefer benchmark runner), unexplained IQR/P95. - **Honesty line** when ranks appear: within one language; cross-lang directional — prefer **one** place (e.g. Statistics / Method), not a second essay block on README. - **Links:** prefer site paths that match MkDocs nav labels (Dashboard, Learn, Method). Avoid “storefront” / “CTA” wording in user-facing labels. @@ -38,15 +39,17 @@ Do not measure or display payload **size** as a library result. Size is the data ### Voice - **Textbook quality for a high-school student.** Complete sentences. One idea, then the reason. Define a term the first time it is not everyday English. +- This Voice section applies to **replies to the user** as well as published copy. A chat explanation is reader-facing. - No slang (“at 3 a.m.”, “chatty ping”, “get user”). - No telegraphic fragments (“Keep YAML on disk. Convert once.”). Write the sentence out. - No unexplained jargon (stream, socket, pickle, gzip) unless the next sentence says what it is. +- **No clipped systems metaphors.** Do not write “pass a handle”, “pointer-bound”, “copy-bound”, “pack collapse”, “size barely moves them”, or “the knee” as if those were ordinary English. First say what the computer does, then what the clocks did. Example: the queue either copies every byte of the message, or it stores a short reference to a message that is already in memory. A larger message then either takes longer (because every byte is copied) or it does not (because only the reference is moved). - Direct, calm. Not marketing hype (“blazing”, “crush”). - “We measure …” not “We revolutionize …”. - **No slogan stacks** under the title (“Same A. Same B. Same C.”). - **No lede hedges** that argue with imaginary critics (“not marketing microbenchmarks”). -This rule applies to **every** reader-facing surface: Dashboard story cards (Why / Example / Trade-off), experiment YAML, `docs/experiments/`, Learn pages, README. The Dashboard reads Example and Trade-off from `experiments/*/experiment.yaml` via `sync-experiments.py`. +This rule applies to **every** reader-facing surface: replies to the user, Dashboard story cards (Why / Example / Trade-off), experiment YAML, `results.md`, `experiments/PLAN.md`, `docs/experiments/`, Learn pages, README. The Dashboard reads Example and Trade-off from `experiments/*/experiment.yaml` via `sync-experiments.py`. ### README-specific (authoritative) diff --git a/.grok/skills/prepare-pr/SKILL.md b/.grok/skills/prepare-pr/SKILL.md index d92c0f4..4121610 100644 --- a/.grok/skills/prepare-pr/SKILL.md +++ b/.grok/skills/prepare-pr/SKILL.md @@ -137,6 +137,7 @@ export PREPARE_PR_LANGS="${PREPARE_PR_LANGS:-$CHANGED_LANGS}" | `schemas/**` **except** prose (see below) | → **all** enabled languages | | `scripts/run-all-benchmarks.sh`, `scripts/lib/**`, `scripts/read-config.py`, `scripts/resolve_run_config.py` | → **all** | | `config/benchmark_config.yaml` | → **all** | +| `config/library/*` (default / smoke run matrix) | → **all** | | **Always ignored (never select langs, never force-all)** | | | `docs/**`, `dashboard/public/data/**`, `logs/**`, `reports/**`, `site/**`, `.grok/**` | meta / published artifacts | | `**/*.md`, `**/*.mdx`, `**/README*`, `LICENSE*` **anywhere** (including under `schemas/`, `scripts/`, or a language tree) | prose-only; e.g. terminology edits in `schemas/v2/README.md` must **not** re-bench the world | diff --git a/.grok/skills/prepare-pr/scripts/detect-changed-langs.sh b/.grok/skills/prepare-pr/scripts/detect-changed-langs.sh index cd28b76..ee08e8d 100755 --- a/.grok/skills/prepare-pr/scripts/detect-changed-langs.sh +++ b/.grok/skills/prepare-pr/scripts/detect-changed-langs.sh @@ -98,7 +98,7 @@ is_shared_force_all() { scripts/run-all-benchmarks.sh|scripts/lib/*|scripts/read-config.py|scripts/resolve_run_config.py|scripts/emit-cells-tsv.py) return 0 ;; - config/benchmark_config.yaml) + config/benchmark_config.yaml|config/library/*) return 0 ;; esac diff --git a/README.md b/README.md index ac67747..51dc79a 100644 --- a/README.md +++ b/README.md @@ -101,7 +101,8 @@ commit before `publish-docs` deploys the site. ## Test data -Shared **payload types**: `message`, `document`, `telemetry`, `strings`, and `event`. +Shared **payload sizes**: 256 B (`size_256`) and 4 KiB (`size_4096`). Every +runner builds an opaque byte string of that length. Catalog and defaults: `schemas/data_catalog_v2.yaml`. Run matrices: `config/library/`. diff --git a/analysis/README.md b/analysis/README.md index 7a36fb8..8be9a60 100644 --- a/analysis/README.md +++ b/analysis/README.md @@ -14,7 +14,7 @@ uv pip install -e . Each run writes a timestamped `YYYY-MM-DD-HHMMSS.csv` (never overwritten). Analysis picks the **latest** per language by default. -Suite type ids: `message`, `document`, `telemetry`, `strings`, `event`. +Suite type ids: `size_256` (256 B), `size_4096` (4 KiB). ```bash analyze-benchmarks diff --git a/analysis/src/benchmark_analysis/reports.py b/analysis/src/benchmark_analysis/reports.py index 5b8340c..7b97e9f 100644 --- a/analysis/src/benchmark_analysis/reports.py +++ b/analysis/src/benchmark_analysis/reports.py @@ -56,13 +56,27 @@ def _stat_get(entry: Any, field_id: str) -> Any: _FIXTURE_KEY_RE = re.compile(r"^(.*?)(?:@n=(\d+))+$", re.IGNORECASE) +_PAYLOAD_SIZE_LABELS = { + "size_1": "1 B", + "size_64": "64 B", + "size_256": "256 B", + "size_4096": "4 KiB", + "size_65536": "64 KiB", + "message": "256 B", + "document": "4 KiB", + "event": "512 B", + "telemetry": "1 KiB", + "strings": "2 KiB", +} + + def _format_data_type_display(label: str) -> str: - """Decode cryptic ``message@n=100`` keys for titles and table headers. + """Decode ``size_256@n=100`` keys for titles and table headers. Examples: - ``message@n=1`` → ``Message · 1 instance`` - ``message@n=100`` → ``Message · 100 instances`` - ``document`` → ``Document`` + ``size_256@n=1`` → ``256 B · 1 item`` + ``size_4096@n=100`` → ``4 KiB · 100 items`` + ``message`` → ``256 B`` """ s = str(label or "").strip() if not s: @@ -77,15 +91,16 @@ def _format_data_type_display(label: str) -> str: else: base, n = s, None - # Suite type_ids are lowercase words; older labels may already be Title/Pascal. - if base and base == base.lower() and re.fullmatch(r"[a-z][a-z0-9_]*", base): - pretty = base.replace("_", " ").title() - else: - pretty = base + pretty = _PAYLOAD_SIZE_LABELS.get(base) + if pretty is None: + if base and base == base.lower() and re.fullmatch(r"[a-z][a-z0-9_]*", base): + pretty = base.replace("_", " ").title() + else: + pretty = base if n is None: return pretty - unit = "instance" if n == 1 else "instances" + unit = "item" if n == 1 else "items" return f"{pretty} · {n} {unit}" @@ -1393,7 +1408,7 @@ def generate_language_results_pages( "", "| Term | Meaning |", "|------|---------|", - "| **data type** | Sample shape: `message`, `document`, `telemetry`, `strings`, or `event` " + "| **payload size** | How many bytes each item is: `size_256` (256 B) or `size_4096` (4 KiB) " "(CSV `TestDataName`) |", "| **bytes mode** | In-memory buffer API (encode to bytes / decode from a buffer). " "On C# this is often the **string** path — see [Modes](../analysis/modes.md). |", diff --git a/analysis/tests/test_run_config_v2.py b/analysis/tests/test_run_config_v2.py index a19867f..91f0ed9 100644 --- a/analysis/tests/test_run_config_v2.py +++ b/analysis/tests/test_run_config_v2.py @@ -21,10 +21,28 @@ _CATALOG = _REPO / "schemas" / "data_catalog_v2.yaml" +_CORE_TYPES = {"message", "document", "telemetry", "strings", "event"} +_SIZE_SWEEP_TYPES = { + "size_1": 1, + "size_64": 64, + "size_256": 256, + "size_4096": 4096, + "size_65536": 65536, +} + + def test_catalog_loads_five_types(): cat = load_catalog(_CATALOG) assert cat["data_model_version"] == 2 - assert set(cat["types"]) == {"message", "document", "telemetry", "strings", "event"} + assert _CORE_TYPES <= set(cat["types"]) + + +def test_catalog_size_sweep_types(): + cat = load_catalog(_CATALOG) + assert set(_SIZE_SWEEP_TYPES) <= set(cat["types"]) + for type_id, nbytes in _SIZE_SWEEP_TYPES.items(): + resolved = resolve_type_config(type_id, {}, cat) + assert resolved["payload_bytes"] == nbytes def test_resolve_empty_type_config_fills_defaults(): @@ -63,17 +81,19 @@ def test_type_config_hash_stable(): def test_smoke_expand_cell_count(): resolved = resolve_run_config(_LIBRARY / "smoke.yaml", catalog_path=_CATALOG, seed=42) - assert resolved["cell_count"] == 1 # message × [1] + assert resolved["cell_count"] == 1 # size_256 × [1] assert resolved["seed"] == 42 assert resolved["run_config"]["content_sha256"] ids = {(c["type_id"], c["data_type_instance_count"]) for c in resolved["cells"]} - assert ids == {("message", 1)} + assert ids == {("size_256", 1)} def test_default_expand_cell_count(): resolved = resolve_run_config(_LIBRARY / "default.yaml", catalog_path=_CATALOG) - # 5 types × 2 counts - assert resolved["cell_count"] == 10 + # 2 sizes × 2 counts + assert resolved["cell_count"] == 4 + ids = {c["type_id"] for c in resolved["cells"]} + assert ids == {"size_256", "size_4096"} counts = {c["data_type_instance_count"] for c in resolved["cells"]} assert counts == {100, 1000} diff --git a/c-sharp/scripts/run-benchmarks.sh b/c-sharp/scripts/run-benchmarks.sh index 2c98c24..520147b 100755 --- a/c-sharp/scripts/run-benchmarks.sh +++ b/c-sharp/scripts/run-benchmarks.sh @@ -25,7 +25,7 @@ else REPS="$(bench_mode_reps "$MODE")" if [[ "$MODE" == "smoke" ]]; then FILTER_SER="${FILTER_SER:-Queue+lock}" - FILTER_DATA="${FILTER_DATA:-message}" + FILTER_DATA="${FILTER_DATA:-size_256}" fi fi diff --git a/c/scripts/run-benchmarks.sh b/c/scripts/run-benchmarks.sh index 3488ab4..7c54746 100755 --- a/c/scripts/run-benchmarks.sh +++ b/c/scripts/run-benchmarks.sh @@ -25,7 +25,7 @@ else REPS="$(bench_mode_reps "$MODE")" if [[ "$MODE" == "smoke" ]]; then FILTER_SER="${FILTER_SER:-mutex-queue}" - FILTER_DATA="${FILTER_DATA:-message}" + FILTER_DATA="${FILTER_DATA:-size_256}" fi fi diff --git a/config/benchmark_config.yaml b/config/benchmark_config.yaml index ddd98dc..a81603f 100644 --- a/config/benchmark_config.yaml +++ b/config/benchmark_config.yaml @@ -53,30 +53,18 @@ data_model_v2: schemas_root: "schemas/v2" generate_script: "scripts/schemas/generate-all.sh" -# Payload shapes. Catalog is normative; names are type_id values. -# For queues these are payload sizes / shapes, not object graphs. +# Payload sizes. Catalog is normative; names are type_id values. +# For queues these are lengths of an opaque byte string, not object graphs. test_data: catalog_file: "schemas/data_catalog_v2.yaml" types: - - name: message - category: flat_mixed - description: "Small message payload (~256 B)" + - name: size_256 + category: opaque_bytes + description: "256-byte payload (small message)" supports_circular: false - - name: document - category: nested - description: "Larger document payload (~4 KiB)" - supports_circular: false - - name: telemetry - category: numeric_bulk - description: "Numeric bulk payload (~1 KiB)" - supports_circular: false - - name: strings - category: text_bulk - description: "Text bulk payload (~2 KiB)" - supports_circular: false - - name: event - category: envelope - description: "Event envelope payload (~512 B)" + - name: size_4096 + category: opaque_bytes + description: "4 KiB payload (copy cost starts to matter)" supports_circular: false # Pattern axis (CSV StringOrStream). Logged names are serializer ABI leftovers: diff --git a/config/library/README.md b/config/library/README.md index 82ab457..6a22369 100644 --- a/config/library/README.md +++ b/config/library/README.md @@ -6,15 +6,16 @@ Named **run configs** select the measurement matrix: - `data_type_instance_count` (axis C): items moved per repetition - `execution.io_modes`: `bytes` = **SPSC**, `stream` = **MPMC** (legacy name; not I/O) -Type ids: `message` · `document` · `telemetry` · `strings` · `event` -(catalog: `schemas/data_catalog_v2.yaml`). +Published type ids: `size_256` (256 B) · `size_4096` (4 KiB). +Older experiment folders still use `message` / `document` (same lengths). +Catalog: `schemas/data_catalog_v2.yaml`. ## Files | File | Purpose | |------|---------| -| `smoke.yaml` | CI / quick sanity (`message`, n=1, SPSC) | -| `default.yaml` | Publication matrix (all five types × [100, 1000], SPSC+MPMC) | +| `smoke.yaml` | CI / quick sanity (`size_256`, n=1, SPSC) | +| `default.yaml` | Publication matrix (256 B and 4 KiB × [100, 1000], SPSC+MPMC) | ## Usage diff --git a/config/library/default.yaml b/config/library/default.yaml index cf99f92..82a3bc9 100644 --- a/config/library/default.yaml +++ b/config/library/default.yaml @@ -1,18 +1,12 @@ # Run config: default (publication matrix) id: default -description: "Five payload types × [100, 1000] items; SPSC, 2P2C, 1P4C, 4P1C, 4P4C" +description: "Two payload sizes (256 B, 4 KiB) × [100, 1000] items; SPSC, 2P2C, 1P4C, 4P1C, 4P4C" data_model_version: 2 types: - - type_id: message + - type_id: size_256 type_config: {} - - type_id: document - type_config: {} - - type_id: telemetry - type_config: {} - - type_id: strings - type_config: {} - - type_id: event + - type_id: size_4096 type_config: {} data_type_instance_count: [100, 1000] diff --git a/config/library/smoke.yaml b/config/library/smoke.yaml index 822401e..6305864 100644 --- a/config/library/smoke.yaml +++ b/config/library/smoke.yaml @@ -1,10 +1,10 @@ # Run config: smoke (CI / fast path) id: smoke -description: "Minimal matrix — message payload, single instance, SPSC only" +description: "Minimal matrix — 256-byte payload, single instance, SPSC only" data_model_version: 2 types: - - type_id: message + - type_id: size_256 type_config: {} data_type_instance_count: [1] diff --git a/dashboard/index.html b/dashboard/index.html index 784871e..d1409bc 100644 --- a/dashboard/index.html +++ b/dashboard/index.html @@ -96,9 +96,9 @@ - Data type: + Payload size:
-
@@ -151,7 +151,7 @@