docs: state the testing bar in CONVENTIONS, COMPONENT_SPEC, and AGENTS - #1381
Conversation
Lands the conventions from #1367 ahead of the test changes themselves, so the rules are in force for new work while that PR is still in review. CONVENTIONS.md § Testing gains the rules the audit behind #1367 showed were missing: a test must be able to fail (name the one-line change it catches), drive the interaction with a real event, assert behavior rather than Tailwind utility strings, pin contracts that cross files, cover the server render, and determinism. COMPONENT_SPEC.md §8 and the AGENTS.md diff-audit checklist point at the same bar. The Vitest configs come along because CONVENTIONS.md § Determinism now states that every test-bearing package sets `restoreMocks`, `unstubEnvs`, and `unstubGlobals`, and that an inline `mockRestore()` is therefore dead code. Without the configs that instruction is false, and it would tell a reader to delete teardown that is still load-bearing. All five packages the bullet names now set the flags. `packages/mantle` also pins TZ/locale in the config rather than only in its `test` script, so the pin survives a single-file run. `restoreMocks` restores spies *before each test*, which breaks a spy installed once in `beforeAll` — `integer-ticks.browser.test.tsx` recorded canvas `fillText` calls that way, so its per-test spy moves to `beforeEach`. That is the only test change here; it is required to keep the suite green, not new coverage. COMPONENT_SPEC.md's scope-and-status counts are restated against this branch (20 of 67 component directories ship no test file), not against #1367's post-audit numbers.
|
🚀 Deploy PreviewsUpdated 2026-07-29 21:34 UTC
|
There was a problem hiding this comment.
Pull request overview
Aligns repo documentation and test-runner configuration around a stricter, explicit testing bar (ability-to-fail, event-driven interaction, determinism), and makes Vitest/Playwright defaults match what the docs now promise.
Changes:
- Expands
CONVENTIONS.mdtesting guidance with concrete rules on interaction, assertions, determinism, and cross-file contracts. - Updates
COMPONENT_SPEC.mdandAGENTS.mdto point at (and restate) the same testing bar. - Hardens Vitest configs across packages/apps with centralized mock/env/global cleanup, pins locale/timezone for Mantle (including Chromium locale/timezone via Playwright context options), and adjusts one browser test to avoid
beforeAllspies underrestoreMocks.
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
packages/mantle/vitest.config.ts |
Adds centralized mock/global/env hygiene and pins deterministic locale/timezone (including Playwright context locale/timezoneId). |
packages/mantle/src/components/chart/integer-ticks.browser.test.tsx |
Moves a long-lived canvas spy from beforeAll to beforeEach to remain compatible with restoreMocks. |
packages/mantle-vite-plugins/vitest.config.ts |
Enables centralized mock/global/env cleanup for determinism across tests. |
packages/mantle-server-syntax-highlighter/vitest.config.ts |
Introduces Vitest config with centralized cleanup and TZ pin moved from scripts to config. |
packages/mantle-server-syntax-highlighter/package.json |
Removes script-level TZ=UTC since TZ is now configured in Vitest config. |
CONVENTIONS.md |
Documents the testing “bar” (tests must be able to fail), interaction-driven tests, and determinism rules. |
COMPONENT_SPEC.md |
Updates scope/status counts and strengthens the testing minimums to match CONVENTIONS.md. |
apps/www/vite.config.ts |
Enables centralized mock/global/env cleanup for determinism in app tests. |
AGENTS.md |
Updates the diff-audit checklist to reference the new testing bar and common failure modes. |
Lands the conventions from #1367 ahead of the test changes themselves, so the rules are in force for new work while that PR is still in review. Docs-only in intent — no shipped source changes, so no changeset.
What's here
CONVENTIONS.md § Testinggains the rules the audit behind #1367 showed were missing:toBeDefined()/not.toThrow()as a test's only assertion, tautologies, unmatchable queries, and call-count-blind spies.renderToStringfor SSR branches; evaluate stringified inline scripts.userEvent.setup(), no order dependence.COMPONENT_SPEC.md §8and theAGENTS.mddiff-audit checklist point at the same bar.Why the Vitest configs came along
CONVENTIONS.md § Determinismnow states that every test-bearing package setsrestoreMocks,unstubEnvs, andunstubGlobals, and that a trailingspy.mockRestore()is therefore dead code. Without the configs that sentence is false, and it would tell a reader to delete teardown that is still load-bearing. All five packages the bullet names now set the flags.packages/mantlealso pinsTZ/LC_ALLin the config rather than only in itstestscript, so the pin survives a single-file or editor-driven run, and pins Chromium's ownlocale/timezoneIdthrough the PlaywrightcontextOptions, whichprocess.envcannot reach.The one test change
restoreMocksrestores spies before each test, which breaks a spy installed once inbeforeAll.integer-ticks.browser.test.tsxrecorded canvasfillTextcalls that way, so all 4 of its tests read an empty label array under the new config. Its spy moves tobeforeEach— the samebeforeAll→beforeEachhazard the new § Determinism bullet documents. This is required to keep the suite green, not new coverage; it was the only file in the repo with that pattern.Numbers restated against this branch
COMPONENT_SPEC.md's scope-and-status counts in #1367 describe the tree after its test work. Re-derived againstmain: 20 of 67 component directories ship no test file (not 13). The other three counts in that sentence — 67 directories, 20 with noasChild, 4 with nodata-slot, exactly one docs page with a data-attribute table — hold as written.Known pre-existing flake, left to #1367
code-block.test.tsx→ "fires onCopyError when clipboard write fails" fails under shuffled order (seed 7 of the 4 tried). It patchesnavigator.clipboard.writeTextbeforeuserEvent.setup(), which replacesnavigator.clipboardwholesale. Verified identical on cleanmainat the same seed — this branch neither causes nor worsens it, and #1367 fixes it. It is the exact violation the new "install spies afteruserEvent.setup()" bullet describes.Verification
```
lint 0 errors
fmt:check clean (722 files)
typecheck 7/7 tasks, 0 errors
test 8/8 tasks green
@ngrok/mantle 127 files, 1821 tests
@app/www 21 files, 169 tests
@ngrok/mantle-vite-plugins 5 files, 123 tests
@ngrok/mantle-server-syntax-highlighter 7 files, 109 tests
shuffle seeds 1, 42, 1234 green; seed 7 hits the pre-existing code-block flake above
```