Add e2e + manual QA test harness - #2
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a55173c59e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| # The e2e layer needs the real server, so skip cleanly if it isn't installed | ||
| # (e.g. a minimal `pip install pytest` unit-only environment). | ||
| pytest.importorskip("uvicorn") |
There was a problem hiding this comment.
In an environment where uvicorn is not installed, this top-level pytest.importorskip runs while pytest is importing tests/e2e/conftest.py, so collection aborts with a skipped-exception error before marker selection; even the documented python -m pytest -m "not e2e" unit-only flow cannot run in a minimal unit-test environment. Put the dependency check in the e2e fixture/test modules or a collection hook that skips only the e2e tests.
Useful? React with 👍 / 👎.
Two new layers on top of the in-process unit suite: - tests/e2e/: boots the real ASGI app under uvicorn on an ephemeral port and drives it over a real socket (stdlib client). Validates routing/redirects, CORS over the wire, /speak dispatch, and PATCH /config redaction with a real config-file round-trip. Audio/model leaf calls stubbed (no torch/kokoro). Marked 'e2e'; skips cleanly if uvicorn absent. - docs/QA.md + scripts/qa_probe.py + scripts/smoke.sh: manual/dev layer for what CI can't do (real audio, model download, live keys, the extension). qa_probe drives a running daemon; smoke.sh boots it as a subprocess. 72 tests pass (51 unit + 21 e2e); 'pytest -m "not e2e"' runs the fast 51. uvicorn added to requirements-dev.txt; e2e marker registered in pytest.ini.
Since this branch was cut, main hardened the CORS policy: extension origins are no longer allowed by default and must be configured via allowed_origins / READOUT_ALLOWED_ORIGINS (see tests/test_server_cors.py, which configures the same test extension ID). The live-server fixture now sets READOUT_ALLOWED_ORIGINS to the test extension origin so the over-the-wire CORS assertions exercise the same contract.
a55173c to
a43f2ae
Compare
2 participants
Why
The merged PR #1 added an in-process unit/integration suite (Starlette
TestClient, heavy stack mocked). That never opens a socket or runs uvicorn's lifespan, and it can't touch real audio/model/extension. This PR adds the two layers that fill those gaps.Layer 1 — Live end-to-end (
tests/e2e/, CI-runnable)Boots the real ASGI app under uvicorn on an ephemeral port in a background thread and drives it with a stdlib HTTP client (no httpx/requests). Proves the contract holds through a real HTTP stack:
GET /→ 307 →/control;/controlserves HTML/status,/voicesshapes over the wire/speakdispatch and 422 validation;/stopPATCH /configredacts***in the response while round-tripping the real key to an actual config file on disk (the unique e2e check)nullorigins get no header, never*, preflight allows PATCH from the extension and is blocked for disallowed originsAudio/model leaf calls (
tts_engine.speak/stop_audio) are stubbed, so it needs no torch/kokoro/audio. Tests are markede2eand skip cleanly ifuvicornisn't installed.Layer 2 — Manual / dev QA (
docs/QA.md,scripts/)For what CI physically can't do:
docs/QA.md— structured checklist: first-run model download, real audio, file save, OpenAI/ElevenLabs with live keys, control panel, the Chrome extension, security smoke (issue 002/003), and the macOS 26 Tk skip (issue 001).scripts/qa_probe.py— zero-dep stdlib driver against a running daemon:status/voices/speak/stop/config/cors.speakplays real audio;cors --originreports ALLOWED/BLOCKED and warns if a plaintext key ever appears in a response.scripts/smoke.sh— boots the app as a subprocess (isolatedHOME), probes/status//voicesand asserts CORS rejects a hostile origin.Plumbing
uvicornadded torequirements-dev.txte2emarker registered inpytest.iniCLAUDE.mdTesting section documents all three layers + commandsTesting
python -m pytest→ 72 passed (51 unit + 21 e2e).python -m pytest -m "not e2e"→ 51 passed, 21 deselected.scripts/smoke.sh→ SMOKE PASSED. Verified locally; CI runs layers 1+2 on 3.10–3.12.