test: cover the quiz pipeline - #5
Merged
Merged
Conversation
The suite was one assertion against the root route. Every defect fixed in PRs #1 through #4 shipped past it, including one that returned HTTP 200 while generating questions about empty files - a status-code check could never have caught that. 61 tests: 51 backend under pytest, 10 frontend under node:test. The cases encode the bugs that actually happened rather than chasing coverage. Selection is pinned against the ascending sort that picked empty __init__.py files, and against the prefix match that let nested vendored directories through - together with the substring fix that was rejected, since "dist/" must not match "mydist/". Parsing is pinned against a malformed complexity object reaching a required schema field and 500ing the request. The API tests pin that /submit returns a follow-up and not a score, so anything relying on the old contract fails loudly, and that a client omitting the paste fields still succeeds. One test documents rather than asserts correctness: pasting after a long pause clears the 100ms timing guard. Pinning it makes a future change to the thresholds a decision instead of an accident. Every test is hermetic. Gemini and Mongo are mocked throughout, so the suite spends no API quota, needs no credentials, and passes in CI where no .env exists - verified by running it from the repo root the way CI invokes it, where .env does not resolve. Three things beyond the tests themselves: tests/test_quiz.py is removed. Its docstring described it as a placeholder to replace once mocked tests existed, and its only assertion now lives in test_quiz_api.py. backend/pytest.ini adds asyncio_mode = auto for the async service tests, and filters the warning the EOL google-generativeai package prints on every import. CI now runs the frontend tests, which otherwise would not run at all, and uses npm ci rather than npm install. That is the correct pairing with the lockfile committed in PR #2 - npm install can silently update it, which defeats the point of committing it. The frontend suite uses Node's built-in runner, so this adds no dependency. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PET9qKZXhgjEbZK7MReYQj
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The suite was one assertion against the root route. Every defect fixed in PRs #1–#4
shipped past it — including one that returned
HTTP 200while generating questionsasking why empty files were empty. A status-code check could never have caught that.
61 tests: 51 backend (pytest), 10 frontend (
node:test).test_gemini_client.pytest_github_client.pySKIP_DIRS, branch fallbacktest_quiz_service.pycreate_quiz, follow-up round, gradingtest_quiz_api.pypasteDetect.test.jsThe cases encode bugs that actually happened
Rather than chasing coverage, these pin the regressions:
test_prefers_largest_files_and_drops_stubs— the ascending sort that selected empty__init__.pyfiles while still returning 200test_excludes_nested_vendored_directoriesandtest_does_not_exclude_lookalike_paths— the prefix-match bug, plus the substring fix that was rejected:
dist/must notmatch
mydist/test_unusable_complexity_degrades_but_keeps_questions— a malformed reply reaching arequired schema field and 500ing the request
test_generate_preserves_category_through_the_response_model—response_modelsilently stripping undeclared fields
test_submit_returns_a_followup_and_no_score— the contract change, so anythingrelying on score-at-submit fails loudly
test_submit_defaults_paste_flags_when_client_omits_them— an older client must not 500One test documents rather than asserts correctness:
known gap: pasting after a long pause clears the timing guard. Pinning it makes afuture threshold change a decision instead of an accident.
Hermetic by construction
Gemini and Mongo are mocked throughout. The suite spends no API quota, needs no
credentials, and passes where no
.envexists — verified by running it from the reporoot exactly as CI invokes it, where
.envdoes not resolve and credentials aregenuinely empty.
This matters practically: the Gemini free tier allows 20 requests/day/model, and a
single full quiz costs 3. A suite that hit the real API would exhaust the day's budget
in seven runs.
Three changes beyond the tests
tests/test_quiz.pyremoved. Its own docstring described it as a placeholder toreplace once mocked tests existed. Its only assertion now lives in
test_quiz_api.py.backend/pytest.iniadded —asyncio_mode = autofor the async service tests, plusa filter for the warning the EOL
google-generativeaipackage prints on every import.CI now runs the frontend tests, which otherwise would not run at all, and uses
npm cirather thannpm install. That is the correct pairing with the lockfilecommitted in PR #2 —
npm installcan silently update it, defeating the point ofcommitting it. Confirmed the lockfile stays in sync after the
package.jsonedit(
scriptsis not tracked in lockfile v3).The frontend suite uses Node's built-in runner, so this adds no dependency and works
on CI's Node 20.
Still not covered
The React components themselves.
pasteDetect.jsis pure and fully tested, butQuestionCardandQuizPage— timers, auto-submit on expiry, the two-phase form — haveno tests, because component testing needs a DOM environment and a test renderer that
this repo does not have. That is a real gap, and it overlaps with the interactive
browser flow that has never been manually run either.
🤖 Generated with Claude Code
https://claude.ai/code/session_01PET9qKZXhgjEbZK7MReYQj