Skip to content

test: cover the quiz pipeline - #5

Merged
krishhimself merged 1 commit into
mainfrom
test/quiz-pipeline-coverage
Aug 22, 2026
Merged

test: cover the quiz pipeline#5
krishhimself merged 1 commit into
mainfrom
test/quiz-pipeline-coverage

Conversation

@krishhimself

Copy link
Copy Markdown
Owner

Summary

The suite was one assertion against the root route. Every defect fixed in PRs #1#4
shipped past it — including one that returned HTTP 200 while generating questions
asking why empty files were empty. A status-code check could never have caught that.

61 tests: 51 backend (pytest), 10 frontend (node:test).

File Tests Covers
test_gemini_client.py 16 response parsing, tier normalization, code-fence stripping
test_github_client.py 7 file selection, SKIP_DIRS, branch fallback
test_quiz_service.py 15 suspect selection, create_quiz, follow-up round, grading
test_quiz_api.py 13 HTTP contract, status mapping, signal forwarding
pasteDetect.test.js 10 paste detection and its false-positive cases

The cases encode bugs that actually happened

Rather than chasing coverage, these pin the regressions:

  • test_prefers_largest_files_and_drops_stubs — the ascending sort that selected empty
    __init__.py files while still returning 200
  • test_excludes_nested_vendored_directories and test_does_not_exclude_lookalike_paths
    — the prefix-match bug, plus the substring fix that was rejected: dist/ must not
    match mydist/
  • test_unusable_complexity_degrades_but_keeps_questions — a malformed reply reaching a
    required schema field and 500ing the request
  • test_generate_preserves_category_through_the_response_modelresponse_model
    silently stripping undeclared fields
  • test_submit_returns_a_followup_and_no_score — the contract change, so anything
    relying on score-at-submit fails loudly
  • test_submit_defaults_paste_flags_when_client_omits_them — an older client must not 500

One test documents rather than asserts correctness:
known gap: pasting after a long pause clears the timing guard. Pinning it makes a
future threshold change a decision instead of an accident.

Hermetic by construction

Gemini and Mongo are mocked throughout. The suite spends no API quota, needs no
credentials, and passes where no .env exists — verified by running it from the repo
root exactly as CI invokes it, where .env does not resolve and credentials are
genuinely empty.

This matters practically: the Gemini free tier allows 20 requests/day/model, and a
single full quiz costs 3. A suite that hit the real API would exhaust the day's budget
in seven runs.

Three changes beyond the tests

tests/test_quiz.py removed. Its own docstring described it as a placeholder to
replace once mocked tests existed. Its only assertion now lives in test_quiz_api.py.

backend/pytest.ini addedasyncio_mode = auto for the async service tests, plus
a filter for the warning the EOL google-generativeai package prints on every import.

CI now runs the frontend tests, which otherwise would not run at all, and uses
npm ci rather than npm install. That is the correct pairing with the lockfile
committed in PR #2npm install can silently update it, defeating the point of
committing it. Confirmed the lockfile stays in sync after the package.json edit
(scripts is not tracked in lockfile v3).

The frontend suite uses Node's built-in runner, so this adds no dependency and works
on CI's Node 20.

Still not covered

The React components themselves. pasteDetect.js is pure and fully tested, but
QuestionCard and QuizPage — timers, auto-submit on expiry, the two-phase form — have
no tests, because component testing needs a DOM environment and a test renderer that
this repo does not have. That is a real gap, and it overlaps with the interactive
browser flow that has never been manually run either.

🤖 Generated with Claude Code

https://claude.ai/code/session_01PET9qKZXhgjEbZK7MReYQj

The suite was one assertion against the root route. Every defect fixed in PRs #1
through #4 shipped past it, including one that returned HTTP 200 while generating
questions about empty files - a status-code check could never have caught that.

61 tests: 51 backend under pytest, 10 frontend under node:test.

The cases encode the bugs that actually happened rather than chasing coverage.
Selection is pinned against the ascending sort that picked empty __init__.py files,
and against the prefix match that let nested vendored directories through - together
with the substring fix that was rejected, since "dist/" must not match "mydist/".
Parsing is pinned against a malformed complexity object reaching a required schema
field and 500ing the request. The API tests pin that /submit returns a follow-up and
not a score, so anything relying on the old contract fails loudly, and that a client
omitting the paste fields still succeeds.

One test documents rather than asserts correctness: pasting after a long pause clears
the 100ms timing guard. Pinning it makes a future change to the thresholds a decision
instead of an accident.

Every test is hermetic. Gemini and Mongo are mocked throughout, so the suite spends no
API quota, needs no credentials, and passes in CI where no .env exists - verified by
running it from the repo root the way CI invokes it, where .env does not resolve.

Three things beyond the tests themselves:

tests/test_quiz.py is removed. Its docstring described it as a placeholder to replace
once mocked tests existed, and its only assertion now lives in test_quiz_api.py.

backend/pytest.ini adds asyncio_mode = auto for the async service tests, and filters
the warning the EOL google-generativeai package prints on every import.

CI now runs the frontend tests, which otherwise would not run at all, and uses npm ci
rather than npm install. That is the correct pairing with the lockfile committed in
PR #2 - npm install can silently update it, which defeats the point of committing it.
The frontend suite uses Node's built-in runner, so this adds no dependency.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PET9qKZXhgjEbZK7MReYQj
@krishhimself
krishhimself merged commit efceb3b into main Aug 22, 2026
2 checks passed
@krishhimself
krishhimself deleted the test/quiz-pipeline-coverage branch August 22, 2026 10:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant