feat: anti-gaming — timer, paste blocking, adaptive follow-up - #3
Merged
Conversation
The obvious attack on a comprehension quiz is to paste the question into a chatbot and paste the answer back. Three measures raise the cost of that. Per-question 75s clock. It starts when the question renders and is computed from wall-clock time rather than accumulated ticks, so a backgrounded tab cannot buy extra seconds by having its interval throttled. On expiry the answer commits as-is, blank included, and the field locks; when every question has expired the quiz submits itself. Time remaining at commit is sent with each answer. Paste is blocked in the answer field. Both paste and drop are prevented, since drag-and-drop text would otherwise walk straight past a paste-only guard. One adaptive follow-up, asked before anything is graded. The answer least likely to have been typed by its author is selected by weighting typing rate by length - (length/elapsed) * length - so a suspiciously fast essay outranks a fast one-liner, with fallbacks to the first answered question and then the first question so the round always happens even with no timing data. A single follow-up is generated that quotes that answer's specific wording back and pushes on what it implies. Grading is deferred until after this round on purpose: grading at submit would let a candidate bank a score and walk away from the round they cannot pass. This restructures the flow into generate -> submit -> followup. /submit no longer returns a score, it returns the follow-up question; /followup returns the final result in the shape /submit previously returned, so ScoreResult needs no changes. Answers, timing, and the follow-up exchange are all persisted on the attempt, and status gains an awaiting_followup state between generated and graded. Measured on psf/requests with the same repo and time budget: a pasted AI answer committed with 63s of 75s left, unable to defend its own wording, scored 0/100; a genuine author typing distinct answers and defending the follow-up scored 100/100, with the follow-up appearing as its own row in the breakdown. Known gap, documented in the README: the timer and paste block are client-side and do not stop anyone calling the API directly with a forged seconds_left. The follow-up is the measure that actually holds, since it demands understanding at response time regardless of transport. Server-issued timestamps are the real fix and are not built here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PET9qKZXhgjEbZK7MReYQj
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The obvious attack on a comprehension quiz is to paste the question into a chatbot and
paste the answer back. Three measures raise the cost of that, and the flow is
restructured to
generate → submit → followup.Measured result
Same repo (
psf/requests), same time budget, same flow:The follow-up quotes the candidate's wording back, which is what makes it un-fakeable:
The genuine run scored
10/10 — "Exceptional defense under pressure"on the follow-up row.1. Per-question 75s clock
Starts when the question renders. Computed from wall-clock time, not accumulated
ticks — a backgrounded tab can't buy seconds by having its interval throttled. On
expiry the answer commits as-is (blank included) and the field locks; when every
question has expired the quiz submits itself. Time remaining at commit rides along with
each answer.
2. Paste blocked
pasteanddropare both prevented. Drag-and-drop text walks straight past apaste-only guard.
3. Adaptive follow-up, before grading
The answer least likely to have been typed by its author is selected by weighting typing
rate by length —
(length/elapsed) × length— so a suspiciously fast essay outranks afast one-liner. Falls back to the first answered question, then the first question, so
the round always happens even with no timing data.
Grading is deferred until after this round on purpose. Grading at submit would let a
candidate bank a score and walk away from the round they can't pass.
API change
/submitno longer returns a score — it returns the follow-up question./followupreturns the final result in the shape
/submitpreviously returned, soScoreResultneeded no changes. Attempt documents gain
answers,followup, and anawaiting_followupstatus betweengeneratedandgraded.What this does NOT do
The timer and paste block are client-side. They raise the cost of casual cheating;
they do not stop anyone calling the API directly with a forged
seconds_left. Thefollow-up round is the measure that actually holds, because it demands understanding at
response time regardless of transport. Server-issued timestamps at generation, with
elapsed time computed server-side, are the real fix and are not built here. This is
stated plainly in the README rather than footnoted — overstating it would be a security
claim that isn't true.
Operational note
The follow-up adds a third Gemini call per quiz. On the free tier's 20 requests/day/model
cap that means roughly 6 complete quizzes per day, down from 10.
Verification
pytestpasses,vite buildclean, documents persist throughgenerated → awaiting_followup → gradedwith answers and follow-up recorded, test datacleared. Suspect-selection heuristic verified against four cases including no-timing-data
and all-blank fallbacks.
Not verified: the interactive browser flow. The Chrome extension would not connect,
so frontend checks were server-rendered. This PR adds the most timing-sensitive UI in the
project — countdowns, auto-submit on expiry, a two-phase form — and none of it has been
exercised by an actual click. Worth a manual pass before merge.
🤖 Generated with Claude Code
https://claude.ai/code/session_01PET9qKZXhgjEbZK7MReYQj