Let learners decline a placement question instead of guessing - #302
Open
arqo123 wants to merge 1 commit into
Open
Let learners decline a placement question instead of guessing#302arqo123 wants to merge 1 commit into
arqo123 wants to merge 1 commit into
Conversation
The placement quiz offered four options and no way out, so a learner who did not know an item had to guess. With four options a guess is right about a quarter of the time, the scoring cannot tell it from knowledge, and the resulting level is too high — which then feeds the study plan, lesson difficulty and the conversation end-of-speech window. Each question now offers an explicit "I don't know" next to the options, kept visually apart so it does not read as a fifth answer. It is submitted as its own signal, `dont_know: true`, so the evaluator can separate a declared gap from a wrong answer: - A declared gap never counts as correct, whatever else the client sends. - A skill is a weakness when the learner declared a gap on at least half of its questions, even if the remaining answers keep its score above the usual threshold. - The legacy LLM evaluation prompt is told that a declared gap is reliable evidence the item is above the learner, while a wrong answer can still show partial knowledge. For the adaptive question selection a declared gap behaves exactly like a wrong answer. Removing the guess is the fix; penalising the honest answer on top of it would only trade one misplacement for another.
Owner
|
Hi, thank you for working on this and for the clear implementation. I noticed that the new “I don’t know” action may be difficult to discover and interact with, especially in the light theme. It uses 11px Since this is the learner’s only alternative to guessing, could you please give it stronger contrast and adequate padding while keeping it visually secondary to the answer options? Thanks again! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #294.
Problem
AdaptiveQuizCardrenders one button per option and nothing else, so a learner who does not know an item has exactly one legal move: guess. With four options a guess is right about a quarter of the time, and nothing downstream can tell a lucky guess from knowledge. The level that comes out is too high, and it is not only a label — it drives the study plan, lesson difficulty and the voice end-of-speech window.Change
An explicit way out. Each question now offers "I don't know" below the four options, styled as a muted secondary action so it does not read as a fifth answer.
Its own signal, not a wrong option. The card emits a sentinel that
lib/assessment-answers.tsturns into an answer record withdont_know: true;AnswerRecordon the backend gains the same field, defaulting tofalse, so existing clients are unaffected.The evaluator separates the two. In
evaluate_adaptive_quiz:correct and not dont_know);< 0.45threshold.The second rule fires in a narrow band (a declared gap is already an incorrect answer, so the score threshold catches most cases), but it is the case that matters here: two questions in a skill, one answered, one declared unknown. Strengths are unaffected — a skill at
>= 0.65cannot carry gaps on half its questions.The legacy LLM evaluation prompt — the one quoted in the issue — now states that a declared gap must never be scored as correct and is reliable evidence the item is above the learner, while an incorrect answer can still show partial knowledge. One sentence appended; nothing reworded.
Two deliberate judgment calls:
Tests
backend/tests/test_assessment_router.py— payloads withoutdont_knowbehave as before, a declared gap markedcorrect: truestill scores as incorrect and does not pass the level, the weakness rule fires on a half-gap skill and not on an equally-scoring skill without gaps, and a skill already weak by score is not listed twice.frontend/tests/lib/assessment-answers.test.ts— record building for correct, wrong and declared-gap answers; the sentinel cannot collide with a real option.frontend/tests/components/AdaptiveQuizCard.test.tsx— the option is rendered, choosing an option reports that option, and the declared gap reports the sentinel rather than any answer.Backend: 977 passed, coverage 85%. Frontend: 454 passed, lint and
tsc --noEmitclean.Docs
specs/phase-1-platform.instructions.md(placement flow and deterministic evaluation),specs/api-endpoints.instructions.md(/evaluatebody),specs/prompts.instructions.md(legacy evaluation prompt) and the lib listing inspecs/architecture-frontend.instructions.mdare updated.Two omissions on purpose: the module/test count numbers in
specs/architecture-frontend.instructions.mdare left alone, since PR #301 already edits those exact lines and bumping them here would make my own two PRs conflict;CHANGELOG.mdandspecs/version.mdare left to the maintainer as in the previous PRs.Not covered here
The end-of-level test (
/assessment/level-test) has the same forced-guess shape and its own quiz UI. It sharesAnswerRecord, so it keeps working unchanged and would only need its page updated — but it is a different flow from the placement test this issue is about, so I left it out.