Skip to content

Let learners decline a placement question instead of guessing - #302

Open
arqo123 wants to merge 1 commit into
artcc:developfrom
arqo123:feat/placement-test-dont-know
Open

Let learners decline a placement question instead of guessing#302
arqo123 wants to merge 1 commit into
artcc:developfrom
arqo123:feat/placement-test-dont-know

Conversation

@arqo123

@arqo123 arqo123 commented Aug 19, 2026

Copy link
Copy Markdown

Closes #294.

Problem

AdaptiveQuizCard renders one button per option and nothing else, so a learner who does not know an item has exactly one legal move: guess. With four options a guess is right about a quarter of the time, and nothing downstream can tell a lucky guess from knowledge. The level that comes out is too high, and it is not only a label — it drives the study plan, lesson difficulty and the voice end-of-speech window.

Change

An explicit way out. Each question now offers "I don't know" below the four options, styled as a muted secondary action so it does not read as a fifth answer.

Its own signal, not a wrong option. The card emits a sentinel that lib/assessment-answers.ts turns into an answer record with dont_know: true; AnswerRecord on the backend gains the same field, defaulting to false, so existing clients are unaffected.

The evaluator separates the two. In evaluate_adaptive_quiz:

  • a declared gap never counts as correct, whatever the client sends alongside it (correct and not dont_know);
  • a skill is a weakness when the learner declared a gap on at least half of its questions, even when the remaining answers keep its score above the usual < 0.45 threshold.

The second rule fires in a narrow band (a declared gap is already an incorrect answer, so the score threshold catches most cases), but it is the case that matters here: two questions in a skill, one answered, one declared unknown. Strengths are unaffected — a skill at >= 0.65 cannot carry gaps on half its questions.

The legacy LLM evaluation prompt — the one quoted in the issue — now states that a declared gap must never be scored as correct and is reliable evidence the item is above the learner, while an incorrect answer can still show partial knowledge. One sentence appended; nothing reworded.

Two deliberate judgment calls:

  • For the adaptive question selection a declared gap behaves exactly like a wrong answer. Stepping down faster would be a new penalty on the honest answer and risks under-placement on a single unknown word; the issue scoped itself smaller than scoring-model changes, and removing the guess is the fix.
  • The level assignment rule (highest level with >= 2 questions and >= 60% correct) is untouched. A declared gap is already an incorrect answer there, so any extra gap rule on top of it would be unreachable.

Tests

  • backend/tests/test_assessment_router.py — payloads without dont_know behave as before, a declared gap marked correct: true still scores as incorrect and does not pass the level, the weakness rule fires on a half-gap skill and not on an equally-scoring skill without gaps, and a skill already weak by score is not listed twice.
  • frontend/tests/lib/assessment-answers.test.ts — record building for correct, wrong and declared-gap answers; the sentinel cannot collide with a real option.
  • frontend/tests/components/AdaptiveQuizCard.test.tsx — the option is rendered, choosing an option reports that option, and the declared gap reports the sentinel rather than any answer.

Backend: 977 passed, coverage 85%. Frontend: 454 passed, lint and tsc --noEmit clean.

Docs

specs/phase-1-platform.instructions.md (placement flow and deterministic evaluation), specs/api-endpoints.instructions.md (/evaluate body), specs/prompts.instructions.md (legacy evaluation prompt) and the lib listing in specs/architecture-frontend.instructions.md are updated.

Two omissions on purpose: the module/test count numbers in specs/architecture-frontend.instructions.md are left alone, since PR #301 already edits those exact lines and bumping them here would make my own two PRs conflict; CHANGELOG.md and specs/version.md are left to the maintainer as in the previous PRs.

Not covered here

The end-of-level test (/assessment/level-test) has the same forced-guess shape and its own quiz UI. It shares AnswerRecord, so it keeps working unchanged and would only need its page updated — but it is a different flow from the placement test this issue is about, so I left it out.

The placement quiz offered four options and no way out, so a learner who
did not know an item had to guess. With four options a guess is right
about a quarter of the time, the scoring cannot tell it from knowledge,
and the resulting level is too high — which then feeds the study plan,
lesson difficulty and the conversation end-of-speech window.

Each question now offers an explicit "I don't know" next to the options,
kept visually apart so it does not read as a fifth answer. It is
submitted as its own signal, `dont_know: true`, so the evaluator can
separate a declared gap from a wrong answer:

- A declared gap never counts as correct, whatever else the client sends.
- A skill is a weakness when the learner declared a gap on at least half
  of its questions, even if the remaining answers keep its score above
  the usual threshold.
- The legacy LLM evaluation prompt is told that a declared gap is
  reliable evidence the item is above the learner, while a wrong answer
  can still show partial knowledge.

For the adaptive question selection a declared gap behaves exactly like a
wrong answer. Removing the guess is the fix; penalising the honest answer
on top of it would only trade one misplacement for another.
@artcc

artcc commented Aug 19, 2026

Copy link
Copy Markdown
Owner

Hi, thank you for working on this and for the clear implementation.

I noticed that the new “I don’t know” action may be difficult to discover and interact with, especially in the light theme. It uses 11px text-fl-muted-4 text with no vertical padding, resulting in low contrast and a very small touch target.

Since this is the learner’s only alternative to guessing, could you please give it stronger contrast and adequate padding while keeping it visually secondary to the answer options?

Thanks again!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Placement test forces guessing: no "I don't know" option, and guesses inflate the assigned CEFR level

2 participants