feat(vision): state the box extent in the grounding prompts (gate item 3) - #11
Merged
Merged
Conversation
Every prompt that emits a box which then crops an object for embedding now
carries one extent rule naming what to include: the whole object a person
would point to -- a keyring's ring, fob and lanyard, a mug's handle, glasses'
arms -- not its single most recognizable part. Applied to the event grounder
(_PROMPT) and both enrollment localizers (_LOCALIZE_PROMPT,
_TEMPORAL_LOCALIZE_PROMPT); the reference prompt emits no box and is untouched.
Spike 12c: naming the extent takes the worst noun from grounding IoU 0.12 ->
0.92 and kills a temperature-0 bimodal swing. Spike 3c: the shipped wording
conflated validity with extent and pinned the keys crop to the near-generic
metal blade, costing identity F1 0.776 -> 0.939 on the same images/backbone.
So the keys requirement now separates the two jobs -- a metal key blade must be
*present* for validity (keyboard still rejects to NO_OBJECT), but the box covers
the whole keyring. The localizers' exclusion lists stop excluding the attached
cord/lanyard/strap they should now include, keeping only hand/screen/background.
The extent rule's closing clause ("never ... the whole frame") guards the
failure that held an earlier attempt, where "box the whole object" was read as
"box everything" and re-admitted the keyboard. Prompt versions bumped
(reason v2, localize v4, temporal v2) so recorded runs carry the wording.
Tests: golden-prompt test asserts the extent rule is present on all three
box-emitting prompts; a second asserts the keys box is the whole keyring, not
the blade alone. 174 passed, ruff + pyright clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Gate item 3 from docs/19-Post-Spike-Build-Plan.md. Every prompt that emits a box which then crops an object for embedding now carries one extent rule naming what to include — the whole object a person would point to (a keyring's ring/fob/lanyard, a mug's handle, glasses' arms), not its single most recognizable part.
Applied in
reason/cosmos.pyto:_PROMPT— the event grounder (live query path,pipeline.py:290)_LOCALIZE_PROMPT— thesemantic_boxre-grounding (also live,pipeline.py:387)_TEMPORAL_LOCALIZE_PROMPT— enrollment_REFERENCE_PROMPTemits no box (VALID/REJECT) and is untouched.Why
So the keys requirement now separates the two jobs: a metal key blade must be present for validity (keyboard still rejects to
NO_OBJECT), but the box covers the whole keyring. The localizers' exclusion lists stop excluding the attached cord/lanyard/strap they should now include, keeping only hand/screen/background. The extent rule's closing clause ("never … the whole frame") guards the failure that held an earlier attempt, where "box the whole object" was read as "box everything" and re-admitted the keyboard.Prompt versions bumped (
reason-v2,localize-v4,temporal-v2) so recorded runs carry the wording.Tests
Local validation (mechanism), and its limits
Ran a real LFM2.5-VL locally (native grounding prompt ± a short extent clause) over the 36 keys probe stills, measuring box-area fraction of frame (EXTENT.md's metric; no ground-truth boxes locally):
In aggregate the extent sentence enlarges the box on both models — the item-3 mechanism. On the 3B it is bimodal per-image (19 up / 15 down), which independently reproduces the temperature-0 swing spike 12c flagged and said the validated wording is what kills.
Caveats (deliberate): the local run used a paraphrase clause on LFM's native prompt, not the committed production
_EXTENT_RULEon the Cosmos-format prompt; and box-area is not IoU. The graded 0.12 → 0.92 IoU was measured on hosted Gemini at temp 0 and needs the validated prompt + ground truth on the H200 target to reproduce — not this laptop. The keyboard-rejection guard is likewise unverifiable locally (LFM never emitsNO_OBJECT, per spike 3c).🤖 Generated with Claude Code