Skip to content

feat(vision): state the box extent in the grounding prompts (gate item 3) - #11

Merged
AlexSKuznetsov merged 1 commit into
mainfrom
feat/grounding-extent-prompt
Aug 27, 2026
Merged

feat(vision): state the box extent in the grounding prompts (gate item 3)#11
AlexSKuznetsov merged 1 commit into
mainfrom
feat/grounding-extent-prompt

Conversation

@AlexSKuznetsov

Copy link
Copy Markdown
Collaborator

What

Gate item 3 from docs/19-Post-Spike-Build-Plan.md. Every prompt that emits a box which then crops an object for embedding now carries one extent rule naming what to include — the whole object a person would point to (a keyring's ring/fob/lanyard, a mug's handle, glasses' arms), not its single most recognizable part.

Applied in reason/cosmos.py to:

  • _PROMPT — the event grounder (live query path, pipeline.py:290)
  • _LOCALIZE_PROMPT — the semantic_box re-grounding (also live, pipeline.py:387)
  • _TEMPORAL_LOCALIZE_PROMPT — enrollment

_REFERENCE_PROMPT emits no box (VALID/REJECT) and is untouched.

Why

  • Spike 12c: naming the extent takes the worst noun from grounding IoU 0.12 → 0.92 and kills a temperature-0 bimodal swing.
  • Spike 3c: the shipped wording conflated validity with extent and pinned the keys crop to the near-generic metal blade, costing identity F1 0.776 → 0.939 on the same images/backbone.

So the keys requirement now separates the two jobs: a metal key blade must be present for validity (keyboard still rejects to NO_OBJECT), but the box covers the whole keyring. The localizers' exclusion lists stop excluding the attached cord/lanyard/strap they should now include, keeping only hand/screen/background. The extent rule's closing clause ("never … the whole frame") guards the failure that held an earlier attempt, where "box the whole object" was read as "box everything" and re-admitted the keyboard.

Prompt versions bumped (reason-v2, localize-v4, temporal-v2) so recorded runs carry the wording.

Tests

  • Golden-prompt test asserts the extent rule is present on all three box-emitting prompts.
  • A second asserts the keys box is the whole keyring, not the blade alone.
  • 174 passed, 2 deselected; ruff + pyright clean.

Local validation (mechanism), and its limits

Ran a real LFM2.5-VL locally (native grounding prompt ± a short extent clause) over the 36 keys probe stills, measuring box-area fraction of frame (EXTENT.md's metric; no ground-truth boxes locally):

1.6B 3B (spike's graded model)
median box-area, native → +extent 0.072 → 0.101 (+40%) 0.099 → 0.113 (+14%)
widened / shrank / same (of 36) 29 / 5 / 2 19 / 15 / 2

In aggregate the extent sentence enlarges the box on both models — the item-3 mechanism. On the 3B it is bimodal per-image (19 up / 15 down), which independently reproduces the temperature-0 swing spike 12c flagged and said the validated wording is what kills.

Caveats (deliberate): the local run used a paraphrase clause on LFM's native prompt, not the committed production _EXTENT_RULE on the Cosmos-format prompt; and box-area is not IoU. The graded 0.12 → 0.92 IoU was measured on hosted Gemini at temp 0 and needs the validated prompt + ground truth on the H200 target to reproduce — not this laptop. The keyboard-rejection guard is likewise unverifiable locally (LFM never emits NO_OBJECT, per spike 3c).

🤖 Generated with Claude Code

Every prompt that emits a box which then crops an object for embedding now
carries one extent rule naming what to include: the whole object a person
would point to -- a keyring's ring, fob and lanyard, a mug's handle, glasses'
arms -- not its single most recognizable part. Applied to the event grounder
(_PROMPT) and both enrollment localizers (_LOCALIZE_PROMPT,
_TEMPORAL_LOCALIZE_PROMPT); the reference prompt emits no box and is untouched.

Spike 12c: naming the extent takes the worst noun from grounding IoU 0.12 ->
0.92 and kills a temperature-0 bimodal swing. Spike 3c: the shipped wording
conflated validity with extent and pinned the keys crop to the near-generic
metal blade, costing identity F1 0.776 -> 0.939 on the same images/backbone.

So the keys requirement now separates the two jobs -- a metal key blade must be
*present* for validity (keyboard still rejects to NO_OBJECT), but the box covers
the whole keyring. The localizers' exclusion lists stop excluding the attached
cord/lanyard/strap they should now include, keeping only hand/screen/background.

The extent rule's closing clause ("never ... the whole frame") guards the
failure that held an earlier attempt, where "box the whole object" was read as
"box everything" and re-admitted the keyboard. Prompt versions bumped
(reason v2, localize v4, temporal v2) so recorded runs carry the wording.

Tests: golden-prompt test asserts the extent rule is present on all three
box-emitting prompts; a second asserts the keys box is the whole keyring, not
the blade alone. 174 passed, ruff + pyright clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@AlexSKuznetsov
AlexSKuznetsov merged commit c6116e7 into main Aug 27, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant