Skip to content

feat(pipeline): widen the reasoning window to see placements (gate item 6) - #13

Merged
AlexSKuznetsov merged 1 commit into
mainfrom
feat/widen-placement-detection-window
Aug 28, 2026
Merged

feat(pipeline): widen the reasoning window to see placements (gate item 6)#13
AlexSKuznetsov merged 1 commit into
mainfrom
feat/widen-placement-detection-window

Conversation

@AlexSKuznetsov

Copy link
Copy Markdown
Collaborator

What

Placement detection reads change — the reasoner contrasts widely-separated frames and sees "holding" → "put down, walked away". A tight window shows only continuous motion toward the surface, so the transition is invisible. Spike 5b measured this: a ±5s window took detection 2/2 → 0/2 on real clips. The shipped reason_window_seconds was 6s — tighter still.

Knob Was Now Why
reason_window_seconds 6 20 Straddle the placement transition; stays under the 30s evidence ring
reason_max_frames 4 8 Over a 20s window, 4 frames sit ~7s apart and fall either side of the transition
event_cooldown_seconds 20 25 Wide windows now overlap, so one placement is seen by several — cooldown must cover a window span

Two model_validator invariants lock the coupling (window ≤ ring retention, cooldown ≥ window), with new test_config.py.

Local validation on LFM2.5-VL-3B (proxy for Cosmos)

Ran on spike 5b's own clips (clips/recordings/VID_20260819_1207{27,02}_{hi,win}.mp4; _hi = full 720p, _win = ±5s 720p), sampling frames endpoint-inclusive exactly as cosmos does:

object   arm      K=4   K=5   K=6   K=7   K=8
keys     narrow   NO    NO    NO    NO    NO
keys     full     NO    NO    NO    NO   PLACED
wallet   narrow   NO    NO    NO    NO    NO
wallet   full     NO   PLACED PLACED PLACED PLACED
  • The narrow ±5s window never detects the placement — at any frame count, either object. Window width is the binding constraint (reproduces 0/2), confirming the core fix.
  • A wide window detects it, but only if sampled densely enough: the large wallet from 5 frames, the small/thin keyring (the demo's hard case) only at 8. At production's reason_max_frames=4, even the wide clip missed both — so widening alone is not sufficient, which is why reason_max_frames is bumped too.

Caveats: LFM is a proxy for Cosmos; the absolute frame floor (8) is model-specific, so reason_max_frames is a knob to re-tune against real Cosmos (watching the 8192-token context and latency). The direction — narrow never works, density matters for small objects — is robust.

Notes

Cosmos already endpoint-subsamples the window (np.linspace), so widening the window costs tokens only through the frame-count bump, not the width. A wider window also yields a richer evidence clip (hold → place → walk away). Full suite 178 passed, ruff and pyright clean.

🤖 Generated with Claude Code

…em 6)

Placement detection reads *change*: the VLM contrasts widely-separated frames
and sees "holding" -> "put down, walked away". A tight window shows only
continuous motion toward the surface, so the transition is invisible. Spike 5b
measured this directly -- a +/-5 s window took placement detection from 2/2 to
0/2 on real glasses clips -- and the shipped `reason_window_seconds` was 6 s,
tighter still.

Local validation on LFM2.5-VL-3B (proxy for Cosmos), on spike 5b's own clips
(`clips/recordings/VID_20260819_1207{27,02}_{hi,win}.mp4`), sampling frames
endpoint-inclusive exactly as cosmos does:

  * The narrow +/-5 s window answered NOT_PLACED at every frame count 4..8 for
    both objects -- window WIDTH is the binding constraint, reproducing 0/2.
  * The wide clip detected the placement, but only when sampled densely enough:
    the large wallet from 5 frames, the small/thin keyring only at 8. At 4
    frames even the wide clip missed both -- widening alone is not sufficient.

So two coupled changes:
- reason_window_seconds 6 -> 20 (kept below the 30 s ring; spike bracketed the
  minimum between a ~10 s window that failed and a 28 s clip that passed).
- reason_max_frames 4 -> 8: over a 20 s window 4 frames sit ~7 s apart and fall
  either side of the transition; 8 (~3 s apart) recovered detection for the hard
  small-object case in the proxy. Stays within the 8192 context at some latency;
  exact floor is model-specific, so it is a knob to revisit against real Cosmos.
- event_cooldown_seconds 20 -> 25: wide windows overlap, so one placement is seen
  by several; the cooldown must cover a window span or it is rewritten once per
  interval across it.
- Two model-validator invariants lock the coupling: window <= ring retention and
  cooldown >= window. New test_config.py covers both.

Cosmos already endpoint-subsamples the window, so a wider window costs tokens
only through the frame-count bump, not the width. A wider window also yields a
richer evidence clip (hold -> place -> walk away). Full suite 178 passed, ruff
and pyright clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@AlexSKuznetsov
AlexSKuznetsov merged commit 9b05b5c into main Aug 28, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant