Feat/text multi image eval - #148
Open
camilobrownpinilla wants to merge 4 commits into
Open
Conversation
Pack N images from a chat request as an ordered clip (a single image is the 1-frame case), reusing the existing video frame/clip machinery. Images past frames_per_clip are truncated downstream, with a warning surfaced at render. Image checkpoints still reject multiple images; text-only stays out of scope. Flips the video-path multi-image test to expect frames and adds coverage for the truncation warning. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01STRsAc9nZjMhkLzYkkYp67
Add a text-only (no-visual) forward path for the generative VLM arches so generate_until text benchmarks (GSM8K, IFEval, ...) can measure text-backbone drift on image and video checkpoints, in the same lmms-eval harness as the multimodal tasks. Model core (backwards-compatible, no new parameters or state-dict keys): - vlm.py: widen pixel_values to `Tensor | None` across the ModalityStrategy protocol, all four strategies, and VLMWrapper.forward. A None (text-only) request yields an empty ModalityContext for joint_decoder/cross_attention/mot and is rejected for the non-generative moma arch. - transformer.py: skip cross-attention blocks when image_features is None; run the MoT branch with n_image=0 when there is neither a prefix nor modality_ids. - mot.py: reshape the MoT projections with explicit head counts so a zero-length image stream flows through (bit-identical for the non-empty path). Only previously-erroring paths change behavior; existing image/video forwards, checkpoints, and configs are untouched. Adapter (examples/vlm-evaluation): - _render_request renders a no-visual request as empty frames on both image and video checkpoints. - _generate_batch accepts pixel_values=None (skip vision, 0 image-token budget, device from the prompts). - generate_until segregates a gen_kwargs chunk into visual and text-only sub-batches, decodes each, and scatters results back in original order. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01STRsAc9nZjMhkLzYkkYp67
camilobrownpinilla
marked this pull request as ready for review
July 22, 2026 14:52
camilobrownpinilla
force-pushed
the
feat/text-multi-image-eval
branch
from
July 22, 2026 17:38
955929e to
c803dd0
Compare
amazloumi
requested changes
Jul 29, 2026
amazloumi
left a comment
amazloumi
left a comment
Member
There was a problem hiding this comment.
This PR adding two independent features and one fix. Please break it to three individual smaller PRs.
Base automatically changed from
refactor/migrate-eval-to-examples
to
multimodal-video
July 30, 2026 19:01
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Testing
uv run ruff check kempnerforge/ tests/passesuv run ruff format --check kempnerforge/ tests/ scripts/passesuv run pyright kempnerforge/passes (0 errors)uv run pytest tests/unit/ -v --timeout=60passesCloses #144