Skip to content

Migrate vlm evaluation pipeline to examples - #147

Merged
camilobrownpinilla merged 9 commits into
multimodal-videofrom
refactor/migrate-eval-to-examples
Jul 30, 2026
Merged

Migrate vlm evaluation pipeline to examples#147
camilobrownpinilla merged 9 commits into
multimodal-videofrom
refactor/migrate-eval-to-examples

Conversation

@camilobrownpinilla

Copy link
Copy Markdown
Collaborator

Summary

Moves eval adapter and script from kempnerforge core to examples.

Testing

  • uv run ruff check kempnerforge/ tests/ passes
  • uv run ruff format --check kempnerforge/ tests/ scripts/ passes
  • uv run pyright kempnerforge/ passes (0 errors)
  • uv run pytest tests/unit/ -v --timeout=60 passes

Closes #143

@codecov

codecov Bot commented Jul 15, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

Files with missing lines Coverage Δ
kempnerforge/config/vlm.py 100.00% <ø> (ø)
kempnerforge/data/vlm_dataset.py 99.11% <ø> (+0.02%) ⬆️

... and 6 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@camilobrownpinilla
camilobrownpinilla marked this pull request as ready for review July 15, 2026 19:14
@camilobrownpinilla camilobrownpinilla linked an issue Jul 23, 2026 that may be closed by this pull request
Comment thread examples/vlm-evaluation/vlm_eval_harness.py
Comment thread CHANGELOG.md Outdated
Comment thread docs/how-to/index.md
Comment thread examples/vlm-evaluation/adapter.py Outdated
Comment thread examples/vlm-evaluation/vlm_eval_harness.py Outdated

@amazloumi amazloumi left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just fix the minor issue below and then feel free to merge it to multimodal-video branch.

Comment thread examples/vlm-evaluation/adapter.py Outdated
return
return metadata
metadata["step"] = meta.get("step")
metadata["tokens_seen"] = meta.get("tokens_seen")

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this two lines are unprotected. Move them into try and have the except to catch ValueError

try:
      meta = json.loads(meta_file.read_text())
      metadata["step"] = meta.get("step")
      metadata["tokens_seen"] = meta.get("tokens_seen")
  except (ValueError, OSError, AttributeError) as exc:
      logger.warning(f"Could not read {meta_file}: {exc}")
      return metadata

@camilobrownpinilla
camilobrownpinilla merged commit 0033dea into multimodal-video Jul 30, 2026
6 checks passed
@camilobrownpinilla
camilobrownpinilla deleted the refactor/migrate-eval-to-examples branch July 30, 2026 19:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Migrate vlm evaluation pipeline to examples

2 participants