Skip to content

Add a standalone Qwen3 video-training example on WebVid-10M - #169

Merged
amazloumi merged 1 commit into
multimodal-videofrom
feat/qwen3-vlm-hf-init
Jul 29, 2026
Merged

Add a standalone Qwen3 video-training example on WebVid-10M#169
amazloumi merged 1 commit into
multimodal-videofrom
feat/qwen3-vlm-hf-init

Conversation

@amazloumi

Copy link
Copy Markdown
Member

Summary

  • Reduces examples/qwen3_vlm/ to one working path: joint-decoder video captioning on WebVid-10M, started from pretrained components.
  • convert_hf_backbone.py (new) — builds the VLM via build_vlm_wrapper (SigLIP2 from HF, fresh adapter), copies HF Qwen3 LLM weights into the backbone, and saves the whole model as one DCP [checkpoint].load_path reads. Honors tied embeddings; dense targets only.
  • One config: vlm_qwen3_0.6b_joint_decoder_webvid.tomlavgpool (pool_window = 2) → 64 tokens/frame, so 18 frames × 64 = 1152 visual + 96 text = 1248 ≤ max_seq_len. LLM frozen; vision encoder + adapter train.
  • Removes convert_multimodal_checkpoint.py (+ test) and the four arch configs — the new converter is dense/JD-only, so those paths weren't runnable here.
  • README rewritten as a single walkthrough; configs use placeholder paths.
  • No kempnerforge/ changes; main README / CHANGELOG / docs untouched.

Net: +494 / −1342. Validated end to end — a run from an HF-init checkpoint completed one WebVid epoch (loss 13.6 → ~2.7) and produced sensible captions on held-out validation clips.

Testing

  • uv run ruff check kempnerforge/ tests/ passes
  • uv run ruff format --check kempnerforge/ tests/ scripts/ passes
  • uv run pyright kempnerforge/ passes (0 errors)
  • uv run pytest tests/unit/ -v --timeout=60 passes (1676 passed)
  • If distributed code changed — n/a
  • If training loop / parallelism / optimizers changed — n/a
  • uv run pytest examples/qwen3_vlm/tests/ passes (18 tests)

@codecov

codecov Bot commented Jul 25, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@amazloumi
amazloumi merged commit 2d57e97 into multimodal-video Jul 29, 2026
4 checks passed
@amazloumi
amazloumi deleted the feat/qwen3-vlm-hf-init branch July 29, 2026 14:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants