Skip to content

Optimize video I/O with seek decoding instead of serial decoding - #172

Merged
camilobrownpinilla merged 5 commits into
multimodal-videofrom
decode-optimization
Aug 10, 2026
Merged

Optimize video I/O with seek decoding instead of serial decoding#172
camilobrownpinilla merged 5 commits into
multimodal-videofrom
decode-optimization

Conversation

@camilobrownpinilla

Copy link
Copy Markdown
Collaborator

Summary

decode_video_frames previously decoded every frame of a clip serially to extract $N$ sampled frames. It now seeks to the keyframe at or before each target timestamp and decodes forward only to that frame, so decode cost scales with frames kept rather than clip length. Any seek that cannot guarantee identical selection (no time_base, missing PTS, seek landing past its target, FFmpeg error) falls back to a single serial pass — the retained reference implementation — so seeking only ever changes speed, never output.


Benchmarks

Frame-selection parity is byte-identical to serial in all cases below. Decode counts via an instrumented container; times single-threaded.

Clip serial seek speedup
Real WebVid H.264 clip (11.3s, 8 targets) 282 frames / 128ms 18 frames / 53ms 15.7× fewer decodes
Synthetic, 0.4s GOPs (WebVid-like), 8 tgts/12s 120 20
Synthetic, 3s GOPs (MLVU-like), 8 tgts/30s 300 127 2.4×
MLVU end-to-end (prior measurement) 11–22×

Known tradeoff: dense sampling inside long GOPs re-decodes the GOP prefix per target (e.g. 5s GOPs with 16 targets/30s: 465 vs 300 serial). Real workloads sample sparsely relative to GOP size, so this doesn't occur there; if it shows up in profiling, the fix is a demux-pass keyframe planner, not a heuristic (a stateful-cursor variant was prototyped, benchmarked at zero gain on sparse workloads, and rejected).

Testing

  • uv run ruff check passes
  • uv run ruff format --check passes
  • uv run pyright kempnerforge/ passes
  • uv run pytest tests/unit/ -v passes (N tests, 0 failures)

Closes #171

Comment thread kempnerforge/data/video_io.py Outdated
Comment thread kempnerforge/data/video_io.py
@amazloumi

Copy link
Copy Markdown
Member

Its look good to me. just two minor comment that can be fixed before merging.

@codecov

codecov Bot commented Aug 10, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 89.70588% with 7 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
kempnerforge/data/video_io.py 89.70% 4 Missing and 3 partials ⚠️
Files with missing lines Coverage Δ
kempnerforge/data/video_io.py 87.15% <89.70%> (+7.15%) ⬆️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@camilobrownpinilla
camilobrownpinilla merged commit aab8ce6 into multimodal-video Aug 10, 2026
6 checks passed
@camilobrownpinilla
camilobrownpinilla deleted the decode-optimization branch August 10, 2026 19:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants