Fix 4 video-source / JoyVASA bugs: relative-motion formula, paste-back override, frame indexing, fps - #199
Open
wegylexy wants to merge 4 commits into
Open
Fix 4 video-source / JoyVASA bugs: relative-motion formula, paste-back override, frame indexing, fps#199wegylexy wants to merge 4 commits into
wegylexy wants to merge 4 commits into
Conversation
added 4 commits
August 13, 2026 21:11
…tuting absolute driving value instead of relative delta) For is_source_video mode, the exp/lip/eyes region and pose region branches were substituting the driving frame's raw/smoothed value directly (x_d_exp_smooth, R_d_i), discarding the source's own baseline expression/ rotation and never subtracting the driving clip's own frame-0 baseline. This desyncs frame 0 (no longer guaranteed to match source's resting pose, e.g. mouth not fully closed even when source's own rest pose is) and undershoots motion amplitude whenever the driving identity's resting expression differs from the source's. Fixed to use the same relative-delta formula as the still-image branch: source's own baseline + (driving's current frame - driving's own frame-0 baseline), smoothing after composing rather than substituting the raw value. Verified against a source video + driving video clip: frame 0 now matches the source's own resting expression exactly, and mouth-open amplitude at fast transients is much closer to the still-image branch's output on the same driving clip.
…ng --cfg run_with_video/run_with_pkl unconditionally did infer_cfg.infer_params.flag_pasteback = args.paste_back, and --paste_back defaults to False (action='store_true'). Any invocation without --paste_back therefore silently disabled paste-back regardless of the --cfg file's own flag_pasteback: True setting, making the output just the raw unmodified source video with no reprojected animation. Changed --paste_back's default to None and only override flag_pasteback when it's explicitly passed on the CLI, matching --cfg's flag_pasteback otherwise (same pattern used for the other CLI overrides in this repo, e.g. --flag_relative_motion).
…sources run_with_video/run_with_pkl in both run.py and api.py always indexed pipe.src_imgs[0]/pipe.src_infos[0] regardless of the current driving frame index, so when the source is itself a video (is_source_video=True), the source's own head motion never advanced past frame 0 - every driving frame was composited against the same static frame-0 source pose. Fixed by indexing pipe.src_imgs[frame_ind]/pipe.src_infos[frame_ind] when pipe.is_source_video, falling back to index 0 for still-image sources as before. Note: run.py's run_with_video/run_with_pkl and api.py's run_with_video were exercised end-to-end with an actual source+driving video pair; the run_with_pkl instance in api.py has the identical bug/fix pattern but wasn't independently run-tested (api.py's pickle-driving path wasn't exercised this session).
…output_fps When is_source_video was true, the rendered output was labeled with the source video's own fps (utils.get_video_info(self.source_path)) instead of the driving motion pickle's own output_fps. These can legitimately differ - e.g. JoyVASA generates motion at a fixed fps regardless of the source video's native fps - causing the render to run at the wrong effective frame rate and drift out of sync with the driving audio increasingly over the clip. Fixed to always use dri_motion_infos['output_fps'], which every driving pickle sets correctly at generation time regardless of source type (both run.py's pickle export and JoyVASA's own gen_motion_sequence set it). Verified with a source video + JoyVASA-generated driving pickle: audio and video duration now match to within 0.01s, both when JoyVASA was run against a still image (the fix is a no-op there, matching prior behavior) and against a video source (previously off by ~4% given a 24fps source vs. JoyVASA's 25fps motion).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Four bugs affecting
is_source_video(video-as-source) mode and the JoyVASA driving path, found and fixed while building a downstream CI pipeline on top of this project. Each is an isolated commit so they can be reviewed/merged independently if preferred.is_source_videorelative-motion formula (src/pipelines/faster_live_portrait_pipeline.py) — the exp/lip/eyes region and pose (R_new) branches substituted the driving frame's raw/smoothed value directly, discarding the source's own baseline and never subtracting the driving clip's own frame-0 baseline. This desyncs frame 0 (mouth not guaranteed closed even when source's resting pose is) and undershoots motion amplitude whenever the driving identity's resting expression/pose differs from the source's. Fixed to use the same relative-delta formula as the still-image branch: source's own baseline + (driving's current frame − driving's own frame-0 baseline).run.pyalways overridingflag_pasteback—run_with_video/run_with_pklunconditionally setinfer_cfg.infer_params.flag_pasteback = args.paste_back, and--paste_backdefaults toFalse. Any invocation without--paste_backsilently disabled paste-back regardless of the--cfgfile's ownflag_pasteback: True, making the output just the raw unmodified source. Fixed to only override when--paste_backis explicitly passed (default changed toNone).src_imgs[0]/src_infos[0]indexing (run.py,api.py) —run_with_video/run_with_pklalways indexed frame 0 regardless of the current driving frame, so a video source's own head motion never advanced past its first frame. Fixed by indexing[frame_ind]whenpipe.is_source_video. (Note:run.py's two call sites andapi.py'srun_with_videowere exercised end-to-end with a real source+driving video pair;api.py'srun_with_pklhas the identical pattern but wasn't independently run-tested.)src/pipelines/gradio_live_portrait_pipeline.py,run_pickle_driving) — whenis_source_videowas true, the render was labeled with the source video's fps instead of the driving pickle's ownoutput_fps. These can legitimately differ (JoyVASA generates motion at its own fixed fps regardless of source fps), causing audio/video drift that grows over the clip. Fixed to always usedri_motion_infos["output_fps"].All four were verified against upstream
masterbefore opening this PR (each file's pre-fix content matches this repo's current code exactly at the relevant lines).摘要(繁體中文)
在既有專案上搭建下游 CI pipeline 時,發現並修復了四個影響「以影片作為來源」(
is_source_video)模式與 JoyVASA 驅動路徑的錯誤。每個修正都是獨立的 commit,方便個別審查/合併。is_source_video相對運動公式錯誤(src/pipelines/faster_live_portrait_pipeline.py)— exp/lip/eyes 區域與 pose(R_new)分支直接套用驅動影格的原始(或平滑後)數值,捨棄了來源自身的基準值,也從未減去驅動影片自己第 0 幀的基準值。這會導致第 0 幀不同步(即使來源的靜止姿勢是閉嘴,也不保證閉嘴),且當驅動身分的靜止表情/姿勢與來源不同時,動作幅度會偏小。已修正為與靜態圖片分支相同的相對差值公式:來源自身基準值 +(驅動目前影格 − 驅動自己第 0 幀基準值)。run.py一律覆寫flag_pasteback—run_with_video/run_with_pkl無條件執行infer_cfg.infer_params.flag_pasteback = args.paste_back,而--paste_back預設為False。只要呼叫時未加上--paste_back,就會無聲地停用貼回效果,即使--cfg設定檔本身寫的是flag_pasteback: True,輸出也只會是原始未經處理的來源影片。已修正為僅在明確傳入--paste_back時才覆寫(預設值改為None)。src_imgs[0]/src_infos[0]索引固定不變(run.py、api.py)—run_with_video/run_with_pkl一律讀取第 0 幀,不論目前驅動到第幾幀,導致以影片作為來源時,來源自身的頭部動作永遠停在第一幀。已修正為當pipe.is_source_video時改用[frame_ind]索引。(註:run.py的兩處呼叫點與api.py的run_with_video已用實際的來源+驅動影片組合完整測試過;api.py的run_with_pkl具有相同的錯誤模式與修法,但尚未獨立實測。)src/pipelines/gradio_live_portrait_pipeline.py的run_pickle_driving)— 當is_source_video為真時,輸出影片使用的是「來源影片」的影格率,而非驅動 pickle 自己的output_fps。這兩者可能本來就不同(JoyVASA 產生動作時使用固定的影格率,與來源影片無關),導致音訊與影片隨著片長逐漸不同步。已修正為一律使用dri_motion_infos["output_fps"]。以上四項修正在送出這個 PR 前,皆已對照 upstream
master逐一確認(每個檔案修正前的內容,與本專案目前程式碼在相關行數完全一致)。