Skip to content

Fix 4 video-source / JoyVASA bugs: relative-motion formula, paste-back override, frame indexing, fps - #199

Open
wegylexy wants to merge 4 commits into
warmshao:masterfrom
wegylexy:upstream-fixes
Open

Fix 4 video-source / JoyVASA bugs: relative-motion formula, paste-back override, frame indexing, fps#199
wegylexy wants to merge 4 commits into
warmshao:masterfrom
wegylexy:upstream-fixes

Conversation

@wegylexy

Copy link
Copy Markdown

Summary

Four bugs affecting is_source_video (video-as-source) mode and the JoyVASA driving path, found and fixed while building a downstream CI pipeline on top of this project. Each is an isolated commit so they can be reviewed/merged independently if preferred.

  1. is_source_video relative-motion formula (src/pipelines/faster_live_portrait_pipeline.py) — the exp/lip/eyes region and pose (R_new) branches substituted the driving frame's raw/smoothed value directly, discarding the source's own baseline and never subtracting the driving clip's own frame-0 baseline. This desyncs frame 0 (mouth not guaranteed closed even when source's resting pose is) and undershoots motion amplitude whenever the driving identity's resting expression/pose differs from the source's. Fixed to use the same relative-delta formula as the still-image branch: source's own baseline + (driving's current frame − driving's own frame-0 baseline).
  2. run.py always overriding flag_pastebackrun_with_video/run_with_pkl unconditionally set infer_cfg.infer_params.flag_pasteback = args.paste_back, and --paste_back defaults to False. Any invocation without --paste_back silently disabled paste-back regardless of the --cfg file's own flag_pasteback: True, making the output just the raw unmodified source. Fixed to only override when --paste_back is explicitly passed (default changed to None).
  3. Static src_imgs[0]/src_infos[0] indexing (run.py, api.py) — run_with_video/run_with_pkl always indexed frame 0 regardless of the current driving frame, so a video source's own head motion never advanced past its first frame. Fixed by indexing [frame_ind] when pipe.is_source_video. (Note: run.py's two call sites and api.py's run_with_video were exercised end-to-end with a real source+driving video pair; api.py's run_with_pkl has the identical pattern but wasn't independently run-tested.)
  4. JoyVASA fps mislabeling (src/pipelines/gradio_live_portrait_pipeline.py, run_pickle_driving) — when is_source_video was true, the render was labeled with the source video's fps instead of the driving pickle's own output_fps. These can legitimately differ (JoyVASA generates motion at its own fixed fps regardless of source fps), causing audio/video drift that grows over the clip. Fixed to always use dri_motion_infos["output_fps"].

All four were verified against upstream master before opening this PR (each file's pre-fix content matches this repo's current code exactly at the relevant lines).


摘要(繁體中文)

在既有專案上搭建下游 CI pipeline 時,發現並修復了四個影響「以影片作為來源」(is_source_video)模式與 JoyVASA 驅動路徑的錯誤。每個修正都是獨立的 commit,方便個別審查/合併。

  1. is_source_video 相對運動公式錯誤src/pipelines/faster_live_portrait_pipeline.py)— exp/lip/eyes 區域與 pose(R_new)分支直接套用驅動影格的原始(或平滑後)數值,捨棄了來源自身的基準值,也從未減去驅動影片自己第 0 幀的基準值。這會導致第 0 幀不同步(即使來源的靜止姿勢是閉嘴,也不保證閉嘴),且當驅動身分的靜止表情/姿勢與來源不同時,動作幅度會偏小。已修正為與靜態圖片分支相同的相對差值公式:來源自身基準值 +(驅動目前影格 − 驅動自己第 0 幀基準值)。
  2. run.py 一律覆寫 flag_pastebackrun_with_videorun_with_pkl 無條件執行 infer_cfg.infer_params.flag_pasteback = args.paste_back,而 --paste_back 預設為 False。只要呼叫時未加上 --paste_back,就會無聲地停用貼回效果,即使 --cfg 設定檔本身寫的是 flag_pasteback: True,輸出也只會是原始未經處理的來源影片。已修正為僅在明確傳入 --paste_back 時才覆寫(預設值改為 None)。
  3. src_imgs[0]src_infos[0] 索引固定不變run.pyapi.py)— run_with_videorun_with_pkl 一律讀取第 0 幀,不論目前驅動到第幾幀,導致以影片作為來源時,來源自身的頭部動作永遠停在第一幀。已修正為當 pipe.is_source_video 時改用 [frame_ind] 索引。(註:run.py 的兩處呼叫點與 api.pyrun_with_video 已用實際的來源+驅動影片組合完整測試過;api.pyrun_with_pkl 具有相同的錯誤模式與修法,但尚未獨立實測。)
  4. JoyVASA 影格率標示錯誤src/pipelines/gradio_live_portrait_pipeline.pyrun_pickle_driving)— 當 is_source_video 為真時,輸出影片使用的是「來源影片」的影格率,而非驅動 pickle 自己的 output_fps。這兩者可能本來就不同(JoyVASA 產生動作時使用固定的影格率,與來源影片無關),導致音訊與影片隨著片長逐漸不同步。已修正為一律使用 dri_motion_infos["output_fps"]

以上四項修正在送出這個 PR 前,皆已對照 upstream master 逐一確認(每個檔案修正前的內容,與本專案目前程式碼在相關行數完全一致)。

Timothy Wong added 4 commits August 13, 2026 21:11
…tuting absolute driving value instead of relative delta)

For is_source_video mode, the exp/lip/eyes region and pose region branches
were substituting the driving frame's raw/smoothed value directly
(x_d_exp_smooth, R_d_i), discarding the source's own baseline expression/
rotation and never subtracting the driving clip's own frame-0 baseline.

This desyncs frame 0 (no longer guaranteed to match source's resting pose,
e.g. mouth not fully closed even when source's own rest pose is) and
undershoots motion amplitude whenever the driving identity's resting
expression differs from the source's.

Fixed to use the same relative-delta formula as the still-image branch:
source's own baseline + (driving's current frame - driving's own frame-0
baseline), smoothing after composing rather than substituting the raw
value. Verified against a source video + driving video clip: frame 0 now
matches the source's own resting expression exactly, and mouth-open
amplitude at fast transients is much closer to the still-image branch's
output on the same driving clip.
…ng --cfg

run_with_video/run_with_pkl unconditionally did
infer_cfg.infer_params.flag_pasteback = args.paste_back, and --paste_back
defaults to False (action='store_true'). Any invocation without
--paste_back therefore silently disabled paste-back regardless of the
--cfg file's own flag_pasteback: True setting, making the output just the
raw unmodified source video with no reprojected animation.

Changed --paste_back's default to None and only override flag_pasteback
when it's explicitly passed on the CLI, matching --cfg's flag_pasteback
otherwise (same pattern used for the other CLI overrides in this repo,
e.g. --flag_relative_motion).
…sources

run_with_video/run_with_pkl in both run.py and api.py always indexed
pipe.src_imgs[0]/pipe.src_infos[0] regardless of the current driving frame
index, so when the source is itself a video (is_source_video=True), the
source's own head motion never advanced past frame 0 - every driving
frame was composited against the same static frame-0 source pose.

Fixed by indexing pipe.src_imgs[frame_ind]/pipe.src_infos[frame_ind] when
pipe.is_source_video, falling back to index 0 for still-image sources as
before.

Note: run.py's run_with_video/run_with_pkl and api.py's run_with_video
were exercised end-to-end with an actual source+driving video pair; the
run_with_pkl instance in api.py has the identical bug/fix pattern but
wasn't independently run-tested (api.py's pickle-driving path wasn't
exercised this session).
…output_fps

When is_source_video was true, the rendered output was labeled with the
source video's own fps (utils.get_video_info(self.source_path)) instead
of the driving motion pickle's own output_fps. These can legitimately
differ - e.g. JoyVASA generates motion at a fixed fps regardless of the
source video's native fps - causing the render to run at the wrong
effective frame rate and drift out of sync with the driving audio
increasingly over the clip.

Fixed to always use dri_motion_infos['output_fps'], which every driving
pickle sets correctly at generation time regardless of source type (both
run.py's pickle export and JoyVASA's own gen_motion_sequence set it).

Verified with a source video + JoyVASA-generated driving pickle: audio and
video duration now match to within 0.01s, both when JoyVASA was run
against a still image (the fix is a no-op there, matching prior behavior)
and against a video source (previously off by ~4% given a 24fps source
vs. JoyVASA's 25fps motion).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant