stage2 reproduce's result looks like incomplete denoising

[kxding]

I am trying to learn the process of stage2 using flowmatching to learn the 4 step process from LQ to HQ. However, the results after four steps are not satisfactory, the result looks like incomplete denoising. Currently, I aim to restore 512x512 and 17-frame video. For the deg_latent part, I am using 500 steps noise.

1742263340_1.mp4

[jshilong]

Thank you for your interest in our work！

Did you reproduce the results yourself, or were they obtained using my provided checkpoint?

If you used my checkpoint, please note it does not support 512 resolution with 17-frame inputs, as this was not included in training.

If you trained the model yourself, could you share the training settings for further analysis?

[jshilong]

If you are trying to reproduce the results, I believe 500 noise steps might still be too strong for 512 resolution with 17-frame inputs.

On the other hand, did you load any pre-trained models? If not, it’s better to use a sufficiently large batch size, as the training can easily collapse early on. However, this might not be immediately noticeable from the loss. I have been troubled by this problem for a long time

[kxding]

I reproduce the results myself and use the original cogvideo5b model. Currently, I have tried to use 5 video data for overfitting experiments, but I am still getting videos that seem to be incompletely denoised；Could you please specify the range for "large batch size"? In my case, the loss converged around 800 steps during training and I use that checkpoint for generating.

[jshilong]

For the overfitting experiment, there should be no issues. Is it possible to fix the noise or even remove the latent noise entirely, and then fit a single video? In this case, the model would purely act as a 4x upsampling model, allowing you to validate that both your training and sampling code are functioning correctly.

[kxding]

My current understanding is that flashvideo uses cogvideo2b's transformer to predict vt (HR - LR). However, is there any trick regarding the handling of positional encoding and time embeddings? Because even when I start from the LQ latent without deg_latent and directly use flow matching for prediction, the results are still blurry.

1742365291_1.mp4

[jshilong]

1. If you have loaded the pretrained model of CogVideoX, please ensure that 1000 steps correspond to ( t = 0 ) in the flow matching(according to your implementation).

2. You may first try fitting a single image using a relatively large batch size. Instabilities during the early stages of training are indeed common when changing model behavior to a new task, so it's advisable to avoid small batch sizes at the start of training.

[kxding]

Yes, in my implementation, Z0 represents lr and Z1 represents hr, and I use z + dzdt in flow matching.But I'm still confused about how to handle the original CogVideoX's 1000 steps of pure noise corresponding to the t = 0 case. What specific modifications would be needed in the code? Would there be modifications to the weight correspondence when loading specific weights？

[kxding]

By the way when infering my timestep start from 0 to 1 that is lr to hr

[jshilong]

Yes, in my implementation, Z0 represents lr and Z1 represents hr, and I use z + dzdt in flow matching.But I'm still confused about how to handle the original CogVideoX's 1000 steps of pure noise corresponding to the t = 0 case. What specific modifications would be needed in the code? Would there be modifications to the weight correspondence when loading specific weights？

Try the following mapping:
step_idx = 1000 - (t * 1000)
This maps the t in your task to the corresponding step in CogVideoX. Note that in the original CogVideoX, when step_idx = 0, it represents the clear video.

[thbupt]

@jshilong I'm also trying to use the process of stage2 for a new task. I have several questions.

1. In your experience, what’s the least amount of data required when transform the raw cogvideo to a new flow-matching task?
2. How about the range for "large batch size"? Maybe the large batch size is required for a full-parameter finetuning task?

[jshilong]

@thbupt Thanks for your interest in our work

In my experience, when transforming raw CogVideo to a new flow-matching task, it’s best to start with small-resolution images (e.g., 540p) to support larger batch sizes early on. The batch size should ideally exceed 256, with diverse t to ensure stable training across intervals of the field.

Reasonable visualizations should appear within a few thousand iterations. Once image outputs look good, you can transition to video data with adjusted resolution and parameters.

If you start with a flow-matching pretrained foundation model, the required data can be significantly reduced. Transformation from noise-to-image to image-to-image, or video-to-video much easier and more efficient.