torch.cuda.OutOfMemoryError: CUDA out of memory.

[sankexin]

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1226.19 GiB. GPU has a total capacity of 63.98 GiB of which 36.29 GiB is free. Of the allocated memory 24.03 GiB is allocated by PyTorch, and 987.10 MiB is reserved by PyTorch but unallocated.

how to reduce memory for infer?

[jshilong]

Could you please provide the full error log for better troubleshooting?

There is also a suggestion in this issur that might help: Reduce the Decoder Memory Usage.

[sankexin]

Could you please provide the full error log for better troubleshooting?

There is also a suggestion in this issur that might help: Reduce the Decoder Memory Usage.

overwrite cfg scale in config of stage-1
##############################  Sampling setting  ##############################
Sampler: VPSDEDPMPP2MSampler
Discretization: ZeroSNRDDPMDiscretization
Guider: DynamicCFG
Sampling with VPSDEDPMPP2MSampler for 52 steps:   0%|                                                                                                                                                                              | 0/52 [00:00<?, ?it/s]/usr/local/lib/python3.10/site-packages/sat/transformer_defaults.py:67: UserWarning: 1Torch was not compiled with flash attention. (Triggered internally at /data/jenkins_workspace/workspace/pytorch@11/aten/src/ATen/native/transformers/hip/sdp_utils.cpp:459.)
  attn_output = torch.nn.functional.scaled_dot_product_attention(
/usr/local/lib/python3.10/site-packages/sat/transformer_defaults.py:67: UserWarning: 1Torch was not compiled with memory efficient attention. (Triggered internally at /data/jenkins_workspace/workspace/pytorch@11/aten/src/ATen/native/transformers/hip/sdp_utils.cpp:508.)
  attn_output = torch.nn.functional.scaled_dot_product_attention(
merge lora of 
merge lora of model.diffusion_model.mixins.lora
rank and world_size 2 4
load from ./example.txt with 25
./vis_270p_1080p_example
rank 2 processing 2
{'prompt': "A beautiful princess with elegantly styled hair and a flowing gown rides gracefully on a majestic horse as she crosses a gently flowing river. The horse moves steadily through the water, creating ripples that shimmer under the soft sunlight. The princess holds the reins with a gentle, yet confident grip, her expression serene and focused. The camera starts with a wide view of the river and approaches the princess as she nears the riverbank, capturing the water's movement and her poised demeanor.", 'negative_prompt': 'very low motion,, low motion,', 'num_frames': tensor([13])}
txt [500]
overwrite cfg scale in config of stage-1
##############################  Sampling setting  ##############################
Sampler: VPSDEDPMPP2MSampler
Discretization: ZeroSNRDDPMDiscretization
Guider: DynamicCFG
Sampling with VPSDEDPMPP2MSampler for 52 steps:   4%|██████▍                                                                                                                                                               | 2/52 [00:14<05:52,  7.06s/it]/usr/local/lib/python3.10/site-packages/sat/transformer_defaults.py:67: UserWarning: 1Torch was not compiled with flash attention. (Triggered internally at /data/jenkins_workspace/workspace/pytorch@11/aten/src/ATen/native/transformers/hip/sdp_utils.cpp:459.)
  attn_output = torch.nn.functional.scaled_dot_product_attention(
/usr/local/lib/python3.10/site-packages/sat/transformer_defaults.py:67: UserWarning: 1Torch was not compiled with memory efficient attention. (Triggered internally at /data/jenkins_workspace/workspace/pytorch@11/aten/src/ATen/native/transformers/hip/sdp_utils.cpp:508.)
  attn_output = torch.nn.functional.scaled_dot_product_attention(
Sampling with VPSDEDPMPP2MSampler for 52 steps:  98%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊   | 51/52 [05:37<00:06,  6.62s/it]
Sampling with VPSDEDPMPP2MSampler for 52 steps:  98%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊   | 51/52 [05:37<00:06,  6.62s/it]
Sampling with VPSDEDPMPP2MSampler for 52 steps:  96%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▋      | 50/52 [05:31<00:13,  6.61s/it]I0217 09:29:31.079838  1406 ProcessGroupNCCL.cpp:1991] [PG 2 Rank 0] ProcessGroupNCCL created ncclComm_ 0x55645e061b00 on CUDA device:
I0217 09:29:31.079867  1406 ProcessGroupNCCL.cpp:1996] [PG 2 Rank 0] NCCL_DEBUG: N/A
I0217 09:29:32.126924  1409 ProcessGroupNCCL.cpp:1991] [PG 2 Rank 0] ProcessGroupNCCL created ncclComm_ 0x55a923951180 on CUDA device: 
I0217 09:29:32.126961  1409 ProcessGroupNCCL.cpp:1996] [PG 2 Rank 0] NCCL_DEBUG: N/A
Sampling with VPSDEDPMPP2MSampler for 52 steps:  98%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊   | 51/52 [05:37<00:06,  6.62s/it]
save to ./vis_270p_1080p_example/0_0_seed_0.mp4
Sampling with VPSDEDPMPP2MSampler for 52 steps:  98%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊   | 51/52 [05:37<00:06,  6.62s/it]
start encoding first stage results to high resolution
save to ./vis_270p_1080p_example/3_0_seed_3.mp4
I0217 09:29:40.379447  1407 ProcessGroupNCCL.cpp:1991] [PG 2 Rank 0] ProcessGroupNCCL created ncclComm_ 0x5591baabbc00 on CUDA device: 
I0217 09:29:40.379491  1407 ProcessGroupNCCL.cpp:1996] [PG 2 Rank 0] NCCL_DEBUG: N/A
start encoding first stage results to high resolution
I0217 09:29:46.256350  1408 ProcessGroupNCCL.cpp:1991] [PG 2 Rank 0] ProcessGroupNCCL created ncclComm_ 0x55facffe4500 on CUDA device: 
I0217 09:29:46.256392  1408 ProcessGroupNCCL.cpp:1996] [PG 2 Rank 0] NCCL_DEBUG: N/A
save to ./vis_270p_1080p_example/1_0_seed_1.mp4
start encoding first stage results to high resolution
save to ./vis_270p_1080p_example/2_0_seed_2.mp4
start encoding first stage results to high resolution
finish encoding first stage results, and starting stage II
{'prompt': 'medium motion,The video begins with a view of a cozy log cabin nestled among lush trees in a dense forest. A gentle plume of white smoke rises gracefully from the chimney, curling against the backdrop of towering pines. The windows emit a warm, soft glow, hinting at a comforting and inviting atmosphere within. The camera slowly pans from right to left, capturing the serene setting and emphasizing the harmony between the cabin and its natural surroundings.', 'negative_prompt': '', 'num_frames': tensor([49])}
txt [459]
sample_kwargs {'batch_size': 1, 'shape': (13, 16, 134, 240), 'num_steps': 5, 'method': 'euler', 'cfg': 13}
2.5
sample: tensor([0.0000, 0.0938, 0.2148, 0.3750, 0.6172, 1.0000], device='cuda:0',
       dtype=torch.bfloat16)
ref_noise_step : 675
weight tensor([0.2640], device='cuda:0')
sample_ref_noise_step 675
tensor([676., 676.], device='cuda:0', dtype=torch.bfloat16) get -2.045283317565918
[rank0]: Traceback (most recent call last):
[rank0]:   File "/home/FlashVideo/flashvideo/dist_inf_text_file.py", line 380, in <module>
[rank0]:     sampling_main(args,
[rank0]:   File "/home/FlashVideo/flashvideo/dist_inf_text_file.py", line 321, in sampling_main
[rank0]:     second_stage_samples = get_second_results(
[rank0]:   File "/home/FlashVideo/flashvideo/dist_inf_text_file.py", line 204, in get_second_results
[rank0]:     samples_z = model.sample(
[rank0]:   File "/usr/local/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
[rank0]:     return func(*args, **kwargs)
[rank0]:   File "/home/FlashVideo/flashvideo/flow_video.py", line 252, in sample
[rank0]:     results = odeint(single_function_evaluation,
[rank0]:   File "/usr/local/lib/python3.10/site-packages/torchdiffeq/_impl/odeint.py", line 80, in odeint
[rank0]:     solution = solver.integrate(t)
[rank0]:   File "/usr/local/lib/python3.10/site-packages/torchdiffeq/_impl/solvers.py", line 114, in integrate
[rank0]:     dy, f0 = self._step_func(self.func, t0, dt, t1, y0)
[rank0]:   File "/usr/local/lib/python3.10/site-packages/torchdiffeq/_impl/fixed_grid.py", line 10, in _step_func
[rank0]:     f0 = func(t0, y0, perturb=Perturb.NEXT if self.perturb else Perturb.NONE)
[rank0]:   File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.10/site-packages/torchdiffeq/_impl/misc.py", line 197, in forward
[rank0]:     return self.base_func(t, y)
[rank0]:   File "/home/FlashVideo/flashvideo/flow_video.py", line 191, in single_function_evaluation
[rank0]:     vt = self.model(x, t=idx, c=real_cond, idx=idx)
[rank0]:   File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:   File "/home/FlashVideo/flashvideo/sgm/modules/diffusionmodules/wrappers.py", line 43, in forward
[rank0]:     return self.diffusion_model(
[rank0]:   File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:   File "/home/FlashVideo/flashvideo/extra_models/dit_res_adapter.py", line 360, in forward
[rank0]:     output = super(DiffusionTransformer, self).forward(**kwargs)[0]
[rank0]:   File "/home/FlashVideo/flashvideo/base_model.py", line 110, in forward
[rank0]:     return self.transformer(*args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:   File "/home/FlashVideo/flashvideo/base_transformer.py", line 363, in forward
[rank0]:     layer_ret = self.hooks['layer_forward'](
[rank0]:   File "/home/FlashVideo/flashvideo/dit_video_concat.py", line 621, in layer_forward
[rank0]:     attention_output = layer.attention(attention_input, mask, **kwargs)
[rank0]:   File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.10/site-packages/sat/model/transformer.py", line 113, in forward
[rank0]:     return HOOKS_DEFAULT['attention_forward'](self, hidden_states, mask, **kw_args)
[rank0]:   File "/usr/local/lib/python3.10/site-packages/sat/transformer_defaults.py", line 105, in attention_forward_default
[rank0]:     context_layer = attention_fn(query_layer, key_layer, value_layer, mask, dropout_fn, **kw_args)
[rank0]:   File "/home/FlashVideo/flashvideo/dit_video_concat.py", line 685, in attention_fn
[rank0]:     return old_impl(
[rank0]:   File "/home/FlashVideo/flashvideo/extra_models/dit_res_adapter.py", line 150, in attention_fn
[rank0]:     return attention_fn_default(
[rank0]:   File "/usr/local/lib/python3.10/site-packages/sat/transformer_defaults.py", line 67, in attention_fn_default
[rank0]:     attn_output = torch.nn.functional.scaled_dot_product_attention(
[rank0]: torch.cuda.OutOfMemoryError: HIP out of memory. Tried to allocate 1226.19 GiB. GPU
I0217 09:30:38.598588  1406 ProcessGroupNCCL.cpp:1126] [PG 0 Rank 0] ProcessGroupNCCL destructor entered.
I0217 09:30:38.598630  1406 ProcessGroupNCCL.cpp:1111] [PG 0 Rank 0] Launching ProcessGroupNCCL abort asynchrounously.
I0217 09:30:38.598753  1406 ProcessGroupNCCL.cpp:1016] [PG 0 Rank 0] future is successfully executed for: ProcessGroup abort
I0217 09:30:38.598763  1406 ProcessGroupNCCL.cpp:1117] [PG 0 Rank 0] ProcessGroupNCCL aborts successfully.
I0217 09:30:38.598778  1406 ProcessGroupNCCL.cpp:1149] [PG 0 Rank 0] ProcessGroupNCCL watchdog thread joined.
I0217 09:30:38.598793  1406 ProcessGroupNCCL.cpp:1153] [PG 0 Rank 0] ProcessGroupNCCL heart beat monitor thread joined.
I0217 09:30:38.991128  1406 ProcessGroupNCCL.cpp:1126] [PG 1 Rank 0] ProcessGroupNCCL destructor entered.
I0217 09:30:38.991170  1406 ProcessGroupNCCL.cpp:1111] [PG 1 Rank 0] Launching ProcessGroupNCCL abort asynchrounously.
I0217 09:30:38.991314  1406 ProcessGroupNCCL.cpp:1016] [PG 1 Rank 0] future is successfully executed for: ProcessGroup abort
I0217 09:30:38.991323  1406 ProcessGroupNCCL.cpp:1117] [PG 1 Rank 0] ProcessGroupNCCL aborts successfully.
I0217 09:30:38.991338  1406 ProcessGroupNCCL.cpp:1149] [PG 1 Rank 0] ProcessGroupNCCL watchdog thread joined.
I0217 09:30:38.991357  1406 ProcessGroupNCCL.cpp:1153] [PG 1 Rank 0] ProcessGroupNCCL heart beat monitor thread joined.
I0217 09:30:38.991371  1406 ProcessGroupNCCL.cpp:1126] [PG 3 Rank 0] ProcessGroupNCCL destructor entered.
I0217 09:30:38.991379  1406 ProcessGroupNCCL.cpp:1111] [PG 3 Rank 0] Launching ProcessGroupNCCL abort asynchrounously.
I0217 09:30:38.991436  1406 ProcessGroupNCCL.cpp:1016] [PG 3 Rank 0] future is successfully executed for: ProcessGroup abort
I0217 09:30:38.991444  1406 ProcessGroupNCCL.cpp:1117] [PG 3 Rank 0] ProcessGroupNCCL aborts successfully.
I0217 09:30:38.991456  1406 ProcessGroupNCCL.cpp:1149] [PG 3 Rank 0] ProcessGroupNCCL watchdog thread joined.
I0217 09:30:38.991465  1406 ProcessGroupNCCL.cpp:1153] [PG 3 Rank 0] ProcessGroupNCCL heart beat monitor thread joined.
finish encoding first stage results, and starting stage II
{'prompt': "medium motion,In a cozy cafe bathed in warm, ambient lighting, a fluffy llama with a cream-colored coat is seated at a wooden table. Wearing a pair of stylish round glasses perched on its snout, the llama intently focuses on the laptop screen, which glows with a soft blue light. Its hooves move delicately across the keyboard, typing codes with surprising dexterity. The camera gently sweeps around the table, capturing the concentration visible in the llama's large, expressive eyes, and revealing the inviting atmosphere of the cafe, complete with rustic decor and other patrons in the background.", 'negative_prompt': '', 'num_frames': tensor([49])}
txt [600]
sample_kwargs {'batch_size': 1, 'shape': (13, 16, 134, 240), 'num_steps': 5, 'method': 'euler', 'cfg': 13}
2.5
sample: tensor([0.0000, 0.0938, 0.2148, 0.3750, 0.6172, 1.0000], device='cuda:3',
       dtype=torch.bfloat16)
ref_noise_step : 675
weight tensor([0.2640], device='cuda:3')
sample_ref_noise_step 675
tensor([676., 676.], device='cuda:3', dtype=torch.bfloat16) get -2.045283317565918
[rank3]: Traceback (most recent call last):
[rank3]:   File "/home/FlashVideo/flashvideo/dist_inf_text_file.py", line 380, in <module>
[rank3]:     sampling_main(args,
[rank3]:   File "/home/FlashVideo/flashvideo/dist_inf_text_file.py", line 321, in sampling_main
[rank3]:     second_stage_samples = get_second_results(
[rank3]:   File "/home/FlashVideo/flashvideo/dist_inf_text_file.py", line 204, in get_second_results
[rank3]:     samples_z = model.sample(
[rank3]:   File "/usr/local/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
[rank3]:     return func(*args, **kwargs)
[rank3]:   File "/home/FlashVideo/flashvideo/flow_video.py", line 252, in sample
[rank3]:     results = odeint(single_function_evaluation,
[rank3]:   File "/usr/local/lib/python3.10/site-packages/torchdiffeq/_impl/odeint.py", line 80, in odeint
[rank3]:     solution = solver.integrate(t)
[rank3]:   File "/usr/local/lib/python3.10/site-packages/torchdiffeq/_impl/solvers.py", line 114, in integrate
[rank3]:     dy, f0 = self._step_func(self.func, t0, dt, t1, y0)
[rank3]:   File "/usr/local/lib/python3.10/site-packages/torchdiffeq/_impl/fixed_grid.py", line 10, in _step_func
[rank3]:     f0 = func(t0, y0, perturb=Perturb.NEXT if self.perturb else Perturb.NONE)
[rank3]:   File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[rank3]:     return self._call_impl(*args, **kwargs)
[rank3]:   File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[rank3]:     return forward_call(*args, **kwargs)
[rank3]:   File "/usr/local/lib/python3.10/site-packages/torchdiffeq/_impl/misc.py", line 197, in forward
[rank3]:     return self.base_func(t, y)
[rank3]:   File "/home/FlashVideo/flashvideo/flow_video.py", line 191, in single_function_evaluation
[rank3]:     vt = self.model(x, t=idx, c=real_cond, idx=idx)
[rank3]:   File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[rank3]:     return self._call_impl(*args, **kwargs)
[rank3]:   File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[rank3]:     return forward_call(*args, **kwargs)
[rank3]:   File "/home/FlashVideo/flashvideo/sgm/modules/diffusionmodules/wrappers.py", line 43, in forward
[rank3]:     return self.diffusion_model(
[rank3]:   File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[rank3]:     return self._call_impl(*args, **kwargs)
[rank3]:   File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[rank3]:     return forward_call(*args, **kwargs)
[rank3]:   File "/home/FlashVideo/flashvideo/extra_models/dit_res_adapter.py", line 360, in forward
[rank3]:     output = super(DiffusionTransformer, self).forward(**kwargs)[0]
[rank3]:   File "/home/FlashVideo/flashvideo/base_model.py", line 110, in forward
[rank3]:     return self.transformer(*args, **kwargs)
[rank3]:   File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[rank3]:     return self._call_impl(*args, **kwargs)
[rank3]:   File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[rank3]:     return forward_call(*args, **kwargs)
[rank3]:   File "/home/FlashVideo/flashvideo/base_transformer.py", line 363, in forward
[rank3]:     layer_ret = self.hooks['layer_forward'](
[rank3]:   File "/home/FlashVideo/flashvideo/dit_video_concat.py", line 621, in layer_forward
[rank3]:     attention_output = layer.attention(attention_input, mask, **kwargs)
[rank3]:   File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[rank3]:     return self._call_impl(*args, **kwargs)
[rank3]:   File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[rank3]:     return forward_call(*args, **kwargs)
[rank3]:   File "/usr/local/lib/python3.10/site-packages/sat/model/transformer.py", line 113, in forward
[rank3]:     return HOOKS_DEFAULT['attention_forward'](self, hidden_states, mask, **kw_args)
[rank3]:   File "/usr/local/lib/python3.10/site-packages/sat/transformer_defaults.py", line 105, in attention_forward_default
[rank3]:     context_layer = attention_fn(query_layer, key_layer, value_layer, mask, dropout_fn, **kw_args)
[rank3]:   File "/home/FlashVideo/flashvideo/dit_video_concat.py", line 685, in attention_fn
[rank3]:     return old_impl(
[rank3]:   File "/home/FlashVideo/flashvideo/extra_models/dit_res_adapter.py", line 150, in attention_fn
[rank3]:     return attention_fn_default(
[rank3]:   File "/usr/local/lib/python3.10/site-packages/sat/transformer_defaults.py", line 67, in attention_fn_default
[rank3]:     attn_output = torch.nn.functional.scaled_dot_product_attention(
[rank3]: torch.cuda.OutOfMemoryError: HIP out of memory. Tried to allocate 1226.19 GiB. GPU  has a total capacity of 63.98 GiB of which 36.29 GiB is free. Of the allocated memory 24.03 GiB is allocated by PyTorch, and 987.10 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_HIP_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
I0217 09:30:41.704511  1409 ProcessGroupNCCL.cpp:1126] [PG 0 Rank 3] ProcessGroupNCCL destructor entered.
I0217 09:30:41.704553  1409 ProcessGroupNCCL.cpp:1111] [PG 0 Rank 3] Launching ProcessGroupNCCL abort asynchrounously.
I0217 09:30:41.704684  1409 ProcessGroupNCCL.cpp:1016] [PG 0 Rank 3] future is successfully executed for: ProcessGroup abort
I0217 09:30:41.704694  1409 ProcessGroupNCCL.cpp:1117] [PG 0 Rank 3] ProcessGroupNCCL aborts successfully.
I0217 09:30:41.704708  1409 ProcessGroupNCCL.cpp:1149] [PG 0 Rank 3] ProcessGroupNCCL watchdog thread joined.
I0217 09:30:41.704731  1409 ProcessGroupNCCL.cpp:1153] [PG 0 Rank 3] ProcessGroupNCCL heart beat monitor thread joined.
I0217 09:30:42.091583  1409 ProcessGroupNCCL.cpp:1126] [PG 1 Rank 3] ProcessGroupNCCL destructor entered.
I0217 09:30:42.091617  1409 ProcessGroupNCCL.cpp:1111] [PG 1 Rank 3] Launching ProcessGroupNCCL abort asynchrounously.
I0217 09:30:42.091708  1409 ProcessGroupNCCL.cpp:1016] [PG 1 Rank 3] future is successfully executed for: ProcessGroup abort
I0217 09:30:42.091717  1409 ProcessGroupNCCL.cpp:1117] [PG 1 Rank 3] ProcessGroupNCCL aborts successfully.
I0217 09:30:42.091732  1409 ProcessGroupNCCL.cpp:1149] [PG 1 Rank 3] ProcessGroupNCCL watchdog thread joined.
I0217 09:30:42.091753  1409 ProcessGroupNCCL.cpp:1153] [PG 1 Rank 3] ProcessGroupNCCL heart beat monitor thread joined.
I0217 09:30:42.091773  1409 ProcessGroupNCCL.cpp:1126] [PG 3 Rank 3] ProcessGroupNCCL destructor entered.
I0217 09:30:42.091781  1409 ProcessGroupNCCL.cpp:1111] [PG 3 Rank 3] Launching ProcessGroupNCCL abort asynchrounously.
I0217 09:30:42.091897  1409 ProcessGroupNCCL.cpp:1016] [PG 3 Rank 3] future is successfully executed for: ProcessGroup abort
I0217 09:30:42.091907  1409 ProcessGroupNCCL.cpp:1117] [PG 3 Rank 3] ProcessGroupNCCL aborts successfully.
I0217 09:30:42.091918  1409 ProcessGroupNCCL.cpp:1149] [PG 3 Rank 3] ProcessGroupNCCL watchdog thread joined.
I0217 09:30:42.091933  1409 ProcessGroupNCCL.cpp:1153] [PG 3 Rank 3] ProcessGroupNCCL heart beat monitor thread joined.
finish encoding first stage results, and starting stage II
{'prompt': 'medium motion,A stylish woman strides confidently down a bustling Tokyo street, which is illuminated by warm glowing neon and animated city signage. She exudes elegance in a black leather jacket, a long red dress flowing gracefully as she moves, and chic black boots, while a black purse swings at her side. Her striking red lipstick complements her sleek sunglasses, enhancing her sophisticated look. Her casual yet assured walk creates an air of confidence. The damp street beneath her feet glistens, reflecting the vibrant array of colorful lights above. The camera follows her fluid motion, capturing the mirror-like effect on the ground and the dynamic hustle around her with pedestrians weaving through the luminous scene.', 'negative_prompt': '', 'num_frames': tensor([49])}
txt [728]
sample_kwargs {'batch_size': 1, 'shape': (13, 16, 134, 240), 'num_steps': 5, 'method': 'euler', 'cfg': 13}
2.5
sample: tensor([0.0000, 0.0938, 0.2148, 0.3750, 0.6172, 1.0000], device='cuda:1',
       dtype=torch.bfloat16)
ref_noise_step : 675
weight tensor([0.2640], device='cuda:1')
sample_ref_noise_step 675
tensor([676., 676.], device='cuda:1', dtype=torch.bfloat16) get -2.045283317565918
[rank1]: Traceback (most recent call last):
[rank1]:   File "/home/FlashVideo/flashvideo/dist_inf_text_file.py", line 380, in <module>
[rank1]:     sampling_main(args,
[rank1]:   File "/home/FlashVideo/flashvideo/dist_inf_text_file.py", line 321, in sampling_main
[rank1]:     second_stage_samples = get_second_results(
[rank1]:   File "/home/FlashVideo/flashvideo/dist_inf_text_file.py", line 204, in get_second_results
[rank1]:     samples_z = model.sample(
[rank1]:   File "/usr/local/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
[rank1]:     return func(*args, **kwargs)
[rank1]:   File "/home/FlashVideo/flashvideo/flow_video.py", line 252, in sample
[rank1]:     results = odeint(single_function_evaluation,
[rank1]:   File "/usr/local/lib/python3.10/site-packages/torchdiffeq/_impl/odeint.py", line 80, in odeint
[rank1]:     solution = solver.integrate(t)
[rank1]:   File "/usr/local/lib/python3.10/site-packages/torchdiffeq/_impl/solvers.py", line 114, in integrate
[rank1]:     dy, f0 = self._step_func(self.func, t0, dt, t1, y0)
[rank1]:   File "/usr/local/lib/python3.10/site-packages/torchdiffeq/_impl/fixed_grid.py", line 10, in _step_func
[rank1]:     f0 = func(t0, y0, perturb=Perturb.NEXT if self.perturb else Perturb.NONE)
[rank1]:   File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[rank1]:     return self._call_impl(*args, **kwargs)
[rank1]:   File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[rank1]:     return forward_call(*args, **kwargs)
[rank1]:   File "/usr/local/lib/python3.10/site-packages/torchdiffeq/_impl/misc.py", line 197, in forward
[rank1]:     return self.base_func(t, y)
[rank1]:   File "/home/FlashVideo/flashvideo/flow_video.py", line 191, in single_function_evaluation
[rank1]:     vt = self.model(x, t=idx, c=real_cond, idx=idx)
[rank1]:   File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[rank1]:     return self._call_impl(*args, **kwargs)
[rank1]:   File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[rank1]:     return forward_call(*args, **kwargs)
[rank1]:   File "/home/FlashVideo/flashvideo/sgm/modules/diffusionmodules/wrappers.py", line 43, in forward
[rank1]:     return self.diffusion_model(
[rank1]:   File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[rank1]:     return self._call_impl(*args, **kwargs)
[rank1]:   File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[rank1]:     return forward_call(*args, **kwargs)
[rank1]:   File "/home/FlashVideo/flashvideo/extra_models/dit_res_adapter.py", line 360, in forward
[rank1]:     output = super(DiffusionTransformer, self).forward(**kwargs)[0]
[rank1]:   File "/home/FlashVideo/flashvideo/base_model.py", line 110, in forward
[rank1]:     return self.transformer(*args, **kwargs)
[rank1]:   File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[rank1]:     return self._call_impl(*args, **kwargs)
[rank1]:   File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[rank1]:     return forward_call(*args, **kwargs)
[rank1]:   File "/home/FlashVideo/flashvideo/base_transformer.py", line 363, in forward
[rank1]:     layer_ret = self.hooks['layer_forward'](
[rank1]:   File "/home/FlashVideo/flashvideo/dit_video_concat.py", line 621, in layer_forward
[rank1]:     attention_output = layer.attention(attention_input, mask, **kwargs)
[rank1]:   File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[rank1]:     return self._call_impl(*args, **kwargs)
[rank1]:   File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[rank1]:     return forward_call(*args, **kwargs)
[rank1]:   File "/usr/local/lib/python3.10/site-packages/sat/model/transformer.py", line 113, in forward
[rank1]:     return HOOKS_DEFAULT['attention_forward'](self, hidden_states, mask, **kw_args)
[rank1]:   File "/usr/local/lib/python3.10/site-packages/sat/transformer_defaults.py", line 105, in attention_forward_default
[rank1]:     context_layer = attention_fn(query_layer, key_layer, value_layer, mask, dropout_fn, **kw_args)
[rank1]:   File "/home/FlashVideo/flashvideo/dit_video_concat.py", line 685, in attention_fn
[rank1]:     return old_impl(
[rank1]:   File "/home/FlashVideo/flashvideo/extra_models/dit_res_adapter.py", line 150, in attention_fn
[rank1]:     return attention_fn_default(
[rank1]:   File "/usr/local/lib/python3.10/site-packages/sat/transformer_defaults.py", line 67, in attention_fn_default
[rank1]:     attn_output = torch.nn.functional.scaled_dot_product_attention(
[rank1]: torch.cuda.OutOfMemoryError: HIP out of memory. Tried to allocate 1226.19 GiB. GPU  has a total capacity of 63.98 GiB of which 36.29 GiB is free. Of the allocated memory 24.03 GiB is allocated by PyTorch, and 987.10 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_HIP_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
I0217 09:30:48.759588  1407 ProcessGroupNCCL.cpp:1126] [PG 0 Rank 1] ProcessGroupNCCL destructor entered.
I0217 09:30:48.759630  1407 ProcessGroupNCCL.cpp:1111] [PG 0 Rank 1] Launching ProcessGroupNCCL abort asynchrounously.
I0217 09:30:48.759766  1407 ProcessGroupNCCL.cpp:1016] [PG 0 Rank 1] future is successfully executed for: ProcessGroup abort
I0217 09:30:48.759776  1407 ProcessGroupNCCL.cpp:1117] [PG 0 Rank 1] ProcessGroupNCCL aborts successfully.
I0217 09:30:48.759789  1407 ProcessGroupNCCL.cpp:1149] [PG 0 Rank 1] ProcessGroupNCCL watchdog thread joined.
I0217 09:30:48.759805  1407 ProcessGroupNCCL.cpp:1153] [PG 0 Rank 1] ProcessGroupNCCL heart beat monitor thread joined.
I0217 09:30:49.151173  1407 ProcessGroupNCCL.cpp:1126] [PG 1 Rank 1] ProcessGroupNCCL destructor entered.
I0217 09:30:49.151216  1407 ProcessGroupNCCL.cpp:1111] [PG 1 Rank 1] Launching ProcessGroupNCCL abort asynchrounously.
I0217 09:30:49.151352  1407 ProcessGroupNCCL.cpp:1016] [PG 1 Rank 1] future is successfully executed for: ProcessGroup abort
I0217 09:30:49.151362  1407 ProcessGroupNCCL.cpp:1117] [PG 1 Rank 1] ProcessGroupNCCL aborts successfully.
I0217 09:30:49.151372  1407 ProcessGroupNCCL.cpp:1149] [PG 1 Rank 1] ProcessGroupNCCL watchdog thread joined.
I0217 09:30:49.151402  1407 ProcessGroupNCCL.cpp:1153] [PG 1 Rank 1] ProcessGroupNCCL heart beat monitor thread joined.
I0217 09:30:49.151417  1407 ProcessGroupNCCL.cpp:1126] [PG 3 Rank 1] ProcessGroupNCCL destructor entered.
I0217 09:30:49.151423  1407 ProcessGroupNCCL.cpp:1111] [PG 3 Rank 1] Launching ProcessGroupNCCL abort asynchrounously.
I0217 09:30:49.151489  1407 ProcessGroupNCCL.cpp:1016] [PG 3 Rank 1] future is successfully executed for: ProcessGroup abort
I0217 09:30:49.151496  1407 ProcessGroupNCCL.cpp:1117] [PG 3 Rank 1] ProcessGroupNCCL aborts successfully.
I0217 09:30:49.151505  1407 ProcessGroupNCCL.cpp:1149] [PG 3 Rank 1] ProcessGroupNCCL watchdog thread joined.
I0217 09:30:49.151525  1407 ProcessGroupNCCL.cpp:1153] [PG 3 Rank 1] ProcessGroupNCCL heart beat monitor thread joined.
W0217 09:30:49.909000 23141288793920 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1407 closing signal SIGTERM
W0217 09:30:49.910000 23141288793920 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1408 closing signal SIGTERM
W0217 09:30:49.910000 23141288793920 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 1409 closing signal SIGTERM
E0217 09:30:51.589000 23141288793920 torch/distributed/elastic/multiprocessing/api.py:826] failed (exitcode: 1) local_rank: 0 (pid: 1406) of binary: /usr/local/bin/python3.10
Traceback (most recent call last):
  File "/usr/local/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/torch/distributed/run.py", line 879, in main
    run(args)
  File "/usr/local/lib/python3.10/site-packages/torch/distributed/run.py", line 870, in run
    elastic_launch(
  File "/usr/local/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/usr/local/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 263, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
flashvideo/dist_inf_text_file.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2025-02-17_09:30:49
  host      : 176ea06467d5
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 1406)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html

infer on 4*64G:

torchrun --nproc_per_node=4  \
    --nnodes=1 \
    --node_rank=0 \
    --master_port=20023 flashvideo/dist_inf_text_file.py \
    --base "flashvideo/configs/stage1.yaml" \
    --second "flashvideo/configs/stage2.yaml" \
    --inf-ckpt  ./checkpoints/stage1.pt \
    --inf-ckpt2 ./checkpoints/stage2.pt \
    --input-file ./example.txt \
    --output-dir ./vis_270p_1080p_example

I have allso tested on deepspeed zero3 in your .yaml, but "CUDA out of memory"too!

zero_optimization:
    stage: 3
    cpu_offload: true

[jshilong]

I'm afraid that GPU momery inability to handle a 1080P sequence of Stage-2. As a possible workaround, you could try reducing the resolution in the second stage, altering the stage_2_hw = (1080, 1920) definition to a smaller resolution - yet remember to maintain the aspect ratio. However, please note that as this deviates from our original training resolution, it might lead to losses in quality.

[sankexin]

I'm afraid that GPU momery inability to handle a 1080P sequence of Stage-2. As a possible workaround, you could try reducing the resolution in the second stage, altering the stage_2_hw = (1080, 1920) definition to a smaller resolution - yet remember to maintain the aspect ratio. However, please note that as this deviates from our original training resolution, it might lead to losses in quality.

could you suport a modify way of stage_2_hw in *.yaml?, and any plan to reduce memery in future?

[jshilong]

I'm afraid that GPU momery inability to handle a 1080P sequence of Stage-2. As a possible workaround, you could try reducing the resolution in the second stage, altering the stage_2_hw = (1080, 1920) definition to a smaller resolution - yet remember to maintain the aspect ratio. However, please note that as this deviates from our original training resolution, it might lead to losses in quality.

could you suport a modify way of stage_2_hw in *.yaml?, and any plan to reduce memery in future?

all the args can be modified in the [demo.ipynb](https://github.com/FoundationVision/FlashVideo/blob/main/flashvideo/demo.ipynb) or

FlashVideo/flashvideo/dist_inf_text_file.py

Line 262 in a2e8a67

second_img_size = [1080, 1920]

In order to optimize the memory usage of this DiT model, it may be necessary to split the attention heads. As this is a significant change, I'll need to conduct some tests to assess the impact on computational time.

[sankexin]

I'm afraid that GPU momery inability to handle a 1080P sequence of Stage-2. As a possible workaround, you could try reducing the resolution in the second stage, altering the stage_2_hw = (1080, 1920) definition to a smaller resolution - yet remember to maintain the aspect ratio. However, please note that as this deviates from our original training resolution, it might lead to losses in quality.

could you suport a modify way of stage_2_hw in *.yaml?, and any plan to reduce memery in future?

all the args can be modified in the [demo.ipynb](https://github.com/FoundationVision/FlashVideo/blob/main/flashvideo/demo.ipynb) or

FlashVideo/flashvideo/dist_inf_text_file.py

Line 262 in a2e8a67

second_img_size = [1080, 1920]
In order to optimize the memory usage of this DiT model, it may be necessary to split the attention heads. As this is a significant change, I'll need to conduct some tests to assess the impact on computational time.

1、are the results you tested like this on your equipmens? The image has some distortion and abnormal pixel defects, when second_img_size = [270, 480] .

2、I have tested on 4 nodes(4-4-64G)，but "CUDA out of memory" too! like above. anything wrong with the commands?

torchrun --nproc_per_node=4  \
    --nnodes=4 \
    --node_rank=0 \
    --master_addr=172.xx.xx.x \
    --master_port=10001 flashvideo/dist_inf_text_file.py
    ...

[jshilong]

The CUDA memory bottleneck likely stems from the long sequence attention in the second stage, which multi-node training can't alleviate.

You can try demo.ipynb to ensure there's no issue with the launch command.

[jshilong]

@sankexin Adjusting second_img_size = [270, 480] won't provide any improvement for video(it should be large enough comparing to first stage resolution). Ideally, it should only run at the training size of 1080p. However, making minor changes to the second stage size may still yield reasonable results.

The 1080p results of this women(second prompt with default seed) should be as follows:

[sankexin]

@sankexin Adjusting second_img_size = [270, 480] won't provide any improvement for video(it should be large enough comparing to first stage resolution). Ideally, it should only run at the training size of 1080p. However, making minor changes to the second stage size may still yield reasonable results.

The 1080p results of this women(second prompt with default seed) should be as follows:

Thank you! the project is a good idea ,but memory so high! minor changes to the second stage size alse not good . Looking forward to providing a small memory version as soon as possible, 1080p currently requires too much memory.