Skip to content

Fix zero multimodal learning rates in Megatron - #9948

Open
taking-lying-flat wants to merge 1 commit into
modelscope:mainfrom
taking-lying-flat:agent/fix-megatron-multimodal-zero-lr
Open

Fix zero multimodal learning rates in Megatron#9948
taking-lying-flat wants to merge 1 commit into
modelscope:mainfrom
taking-lying-flat:agent/fix-megatron-multimodal-zero-lr

Conversation

@taking-lying-flat

@taking-lying-flat taking-lying-flat commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Summary

  • honor explicit vit_lr=0.0 and aligner_lr=0.0 values in Megatron parameter groups
  • assign concrete max_lr and min_lr values directly instead of representing multimodal overrides as ratios of the LLM learning rate
  • preserve generic learning-rate scaling and decoupled learning-rate behavior

Root cause

The multimodal parameter-group logic checked vit_lr and aligner_lr by truthiness. Explicit zero values therefore fell through to the default LLM learning rate instead of freezing those modules. Nonzero overrides were also converted to override / lr, which could divide by zero when the base learning rate was zero.

Impact

Users can safely set either multimodal learning rate to zero, and configurations with a zero base learning rate no longer encounter division by zero while constructing optimizer groups.

@taking-lying-flat
taking-lying-flat marked this pull request as ready for review August 19, 2026 11:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant