Skip to content

Fix Megatron GRPO vocab-parallel log-prob gradients - #9947

Open
taking-lying-flat wants to merge 1 commit into
modelscope:mainfrom
taking-lying-flat:agent/fix-megatron-grpo-tp-logps
Open

Fix Megatron GRPO vocab-parallel log-prob gradients#9947
taking-lying-flat wants to merge 1 commit into
modelscope:mainfrom
taking-lying-flat:agent/fix-megatron-grpo-tp-logps

Conversation

@taking-lying-flat

@taking-lying-flat taking-lying-flat commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

What changed

  • Replace the manual target-token gather and non-autograd TP all-reduce with Megatron Core's vocab-parallel cross entropy.
  • Preserve -100 label masking by substituting a safe target before cross entropy and masking the resulting token log-probability.
  • Compute the full vocab-parallel log-softmax only when entropy is requested.

Why

The previous forward path produced the correct target-token log-probability, but its ordinary torch.distributed.all_reduce did not propagate the remote autograd graph. When the target token belonged to another TP rank, non-owner vocabulary shards received zero or otherwise incorrect logits gradients.

This affects the differentiable current-policy forward used directly by Megatron GRPO policy loss when tensor parallelism is greater than one.

Megatron Core's vocab-parallel cross entropy provides the required custom backward: every vocabulary shard keeps its softmax gradient, while only the shard owning the target token subtracts the target contribution.

@taking-lying-flat
taking-lying-flat marked this pull request as ready for review August 19, 2026 11:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant