Qwen3.6-35B-A3B模型，使用lora微调，导出时缺少mtp权重

[JTCT-chen]

Checklist / 检查清单

• I have searched existing issues, and this is a new bug report. / 我已经搜索过现有的 issues，确认这是一个新的 bug report。

Bug Description / Bug 描述

Qwen3.6-35B-A3B模型，使用lora微调，导出后使用vllm部署，提示
ValueError: Following weights were not initialized from checkpoint: {'model.layers.0.self_attn.q_norm.weight', 'model.pre_fc_norm_hidden.weight', 'model.layers.0.post_attention_layernorm.weight', 'model.norm.weight', 'model.layers.0.self_attn.k_norm.weight', 'model.pre_fc_norm_embedding.weight', 'model.layers.0.input_layernorm.weight'}

How to Reproduce / 如何复现

Name: ms_swift Version: 4.5.0.dev0
torch Version: 2.13.0
Name: transformers Version: 5.12.1
Name: vllm Version: 0.25.1

Additional Information / 补充信息

No response

[hjh0119]

transformers 加载模型会丢弃 mtp 层权重，有两个办法

1. 手动拷贝原模型的mtp权重
2. megatron训练 https://github.com/modelscope/ms-swift/blob/main/examples/models/qwen3_next/mtp.sh

[LongGieGie]

model.layers.{0...39}.mlp.experts.{0...255}.down_proj.weight | UNEXPECTED |
model.layers.{0...39}.mlp.experts.{0...255}.up_proj.weight | UNEXPECTED |
model.layers.{0...39}.mlp.experts.{0...255}.gate_proj.weight | UNEXPECTED |
model.layers.{0...39}.mlp.experts.down_proj | MISSING |
model.layers.{0...39}.mlp.experts.gate_up_proj | MISSING |

Notes:

• UNEXPECTED: can be ignored when loading from different task/architecture; not ok if you expect identical arch.
• MISSING: those params were newly initialized because missing from the checkpoint. Consider training on your downstream task.

@hjh0119 我使用ms_swift=4.1.2来lora微调Qwen3.6-35B-A3B模型，使用transformers==5.6.1加载lora_merge报错
训练脚本里没有mtp，是这个原因吗
megatron sft
--model ${MODEL_PATH}
--dataset ${TRAIN_DATA}
--load_from_cache_file true
--add_non_thinking_prefix false
--split_dataset_ratio 0.01
--save_safetensors true
--merge_lora true
--tuner_type lora
--lora_rank 8
--lora_alpha 32
--target_modules all-linear
--expert_model_parallel_size 4
--moe_permute_fusion true
--moe_grouped_gemm true
--moe_shared_expert_overlap true
--moe_aux_loss_coeff 1e-6
--micro_batch_size 1
--global_batch_size 16
--recompute_granularity full
--recompute_method uniform
--recompute_num_layers 1
--num_train_epochs 1
--finetune true
--freeze_llm false
--freeze_vit true
--freeze_aligner true
--cross_entropy_loss_fusion true
--lr 1e-4
--lr_warmup_fraction 0.05
--min_lr 1e-5
--output_dir ${OUTPUT_DIR}
--save_strategy epoch
--max_length 8192
--dataloader_num_workers 8
--dataset_num_proc 8
--sequence_parallel true
--attention_backend flash
2>&1 | tee ${TRAIN_LOG_FILE}

[hjh0119]

@LongGieGie 需要设置 mtp_num_layers

[LongGieGie]

@hjh0119 mtp_num_layers数值怎么确定，就是按https://github.com/modelscope/ms-swift/blob/main/examples/models/qwen3_next/mtp.sh中，默认mtp_num_layers=1？

[hjh0119]

https://swift.readthedocs.io/zh-cn/latest/Megatron-SWIFT/Command-line-parameters.html#megatron:~:text=MTP%E5%8F%82%E6%95%B0-,mtp_num_layers,-%3A%20%E5%A4%9Atoken%E9%A2%84%E6%B5%8B

[anxu829]

使用这个脚本试试：
megatron 训练， （True 会直接导出一个包含 mtp 的权重）， 如果训练的时候使用 merge_lora : False ：

参考如下配置：

{
"model": "/mnt/mllm/mllm/pretrain_models/Qwen3.5-4B",
"adapters": "/root/0814/v23-20260819-154424/checkpoint-8", # 包含adatper
"output_dir": "/root/0814/v23-20260819-154424/checkpoint-8-merged",
"merge_lora": true,
"to_hf": true,
// 添加参数，他会先基于参数 构造 mcore model
// 然后把 参数加载进去
"tuner_type" : "lora",
"lora_rank" : 32,
"lora_alpha" : 64,
"target_modules" : ["all-linear"],
"freeze_llm" : false,
"freeze_vit": true ,
"freeze_aligner": true,
"mtp_num_layers": 1 # 重点是这个 !
}

然后用 swift/cli/_megatron/export.py 导出即可

但是 transformers 的 modeling qwen3_5 不支持加载 mtp 权重，这个部分权重的加载需要推理框架支持