Support MOSS-VL

[SSSSuperC]

PR type

• More Models or Datasets Support

PR information

Add MOSS-VL (8B, model_type=moss_vl) as a first-class multimodal model for the Transformers backend. The model ships as remote code (48-layer LLM with vision cross-attention every 4 layers, 27-layer ViT with deep-stack features, merger, and a bare nn.Parameter separator token); this PR integrates it without copying the model implementation.

Supported

• swift infer (image/video) and swift sft: LoRA (target_modules=all-linear) and full-parameter
• Single/multi-image, single/multi-video, mixed media, text-only (native blank-image contract), multi-turn, batch size > 1
• freeze_vit / freeze_aligner / freeze_llm on the real module boundaries; LoRA target scanning skips the bare separator_token, LoRA-LLM saves it by exact name
• Non-reentrant gradient checkpointing, language + vision tower (the reentrant mode is not supported: it double-backwards through the vision deep-stack/shared compute graph)
• Single-node Ulysses sequence parallel (--sequence_parallel_size 2/4/8) with SDPA / FlashAttention-2 / FlashAttention-3, including DP × SP. Cross-attention modules keep full-sequence vision K/V and bypass the all-to-all (marked _sp_local_attention by the model loader); the text-axis masks (cross_attention_mask, full_text_row_masked_out_mask) are padded/split by a loader-registered pre-hook — no model-specific names in generic SP code
• Supervision via the native processor's labels_spans (binary strategies only; non-binary loss-scale raises explicitly)
• Controlled truncation: text-only truncation updates all text-axis fields; truncation that breaks the vision token/media metadata contract raises MaxLengthError
• Full/merged checkpoints reload standalone (remote code + processor + chat template saved); LoRA checkpoints load as base model + adapter and support merge/resume

Non-goals for this PR: packing=true / padding-free, Megatron/TP/PP/CP, RLHF, vLLM/lmdeploy/SGLang, quantization, streaming video.

Generic changes, and why each is required by this model

All are behavior-preserving for existing models and carry regression tests.

• find_all_linears skips non-Module paths when scanning LoRA targets: MOSS-VL's separator_token is a bare nn.Parameter (not an nn.Module); the previous scan crashed on it, blocking target_modules=all-linear.
• lora_llm matches aligner parameters by exact name: required to save the trainable bare separator_token in LoRA-LLM checkpoints — prefix-based matching never hits leaf parameters.
• SP attention plumbing, all required by the MOSS-VL SP support (padding_free=false, SDPA/FA2/FA3): explicit error on eager (previously silently wrong); FA3 registered into the SP-aware mask functions (the missing registration left the mask at local length after the all-to-all and zeroed attention outputs on later ranks at SP≥4); 2D attention masks padded before split (MOSS-VL passes 2D masks through the SP hook).

Tests: 22 lightweight tests (no 8B weight download): registration, template/processor golden alignment, collator padding, mixed-media batches, truncation contract, LoRA scanning, SP mask split/localization. pre-commit run --all-files passes.

Requires: transformers>=4.57.1,<5, torchcodec (0.7.x for torch 2.8), joblib; flash-attn for the SP example.

Experiment results

• Field-level golden alignment (text/tokens/labels/media order/grid/cross-attention mask) with the native processor on text/image/video/mixed/multi-turn samples
• H200 GPU smoke: LoRA and full SFT converge with finite loss/gradients; checkpoint save/resume/merge verified (resume loss bit-exact vs continuous run)
• Sequence parallel: SDPA/FA2/FA3 at SP=2/4/8 and DP4×SP2 all train; first-step loss matches the single-GPU baseline bit-exactly
• Full SFT ZeRO-3 on 8×H200: finite loss/grad norm, peak memory ~27-29 GiB
• Examples use public IDs (OpenMOSS-Team/MOSS-VL-Instruct-0708, lmms-lab/VideoChatGPT)

[tastelikefeet]

1. parameter_name == module_prefix 是否有意义呢 2. --freeze_aligner false 及 --tuner_type lora_llm 是否会运行报错需要测试下

[meichangsu1]

请确认这里对其他模型的开启padding-free/packing时候的影响

[meichangsu1]

if (
self.world_size == 1
or getattr(module, '_sp_local_attention', False)
or module.class not in [m.class for m in text_model.modules()]
)
是否这么处理更合适

[meichangsu1]

if (
self.world_size == 1
or getattr(module, '_sp_local_attention', False)
or module.class not in [m.class for m in text_model.modules()]
)
是否这么处理更合适

[SSSSuperC]

PR #9944 更新说明

一、关于本次 push：SP 支持撤出本 PR

本次更新将 sequence parallel 支持整体撤出本 PR，改动范围严格限定为：

• swift/sequence_parallel/sequence_parallel.py、swift/sequence_parallel/ulysses.py：全部还原为上游实现。
• MossVLTemplate 增加显式护栏：--sequence_parallel_size > 1 时抛出 NotImplementedError: MOSS-VL does not support sequence parallel yet.，避免用户误入未支持路径产生静默错误

SP 支持后续将整理为独立 PR 提交。

撤回后本 PR 的范围：模型/模板接入 + 两个 MOSS-VL 必需的通用小修复（find_all_linears 跳过裸 nn.Parameter、lora_llm 精确名匹配），公共代码不含任何模型特判。

---

二、回复 tastelikefeet 的 review（swift/tuner_plugin/lora_llm.py）

review 原文：1. parameter_name == module_prefix 是否有意义呢 2. --freeze_aligner false 及 --tuner_type lora_llm 是否会运行报错需要测试下

感谢指正，两点分别回复：
1. parameter_name == module_prefix 确实没有意义，已删除。

现已统一为：

if f'.{module_prefix}.' in parameter_name or f'.{parameter_name}'.endswith(f'.{module_prefix}'):

2. 两个组合已实测（H200，MOSS-VL-0708-Instruct），其中 lora_llm 初版确实会报错，已修复：

• --tuner_type lora --freeze_aligner false：正常训练，无报错。
• --tuner_type lora_llm --freeze_aligner false：初版在 prepare_model 的解冻循环中崩溃，deep_getattr(model, 'model.separator_token') 返回 None。根因：MossVLForConditionalGeneration 为 visual/language_model 提供了 BC property（与 Qwen3VL 相同模式），但唯独没有 separator_token 的，属性路径解析中断。修复方式是在 MossVLLoader 中为模型类补上该 property（模型私有改动，未动公共代码）。
• 修复后验证：训练 2 步 → vit.safetensors 包含 base_model.model.model.separator_token 且值为训练后的结果→ 以 lr=0 从 checkpoint resume 后该参数取值逐位一致，保存/加载链路完整正确。
• 另验证 lora_llm + FA3 + 视频样本训练正常。