Skip to content

Fix load_lora to rebuild post-trained LoRA checkpoints on their recorded pretrained base - #83

Open
dineshmane wants to merge 1 commit into
dreamzero0:mainfrom
dineshmane:fix/load-lora-post-trained-base
Open

Fix load_lora to rebuild post-trained LoRA checkpoints on their recorded pretrained base#83
dineshmane wants to merge 1 commit into
dreamzero0:mainfrom
dineshmane:fix/load-lora-post-trained-base

Conversation

@dineshmane

@dineshmane dineshmane commented Jul 24, 2026

Copy link
Copy Markdown

Fixes #82

Problem

VLA.load_lora composes LoRA checkpoints onto the DiT referenced by
config.json's diffusion_model_pretrained_path. For the post-training
workflow (yam_training.sh / agibot_training.sh), that path is the raw
Wan2.1 dir
(from the dit_version flag) — but training loaded the full
DreamZero-AgiBot checkpoint
(pretrained_model_path) before injecting the
adapters, so the saved deltas are relative to the AgiBot base. The shipped
load path therefore silently serves raw Wan + deltas: the model loads and
runs, with the robot pretraining discarded.

Fix

load_lora now reads pretrained_model_path from the checkpoint's
experiment_cfg/conf.yaml (already saved by every training run):

  • Present → replicate training's composition: instantiate with
    defer_lora_injection=True, skip_component_loading=True, load the base
    checkpoint shards (same loop as create_model in
    groot/vla/experiment/base.py), inject_lora_after_loading(), then overlay
    the checkpoint's deltas. If the base dir is missing, raise a clear
    FileNotFoundError instead of silently substituting a different base.
  • Absent (original from-scratch/DROID workflow) → unchanged behavior.

Validation

1000-step yam LoRA fine-tune of DreamZero-AgiBot on
lerobot/aloha_static_screw_driver (rev v2.1, adapted to yam embodiment).
Open-loop replay of 150 frames (scripts/open_loop_yam.py, slices adapted to
the 14-dim ALOHA layout); first predicted action vs ground truth, MSE in rad²:

Load path overall left_gripper
load_lora before this PR (raw-Wan base) 0.00988 0.07602
offline merge onto AgiBot base (ground truth) 0.00100 0.00180
load_lora with this PR 0.00100 0.00180
AgiBot base, no fine-tune (reference) 0.00736 0.00170

The pre-PR path is worse than not fine-tuning at all; the patched path
matches the offline-merged ground truth exactly (bit-identical MSE to 16 significant digits) — the patched loader reconstructs the same weights as the training-time composition.

Hardware: 4× RTX PRO 6000 Blackwell, torch 2.8.0+cu129, flash-attn 2.8.3.post1.

Notes

  • No change for DROID-workflow checkpoints (no pretrained_model_path in conf.yaml).
  • An alternative/complementary fix — rewriting diffusion_model_pretrained_path
    in the saved config at train time — was left out to keep existing checkpoints
    fixable without retraining; happy to add it if maintainers prefer.

…ed pretrained base

LoRA checkpoints produced by the post-training workflow (yam_training.sh /
agibot_training.sh with pretrained_model_path=DreamZero-AgiBot) were loaded
onto the raw Wan2.1 DiT referenced by config.json's
diffusion_model_pretrained_path, silently discarding the robot pretraining
the deltas were trained against.

load_lora now reads pretrained_model_path from the checkpoint's
experiment_cfg/conf.yaml; when present, it replicates the training-time
composition (base checkpoint shards -> inject_lora_after_loading -> delta
overlay) and raises a clear error if the base directory is missing instead
of silently substituting a different base. Checkpoints from the original
from-scratch workflow (no pretrained_model_path) are unaffected.

Measured on a 1000-step yam LoRA fine-tune of DreamZero-AgiBot on
lerobot/aloha_static_screw_driver@v2.1, open-loop action MSE over 150
frames: shipped path 0.00988 overall (worse than the un-fine-tuned base,
0.00736); correct composition 0.00100.
@dineshmane dineshmane changed the title fix: load_lora rebuilds post-trained LoRA checkpoints on their record… Fix load_lora to rebuild post-trained LoRA checkpoints on their recorded pretrained base Jul 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

load_lora composes post-trained LoRA checkpoints on raw Wan instead of the DreamZero-AgiBot base they were trained on

1 participant