Fix load_lora to rebuild post-trained LoRA checkpoints on their recorded pretrained base - #83
Open
dineshmane wants to merge 1 commit into
Open
Conversation
…ed pretrained base LoRA checkpoints produced by the post-training workflow (yam_training.sh / agibot_training.sh with pretrained_model_path=DreamZero-AgiBot) were loaded onto the raw Wan2.1 DiT referenced by config.json's diffusion_model_pretrained_path, silently discarding the robot pretraining the deltas were trained against. load_lora now reads pretrained_model_path from the checkpoint's experiment_cfg/conf.yaml; when present, it replicates the training-time composition (base checkpoint shards -> inject_lora_after_loading -> delta overlay) and raises a clear error if the base directory is missing instead of silently substituting a different base. Checkpoints from the original from-scratch workflow (no pretrained_model_path) are unaffected. Measured on a 1000-step yam LoRA fine-tune of DreamZero-AgiBot on lerobot/aloha_static_screw_driver@v2.1, open-loop action MSE over 150 frames: shipped path 0.00988 overall (worse than the un-fine-tuned base, 0.00736); correct composition 0.00100.
load_lora to rebuild post-trained LoRA checkpoints on their recorded pretrained base
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #82
Problem
VLA.load_loracomposes LoRA checkpoints onto the DiT referenced byconfig.json'sdiffusion_model_pretrained_path. For the post-trainingworkflow (
yam_training.sh/agibot_training.sh), that path is the rawWan2.1 dir (from the
dit_versionflag) — but training loaded the fullDreamZero-AgiBot checkpoint (
pretrained_model_path) before injecting theadapters, so the saved deltas are relative to the AgiBot base. The shipped
load path therefore silently serves
raw Wan + deltas: the model loads andruns, with the robot pretraining discarded.
Fix
load_loranow readspretrained_model_pathfrom the checkpoint'sexperiment_cfg/conf.yaml(already saved by every training run):defer_lora_injection=True, skip_component_loading=True, load the basecheckpoint shards (same loop as
create_modelingroot/vla/experiment/base.py),inject_lora_after_loading(), then overlaythe checkpoint's deltas. If the base dir is missing, raise a clear
FileNotFoundErrorinstead of silently substituting a different base.Validation
1000-step yam LoRA fine-tune of DreamZero-AgiBot on
lerobot/aloha_static_screw_driver(rev v2.1, adapted to yam embodiment).Open-loop replay of 150 frames (
scripts/open_loop_yam.py, slices adapted tothe 14-dim ALOHA layout); first predicted action vs ground truth, MSE in rad²:
load_lorabefore this PR (raw-Wan base)load_lorawith this PRThe pre-PR path is worse than not fine-tuning at all; the patched path
matches the offline-merged ground truth exactly (bit-identical MSE to 16 significant digits) — the patched loader reconstructs the same weights as the training-time composition.
Hardware: 4× RTX PRO 6000 Blackwell, torch 2.8.0+cu129, flash-attn 2.8.3.post1.
Notes
pretrained_model_pathin conf.yaml).diffusion_model_pretrained_pathin the saved config at train time — was left out to keep existing checkpoints
fixable without retraining; happy to add it if maintainers prefer.