docs: add boot_native train-from-scratch recipe to migration guide - #1600
docs: add boot_native train-from-scratch recipe to migration guide#1600priyanka25aug wants to merge 1 commit into
Conversation
jlarson4
left a comment
jlarson4
left a comment
There was a problem hiding this comment.
Hey @priyanka25aug, great work on this! Just a couple small documentation items to update before I merge this. The example itself is well built, thank you for your hard work on this
| import. `cfg.seed` seeds the weight initialiser; omitting it lets the | ||
| global RNG advance normally. Passing a `HookedTransformerConfig` (or any | ||
| other legacy config object) to `boot_native` raises `TypeError` — construct | ||
| a `TransformerBridgeConfig` directly. To re-randomise weights after |
There was a problem hiding this comment.
Under this recipe's own seed=42, init_weights() regenerates identical weights. 0 of 38 parameters changed when I ran it because it rebuilds its generator from cfg.seed. Can you please match the table row's "reinitialize in place" wording (migrating_to_v3.md line 240) and note that re-randomising needs a changed or cleared seed?
| ``` | ||
|
|
||
| `boot_native` makes no HuggingFace Hub call and requires no `transformers` | ||
| import. `cfg.seed` seeds the weight initialiser; omitting it lets the |
There was a problem hiding this comment.
HT's init_weights calls torch.manual_seed(cfg.seed) globally, while boot_native forks the RNG, so a training loop migrated verbatim silently loses reproducible data shuffling. Could this note state that difference explicitly?
2 participants
Closes #1557
The
migrating_to_v3.mdondev-4.xalready has 13 API migration table rows and 4 runnable prose recipes, butboot_native/TransformerBridgeConfigonly appeared as a table row with no runnable before/after snippet.This PR adds the missing
### Build a TL-native model from scratchrecipe, showing the direct replacement forHookedTransformerConfig + HookedTransformer(cfg). Notes included: no HF Hub call,cfg.seedreproducibility,TypeErroron legacy config objects, andbridge.init_weights()for re-randomisation.Base branch:
dev-4.x(targets the v4 preview line where the rest of the migration API section lives).