migration.md

# Schema migration A persisted table's rows are stored with rkyv, positionally. Nothing on disk records which columns produced them, so a table whose columns change reads every existing row through the new layout, and rkyv does exactly what it is asked: it interprets the old bytes as the new shape. That does not surface as an error. It surfaces as fields shifted by one, ids arriving as titles and numbers arriving as garbage, on a table that opened without complaint. A real case, from adding a single column to a five-column table: ```json {"id":"ment)","project_id":"item-03fd09c6-...","title":"proj-846b5542-...","status":"Reflect PR merge state in the UI"} ``` `id` is debris, `project_id` holds the id, `title` holds the project id, `status` holds the title. Every value is present and every value is in the wrong field. Migration exists so a column change is a versioned, explicit transformation instead of a reinterpretation. ## The four pieces | | | |---|---| | `version:` on `worktable!` | Stamps the current schema. Defaults to `1`. | | `worktable_version!` | Declares an *old* schema so its rows can still be read. | | `Migration` | One `migrate` function per hop between versions. | | `migration_engine!` | Generates `migrate(source, target, ctx)`, chaining the hops. | The version is written into the `SpaceInfoPage` at page 0 of `.wt.data`, and `worktable::migration::detect_version` reads it back. That is how the engine knows which hop to start from without being told. ## Versioning a table Bump `version:` in the same change that alters the columns. Treat it as part of editing the column list, not as a follow-up. ```rust worktable!( name: User, version: 2, // was 1 persist: true, columns: { id: UserId primary_key custom, username: String, recovery_code_hash: String optional, // the new column }, indexes: { username_idx: username unique, }, ); ``` > `version` must appear **before** `columns`, `indexes`, `queries` and `config`. The > macro rejects it otherwise, with `version must be specified before > columns/indexes/queries/config`. ## Declaring the old shape Each superseded version stays declared so its rows remain readable, using `worktable_version!` rather than `worktable!`. It is deliberately stripped: name, version, columns. No `persist`, no `indexes`, no `queries`, because nothing queries a historical table. It is read once, in order, and dropped. ```rust mod v1 { use worktable::prelude::*; use worktable::worktable_version; worktable_version!( name: User, version: 1, columns: { id: UserId primary_key custom, username: String, }, ); impl_user_pk_generator!(); } ``` The **name does not change**. `User` at v1 and `User` at v2 are the same table at two points in time, which is what lets the engine find the data. A table with a custom primary key needs its `PrimaryKeyGenerator`, `PrimaryKeyGeneratorState` and `TablePrimaryKey` impls in *every* version module. Write them once as a `macro_rules!` and invoke it per module rather than copying them; a table whose primary key is a plain `String` or an autoincrement integer needs none of this. ## Writing the hops One impl per adjacent pair. The engine composes them, so v1 to v4 is three functions rather than one that has to know about every intermediate shape. ```rust pub struct Migrator; impl Migration for Migrator { type Context = Context; fn migrate(row: v1::UserRow, _ctx: &Self::Context) -> v2::UserRow { v2::UserRow { id: row.id, username: row.username, recovery_code_hash: None, // the new column's value for old rows } } } ``` The final hop targets the current row type, not a version module: ```rust impl Migration for Migrator { .. } ``` `Context` carries anything the transformation needs from outside the row: an encryption key, a clock, a lookup table. It must implement `Default`, and when a migration cannot work without real values, the honest `Default` is one that panics: ```rust impl Default for Context { fn default() -> Self { panic!("User migration requires an explicit Context") } } ``` ## Generating the engine ```rust migration_engine!( migration: Migrator, current: UserWorkTable, ctx: Context, version_tables: { 1 => v1::UserWorkTable, 2 => v2::UserWorkTable, 3 => v3::UserWorkTable, }, ); ``` This emits `MigratorEngine::migrate(source_path, target_path, &ctx)`, which detects the source version, opens the target at the current version, runs the chain, drains with `wait_for_ops`, and returns a `MigrationReport` naming the version it came from. ## Running it **Source and target are different directories, and that is the point.** The original is never modified, so a migration that fails partway leaves the only copy of the data intact. Nothing needs undoing, because nothing was done to the source. A separate binary is the natural home. Migration is not something an application should attempt during startup: a crash halfway through a boot-time migration is exactly the scenario with no good recovery, and running it out of band means you can inspect the result before pointing anything at it. ```rust let report = user::MigratorEngine::migrate( &source_dir.to_string_lossy(), &target_dir.to_string_lossy(), &user::Context { .. }, ).await?; info!(source_version = report.source_version, "User migrated"); ``` ## Tables that did not change Most will not have. They still need to arrive in the target directory, and copying is correct: no schema changed, so there is nothing to transform. ```rust let table_dir = source_dir.join(OtherWorkTable::name_snake_case()); if table_dir.exists() { copy_dir_recursive(&table_dir, &target_dir.join(OtherWorkTable::name_snake_case())).await?; } ``` Forgetting one is silent: the target opens fine and that table is simply empty. ## Running it twice A source already at the current version has no hop to take, and `migrate` reports that as `Unsupported version: N`. Treat it as "already current" and copy instead, which makes the whole run idempotent and safe to re-run after fixing an unrelated table: ```rust match user::MigratorEngine::migrate(&source, &target, &ctx).await { Ok(report) => info!(source_version = report.source_version, "migrated"), Err(e) if e.to_string().contains("Unsupported version: 5") => { info!("already at v5, copying"); copy_dir_recursive(&user_dir, &target.join("user")).await?; } Err(e) => return Err(e).wrap_err("User migration failed"), } ``` Matching on the message text is fragile, and it is what the API offers today. ## Checklist - [ ] `version:` bumped in the same commit as the column change, and placed before `columns` - [ ] The previous shape declared with `worktable_version!`, same `name`, old `version` - [ ] Custom primary-key generator impls repeated in the version module - [ ] A `Migration` impl for the new hop, ending at the current row type - [ ] The new version added to `version_tables` - [ ] Unchanged tables copied to the target - [ ] Run against a copy first, and confirm the row count before swapping anything in