fix: consolidate release hardening and torn-store refusal

Cargo.toml

@@ -3,7 +3,7 @@ members = ["codegen", "examples", "performance_measurement", "performance_measur
 
 [package]
 name = "worktable"
-version = "1.0.0-beta.3"
+version = "1.0.0-beta.4"
 edition = "2024"
 authors = ["Handy-caT"]
 license = "MIT"
@@ -61,7 +61,7 @@ tracing = "0.1"
 url = { version = "2", optional = true }
 uuid = { version = "1.24.0", features = ["v4", "v7"] }
 walkdir = { version = "2", optional = true }
-worktable_codegen = { path = "codegen", version = "=1.0.0-beta.3" }
+worktable_codegen = { path = "codegen", version = "=1.0.0-beta.4" }
 
 [dev-dependencies]
 chrono = "0.4.43"

README.md

@@ -13,7 +13,7 @@ from a macro, and that persisting it is one feature flag away.
 ## Install
 
 ```sh
-cargo add worktable@1.0.0-beta.3
+cargo add worktable@1.0.0-beta.4
 ```
 
 ## What you get
@@ -34,6 +34,15 @@ cargo add worktable@1.0.0-beta.3
 
 ## Persistence
 
+> [!WARNING]
+> WorkTable persistence is **best-effort, not crash-atomic database durability**.
+> A successful mutation means the in-memory change was accepted and queued; it does
+> not mean the change is on stable storage. `wait_for_ops()` and `close()` flush the
+> persistence pipeline, but the current disk format has no transaction journal and
+> does not `fsync` every batch. Process or power loss can therefore lose acknowledged
+> changes. A torn store is refused with `PersistenceLoadError` rather than opened as
+> plausible-but-invented rows. See the [durability and recovery contract](docs/persistence-durability.md).
+
 Persistence is implemented, not planned. `PersistedWorkTable` and `PersistenceConfig` are
 exported from the crate root; the prelude carries `DiskPersistenceEngine`,
 `ReadOnlyPersistenceEngine`, the space and table-of-contents types, and the operation-log
@@ -44,10 +53,10 @@ S3 support layers *on top of* the disk engine rather than replacing it.
 
 ```toml
 [dependencies]
-worktable = { version = "=1.0.0-beta.3", features = ["s3-support"] }   # S3 sync, optional
+worktable = { version = "=1.0.0-beta.4", features = ["s3-support"] }   # S3 sync, optional
 ```
 
-Persisted indexes default to WorkTablesIndex. Vanilla IndexSet can be selected explicitly with `using indexset` while retaining the existing disk/S3 representation. Congee and Arctic are explicitly memory-only and require `persist: false`. The full syntax and capability matrix are documented in [Per-index backends with `using`](docs/index-backend-dsl-proposal.md).
+Persisted indexes default to WorkTablesIndex. Vanilla IndexSet can be selected explicitly with `using indexset` while retaining the existing disk/S3 representation. Congee and Arctic persistence is experimental and uses their native checkpoint/WAL adapters; declarations using either backend must state `persist: true` or `persist: false` explicitly. The full syntax and capability matrix are documented in [Per-index backends with `using`](docs/index-backend-dsl-proposal.md).
 
 ### Persistence lifecycle
 
@@ -58,13 +67,49 @@ table.wait_for_ops().await?; // drain currently queued operations
 table.close().await?;        // stop intake, drain, and join the engine task
 ```
 
+`wait_for_ops()` requires application-level writer quiescence if it is being used as
+a shutdown boundary; it does not prevent another task from queueing later work.
+Neither method is an `fsync` or a transaction commit. The exact guarantees and the
+snapshot-restore/replay procedure are documented in
+[`docs/persistence-durability.md`](docs/persistence-durability.md).
+
 An unrecoverable event gap, queue-analysis error, batch-apply error, or engine-task
 failure moves persistence into a terminal failed state. The original error is
 returned to waiters, graceful close, and later mutation attempts; later durable
 operations are not applied after that failure. Dropping a busy table remains a
 last-resort diagnostic path, so applications should call `close()` during orderly
 shutdown.
 
+These lifecycle calls are not a crash-durability guarantee:
+
+| Boundary | Current guarantee |
+|---|---|
+| Mutation returns | The in-memory change was accepted and its persistence operation was queued. |
+| `wait_for_ops()` returns | The persistence engine completed the queued operations; no fsync or stable-storage guarantee is made. |
+| `close()` returns | Intake stopped, the queue drained, and the engine task joined; no fsync guarantee is made. |
+| Process crash / `SIGKILL` | Acknowledged rows may be lost and the file may be torn. |
+| Power loss | No atomic-batch or stable-storage guarantee. |
+
+The 1.0 beta persistence tier is therefore best-effort rather than a substitute
+for a crash-atomic embedded database. Applications requiring crash durability
+need an external snapshot/rebuild strategy. A graceful persistence error is
+terminal and surfaced consistently, but abrupt termination can currently leave
+a partial multi-file batch. Loading audits archived rows plus primary and
+secondary index consistency before exposing the table; torn state is refused as
+`PersistenceLoadError` and must be restored or rebuilt as documented above.
+
+Generated persisted tables store row-schema, primary-key, and secondary-index
+metadata in `SpaceInfo`. Existing legacy files whose schema metadata is
+completely empty remain readable and are not rewritten merely by loading them;
+their schema therefore cannot be validated. A non-empty schema mismatch is
+rejected before rows are loaded.
+
+Persisted vacuum compacts the live in-memory layout and keeps disk indexes
+consistent with moved rows, but it does not truncate `.wt.data`. Use
+`persisted_data_file_size_bytes().await` on a generated persisted table to
+observe physical growth and decide when to snapshot/rebuild or run future
+offline compaction.
+
 WorkTablesIndex uses its predictable branch-based node search by default in WorkTable. This avoids a measured regression for sequential numeric-key workloads. Alternative search policies remain compile-time feature gates: disable WorkTable's default features and enable one of `wti-hybrid-search`, `wti-std-search`, or `wti-superslice-search` (plus any other features such as `s3-support`). Prefer one search feature for an unambiguous build. If Cargo feature unification enables several, WorkTablesIndex applies the documented deterministic precedence rather than rejecting the graph.
 
 ## Concurrent read/write publication
@@ -131,6 +176,9 @@ with persistence as an option rather than an assumption.
 structs that will be used for table logic.
 
 ```rust
+use worktable::prelude::*;
+use worktable::worktable;
+
 worktable!(
     name: Test,
     columns: {
@@ -181,6 +229,17 @@ Flags list:
 - `primary_key` flag and related to it.
 - `optional` flag.
 
+Column modifiers are intentionally inline in the 1.0 DSL. A separate
+`attributes` section is not supported; the macro emits an actionable diagnostic
+if one is used. This keeps one canonical grammar through the 1.0 compatibility
+freeze:
+
+```text
+id: u64 primary_key autoincrement,
+tenant: String primary_key,
+nickname: String optional,
+```
+
 #### `primary_key` flag declaration
 
 If user want to mark column as primary key `primary_key` flag is used. This flag can be used on multiple columns at a
@@ -296,7 +355,7 @@ method for now is `select_by_<indexed_column_name>`. It will be described below.
 
 There are some default query implementations that are available for all `WorkTable`'s:
 
-- `select(&self, pk: <Name>PrimaryKey) -> Option<<Name>Row>`;
+- `select(&self, pk: impl Into<<Name>PrimaryKey>) -> Option<<Name>Row>`; borrowed `String`, `str`, tuple, and generated primary-key forms are accepted;
 - `insert(&self, row: <Name>Row) -> Result<<Name>PrimaryKey, WorkTableError>`;
 - `upsert(&self, row: <Name>Row) -> Result<(), WorkTableError>`;
 - `update(&self, row: <Name>Row) -> Result<(), WorkTableError>`;
@@ -305,7 +364,7 @@ There are some default query implementations that are available for all `WorkTab
 
 ### `queries` declaration
 
-`indexes` field is used to define table's queries schema. Queries are used to update/select/delete data.
+`queries` field is used to define table's queries schema. Queries are used to update/select/delete data.
 
 ```
 queries: {

codegen/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "worktable_codegen"
-version = "1.0.0-beta.3"
+version = "1.0.0-beta.4"
 edition = "2024"
 license = "MIT"
 description = "Proc-macro companion crate for worktable: the worktable! macro and its derives."

codegen/src/generators/in_memory/primary_key.rs

@@ -4,6 +4,7 @@ use crate::common::model::{GeneratorType, PrimaryKey};
 use crate::common::name_generator::{WorktableNameGenerator, is_unsized_vec};
 use crate::generators::in_memory::InMemoryGenerator;
 use crate::generators::index_backend::primary_key_backend_impl;
+use crate::generators::primary_key::gen_borrowed_primary_key_impl;
 
 use proc_macro2::{Ident, TokenStream};
 use quote::quote;
@@ -67,6 +68,7 @@ impl InMemoryGenerator {
         };
         let (backend_derive, backend_impl) =
             primary_key_backend_impl(self.columns.primary_index_backend, &ident, types)?;
+        let borrowed_impl = gen_borrowed_primary_key_impl(&ident, types);
 
         Ok(quote! {
             #[derive(
@@ -91,6 +93,7 @@ impl InMemoryGenerator {
             #[rkyv(derive(PartialEq, Eq, PartialOrd, Ord, Debug))]
             pub struct #ident(#(#types),*);
 
+            #borrowed_impl
             #backend_impl
         })
     }

codegen/src/generators/mod.rs

@@ -1,4 +1,5 @@
 pub mod in_memory;
 pub(crate) mod index_backend;
 pub mod persist;
+pub(crate) mod primary_key;
 pub mod read_only;

codegen/src/generators/persist/primary_key.rs

@@ -4,6 +4,7 @@ use crate::common::model::{GeneratorType, PrimaryKey};
 use crate::common::name_generator::{WorktableNameGenerator, is_unsized_vec};
 use crate::generators::index_backend::primary_key_backend_impl;
 use crate::generators::persist::PersistGenerator;
+use crate::generators::primary_key::gen_borrowed_primary_key_impl;
 
 use proc_macro2::{Ident, TokenStream};
 use quote::quote;
@@ -63,6 +64,7 @@ impl PersistGenerator {
         };
         let (backend_derive, backend_impl) =
             primary_key_backend_impl(self.columns.primary_index_backend, &ident, types)?;
+        let borrowed_impl = gen_borrowed_primary_key_impl(&ident, types);
 
         Ok(quote! {
             #[derive(
@@ -87,6 +89,7 @@ impl PersistGenerator {
             #[rkyv(derive(PartialEq, Eq, PartialOrd, Ord, Debug))]
             pub struct #ident(#(#types),*);
 
+            #borrowed_impl
             #backend_impl
         })
     }

codegen/src/generators/persist/table/impls.rs

@@ -1,9 +1,9 @@
 use convert_case::{Case, Casing};
-use proc_macro2::{Ident, Span, TokenStream};
+use proc_macro2::{Ident, Literal, Span, TokenStream};
 use quote::quote;
 
 use crate::common::model::GeneratorType;
-use crate::common::name_generator::{WorktableNameGenerator, is_unsized_vec};
+use crate::common::name_generator::{WorktableNameGenerator, is_float, is_unsized_vec};
 use crate::generators::persist::PersistGenerator;
 
 impl PersistGenerator {
@@ -26,6 +26,7 @@ impl PersistGenerator {
         let count_fn = self.gen_table_count_fn();
         let system_info_fn = self.gen_system_info_fn();
         let vacuum_fn = self.gen_table_vacuum_fn();
+        let validate_loaded_secondary_state_fn = self.gen_validate_loaded_secondary_state_fn();
 
         quote! {
             #persisted_impl
@@ -44,6 +45,96 @@ impl PersistGenerator {
                 #iter_with_async_fn
                 #system_info_fn
                 #vacuum_fn
+                #validate_loaded_secondary_state_fn
+            }
+        }
+    }
+
+    fn gen_validate_loaded_secondary_state_fn(&self) -> TokenStream {
+        if self.columns.indexes.is_empty() {
+            return quote! {
+                fn validate_loaded_secondary_state(&self, _path: &str) -> Result<(), PersistenceLoadError> {
+                    Ok(())
+                }
+            };
+        }
+
+        let expected_entries = self
+            .columns
+            .indexes
+            .iter()
+            .map(|(column, index)| {
+                let index_field = &index.name;
+                let row_field = &index.field;
+                let index_name = Literal::string(&index_field.to_string());
+                let field_type = self
+                    .columns
+                    .columns_map
+                    .get(column)
+                    .expect("indexed column should exist")
+                    .to_string();
+                let key = if is_float(&field_type) {
+                    quote! { OrderedFloat(row.#row_field) }
+                } else {
+                    quote! { row.#row_field.clone() }
+                };
+
+                if index.is_unique {
+                    quote! {
+                        if self.0.indexes.#index_field.lookup_for_select(&#key).map(|link| link.0) != Some(offset_link.0) {
+                            return Err(PersistenceLoadError::corrupt(
+                                path,
+                                format!("secondary index {} does not reference primary key {primary_key:?}", #index_name),
+                            ));
+                        }
+                    }
+                } else {
+                    quote! {
+                        if !self.0.indexes.#index_field
+                            .get(&#key)
+                            .any(|(_, candidate_link)| candidate_link.0 == offset_link.0)
+                        {
+                            return Err(PersistenceLoadError::corrupt(
+                                path,
+                                format!("secondary index {} does not reference primary key {primary_key:?}", #index_name),
+                            ));
+                        }
+                    }
+                }
+            })
+            .collect::<Vec<_>>();
+        let entry_counts = self
+            .columns
+            .indexes
+            .values()
+            .map(|index| {
+                let index_field = &index.name;
+                let index_name = Literal::string(&index_field.to_string());
+                quote! {
+                    if self.0.indexes.#index_field.len() != primary_count {
+                        return Err(PersistenceLoadError::corrupt(
+                            path,
+                            format!("secondary index {} contains a different number of rows than the primary index", #index_name),
+                        ));
+                    }
+                }
+            })
+            .collect::<Vec<_>>();
+
+        quote! {
+            fn validate_loaded_secondary_state(&self, path: &str) -> Result<(), PersistenceLoadError> {
+                let primary_count = self.0.primary_index.pk_map.len();
+                #(#entry_counts)*
+                for (primary_key, offset_link) in self.0.primary_index.pk_map.iter_values() {
+                    let row = self.0.data.select_non_ghosted_checked(offset_link.0).map_err(|error| {
+                        PersistenceLoadError::corrupt(
+                            path,
+                            format!("primary key {primary_key:?} references an invalid row: {error}"),
+                        )
+                    })?;
+                    #(#expected_entries)*
+                }
+                Ok(())
             }
         }
     }
@@ -123,7 +214,15 @@ impl PersistGenerator {
                     + 'static,
                 C: Clone + PersistenceConfig,
             {
-                async fn new(engine: E) -> eyre::Result<Self> {
+                async fn new(mut engine: E) -> eyre::Result<Self> {
+                    let schema = Self::space_info_default().inner;
+                    engine
+                        .ensure_schema(
+                            schema.row_schema,
+                            schema.primary_key_fields,
+                            schema.secondary_index_types,
+                        )
+                        .await?;
                     let mut inner = WorkTable::default();
                     inner.table_name = #table_name;
                     #index_setup
@@ -133,13 +232,23 @@ impl PersistGenerator {
                     ))
                 }
 
-                async fn load(engine: E) -> eyre::Result<Self> {
-                    let table_path = engine.config().table_path();
-                    if !std::path::Path::new(table_path).exists() {
+                async fn load(mut engine: E) -> eyre::Result<Self> {
+                    let schema = Self::space_info_default().inner;
+                    engine
+                        .validate_schema(
+                            schema.row_schema,
+                            schema.primary_key_fields,
+                            schema.secondary_index_types,
+                        )
+                        .await?;
+                    let table_path = engine.config().table_path().to_owned();
+                    if !std::path::Path::new(&table_path).exists() {
                         return Self::new(engine).await;
                     };
-                    let space = #space_ident::parse_file(table_path).await?;
-                    let table = space.into_worktable(engine).await;
+                    let space = #space_ident::parse_file(&table_path)
+                        .await
+                        .map_err(|error| PersistenceLoadError::corrupt(&table_path, error))?;
+                    let table = space.into_worktable(engine, &table_path).await?;
                     Ok(table)
                 }
             }