diff --git a/content/storage-engine/projects/catalog-manager/api-reference.md b/content/storage-engine/projects/catalog-manager/api-reference.md index 0f5834a..149ac12 100644 --- a/content/storage-engine/projects/catalog-manager/api-reference.md +++ b/content/storage-engine/projects/catalog-manager/api-reference.md @@ -78,19 +78,17 @@ pub fn load_catalog(bm: &mut BufferManager) -> Catalog ``` **Output:** -- Returns a `Catalog` struct populated with all databases, tables, columns, constraints, and index OIDs from the page backend. +- Returns a `Catalog` struct configured for lazy-loading. In-memory metadata maps are empty; data is loaded on-demand via the `CatalogCache`. **Implementation:** 1. If `catalog_pages/` exists, load from pages via `load_catalog_from_pages(bm)`. -2. On failure, return `Catalog::new()` (empty catalog). +2. On failure, return `Catalog::new()` (empty catalog shell). The page-based loader: -1. Initialises `OidCounter` from `pg_oid_counter.dat`. -2. Scans `pg_database` → populates `catalog.databases`. -3. Scans `pg_table` → attached to parent databases by `db_oid`. -4. Scans `pg_column` → attached to parent tables by `table_oid`, sorted by `column_position`. -5. Scans `pg_constraint` → attached to parent tables by `table_oid`. -6. Scans `pg_index` → index OIDs attached to parent tables by `table_oid`. +1. Initialises the `OidCounter` from `pg_oid_counter.dat`. +2. Sets `page_backend_active = true`. +3. Sets `oid_counter` to the value loaded from the counter file. +4. Returns the `Catalog` shell. No system catalogs are scanned at this stage (lazy loading). --- @@ -162,7 +160,7 @@ pub fn create_database( 2. Allocate a new OID via `catalog.alloc_oid()`. 3. Create the `database/base/{db_name}/` directory. 4. Serialise and insert a record into `pg_database`. -5. Add the `Database` struct to the in-memory catalog. +5. Insert the `Database` struct into `catalog.cache`. 6. Invalidate the database cache entry. --- @@ -183,11 +181,11 @@ pub fn drop_database( ``` **Implementation:** -1. Resolve `db_oid` from the in-memory catalog. +1. Resolve `db_oid` via `get_database()`, which checks the cache then scans `pg_database`. 2. Drop all tables belonging to this database via `drop_table()`. 3. Find and delete the database record from `pg_database`. 4. Remove the database directory from disk. -5. Remove from in-memory catalog and invalidate cache. +5. Invalidate the database cache entry. --- @@ -244,7 +242,7 @@ pub fn create_table( 4. Serialise and insert column records into `pg_column`. 5. Create the table data file (`{db_name}/{table_name}.dat`) and initialise it. 6. Serialise and insert a record into `pg_table`. -7. Add the `Table` to the in-memory catalog and invalidate cache. +7. Insert the `Table` into `catalog.cache` and invalidate the table cache entry. 8. Process each constraint definition (PK, FK, UNIQUE, NOT NULL) via the respective constraint functions. --- @@ -265,12 +263,12 @@ pub fn drop_table( ``` **Implementation:** -1. Check for foreign key dependencies from other tables — return `ForeignKeyDependency` error if found. +1. Check for foreign key dependencies from other tables by scanning `pg_constraint`. 2. Drop all indexes on this table via `drop_index()`. -3. Locate the table's database name and table name. +3. Locate the table's database name and table name via `get_table()` and a scan of `pg_database`. 4. Remove the table data file from disk. 5. Delete the record from `pg_table`. -6. Remove from in-memory catalog and invalidate all related cache entries. +6. Invalidate all related cache entries (table, constraints, indexes). --- @@ -307,7 +305,7 @@ Display all tables in a database from the page-based catalog with statistics. **Function:** ```rust pub fn show_tables( - catalog: &Catalog, + catalog: &mut Catalog, pm: &mut CatalogPageManager, bm: &mut BufferManager, db_name: &str, @@ -327,8 +325,8 @@ Retrieve complete table metadata including resolved columns, constraints, and in **Function:** ```rust pub fn get_table_metadata( - catalog: &Catalog, - pm: &CatalogPageManager, + catalog: &mut Catalog, + pm: &mut CatalogPageManager, bm: &mut BufferManager, db_name: &str, table_name: &str, @@ -497,7 +495,7 @@ pub fn create_index( 2. Generate an index name if not provided (`idx_{table_oid}_{col_oids}`). 3. Create the indexes directory and `.idx` file with an initialised B-Tree root page. 4. Allocate an `index_oid` and persist the index record to `pg_index`. -5. Add the index OID to the table's in-memory `indexes` list. +5. Invalidate the index cache for the table. --- @@ -600,6 +598,211 @@ pub fn scan_catalog( Returns all live tuples from the catalog (skips logically deleted slots with `length == 0`). +### `find_catalog_tuple` + +```rust +pub fn find_catalog_tuple( + &self, bm: &mut BufferManager, catalog_name: &str, predicate: F, +) -> Result<(u32, u32, Vec), CatalogError> +where + F: Fn(&[u8]) -> bool, +``` + +Scans the catalog until the predicate function returns `true` for a tuple. Returns `(page_num, slot_id, raw_bytes)` of the matching tuple, or `CatalogError::NotFound`. + +### `delete_catalog_tuple` + +```rust +pub fn delete_catalog_tuple( + &mut self, bm: &mut BufferManager, catalog_name: &str, page_num: u32, slot_id: u32, +) -> Result<(), CatalogError> +``` + +Marks a tuple as logically deleted by zeroing its slot's length field. The slot remains on the page but is skipped during scans. + +--- + +## OID Management + +### `alloc_oid` + +**Description:** +Allocate a unique 32-bit OID for a new database object. + +**Function:** +```rust +pub fn alloc_oid(&mut self) -> u32 +``` + +**Behavior:** +- Increments the internal OID counter. +- When `page_backend_active == true`, writes the new counter value to `pg_oid_counter.dat` immediately. +- In JSON legacy mode, the counter is implicitly captured in `catalog.json`. +- **Crash-safe:** OID is persisted to disk before returning. + +--- + +## Cache Management + +### `invalidate_database` + +**Description:** +Invalidate cache entries for a database by name. + +**Function:** +```rust +pub fn invalidate_database(&mut self, db_name: &str) +``` + +**Usage:** Called after `create_database`, `drop_database`, and `alter_table_add_column`. + +### `invalidate_table` + +**Description:** +Invalidate cache entries for a specific table. + +**Function:** +```rust +pub fn invalidate_table(&mut self, db_oid: u32, table_name: &str) +``` + +### `invalidate_constraints` + +**Description:** +Invalidate all constraints associated with a table. + +**Function:** +```rust +pub fn invalidate_constraints(&mut self, table_oid: u32) +``` + +### `invalidate_indexes` + +**Description:** +Invalidate all indexes associated with a table. + +**Function:** +```rust +pub fn invalidate_indexes(&mut self, table_oid: u32) +``` + +### `invalidate_all` + +**Description:** +Clear all cache entries. Used sparingly (after major catalog restructuring). + +**Function:** +```rust +pub fn invalidate_all(&mut self) +``` + +--- + +## Lookup Functions + +### `get_database` + +**Description:** +Retrieve a database by name via cache lookup or disk scan. + +**Function:** +```rust +pub fn get_database( + catalog: &Catalog, + pm: &CatalogPageManager, + bm: &mut BufferManager, + db_name: &str, +) -> Result +``` + +**Behavior:** +1. Check `catalog.cache` for the database. +2. On cache miss, scan `pg_database` and return the first match. + +### `get_table` + +**Description:** +Retrieve a table by name within a database. + +**Function:** +```rust +pub fn get_table( + catalog: &mut Catalog, + pm: &mut CatalogPageManager, + bm: &mut BufferManager, + db_oid: u32, + table_name: &str, +) -> Result +``` + +### `get_columns` + +**Description:** +Retrieve all columns for a table, sorted by position. + +**Function:** +```rust +pub fn get_columns( + pm: &CatalogPageManager, + bm: &mut BufferManager, + table_oid: u32, +) -> Result, CatalogError> +``` + +--- + +## Error Handling + +All public functions return `Result`. The `CatalogError` enum provides detailed context for failures: + +| Error | Meaning | +|-------|---------| +| `DatabaseNotFound(String)` | Database with given name does not exist | +| `DatabaseAlreadyExists(String)` | Database name is in use | +| `TableNotFound(String)` | Table not found in the database | +| `TableAlreadyExists(String)` | Table name already used in this database | +| `ColumnNotFound(String)` | Column not found in table | +| `TypeNotFound(String)` | Type name does not match any registered type | +| `AlreadyHasPrimaryKey` | Table already has a PRIMARY KEY constraint | +| `ForeignKeyDependency(String)` | Cannot drop table; foreign key references exist | +| `ColumnCountMismatch` | Column count in constraint doesn't match referenced table | +| `InvalidOperation(String)` | Operation not allowed in current state (e.g., NOT NULL column without default) | + +--- + +## Example: Complete DDL Sequence + +```rust +// Initialize catalog at startup +init_catalog(&mut bm); +let mut catalog = load_catalog(&mut bm); +let mut pm = init_catalog_page_storage()?; + +// Create a database +let db_oid = create_database( + &mut catalog, &mut pm, &mut bm, + "myapp", "appuser", Encoding::UTF8 +)?; + +// Create a table with columns and constraints +let table_oid = create_table( + &mut catalog, &mut pm, &mut bm, + "myapp", "users", + vec![ + ColumnDefinition { name: "id".into(), type_name: "INT".into(), is_nullable: false, .. }, + ColumnDefinition { name: "email".into(), type_name: "VARCHAR(255)".into(), is_nullable: false, .. }, + ], + vec![ + ConstraintDefinition::PrimaryKey { columns: vec!["id".into()], name: None }, + ConstraintDefinition::Unique { columns: vec!["email".into()], name: None }, + ] +)?; + +// Retrieve and display table metadata +let metadata = get_table_metadata(&mut catalog, &mut pm, &mut bm, "myapp", "users")?; +println!("Table: {} with {} columns", metadata.table_name, metadata.columns.len()); +``` + ### `delete_catalog_tuple` ```rust diff --git a/content/storage-engine/projects/catalog-manager/architecture.md b/content/storage-engine/projects/catalog-manager/architecture.md index 52c96f8..e0afbd0 100644 --- a/content/storage-engine/projects/catalog-manager/architecture.md +++ b/content/storage-engine/projects/catalog-manager/architecture.md @@ -22,7 +22,7 @@ database/ │ │ ├── pg_index.dat # System catalog: indexes │ │ └── pg_type.dat # System catalog: data types │ ├── pg_oid_counter.dat # Persistent OID counter -│ └── catalog.json # DEPRECATED: Legacy format +│ └── catalog.json # Unused legacy file (can be removed) └── base/ └── {database}/ ├── {table}.dat # User table data files @@ -111,26 +111,194 @@ pub fn alloc_oid(&mut self) -> u32 { --- -## Catalog Cache +## Core Components + +The Catalog Manager consists of eight core modules (~3,000 lines of Rust) orchestrated by the main `Catalog` struct. Each module handles a specific aspect of metadata management. + +### 1. Type System (`types.rs`, 485 lines) + +Defines all data structures and enums used throughout the catalog. Key types: + +- **`DataType`** — Represents a single data type (OID, name, category, alignment, length) + - 10 built-in types with PostgreSQL-like OID ranges (1-10 for built-ins, 10,000+ for user types) + - Methods: `from_name()` for type resolution, alias support (INTEGER → INT, REAL → FLOAT, BYTEA → BYTES) + +- **`Column`** — Represents a column within a table + - Stores: OID, name, position, type, modifiers, nullable flag, default value, constraint OIDs + - Position is 1-based and immutable after column creation + +- **`Constraint`** — Represents a constraint with type-specific metadata + - **PrimaryKey**: Backed by unique index (stores `index_oid`) + - **ForeignKey**: Stores referenced table/columns and ON DELETE/UPDATE actions (Cascade, SetNull, Restrict, NoAction) + - **Unique**: Backed by unique index + - **NotNull**: Simple flag on column + - **Check**: Stores expression string for future validation + +- **`Index`** — Represents a B-Tree index + - Supports multi-column indexes + - Tracks unique and primary key flags + - Stores page count for statistics + +### 2. Binary Serialization (`serialize.rs`, 416 lines) + +Converts catalog structs to/from compact little-endian binary format for disk storage. Each tuple type (Database, Table, Column, Constraint, Index, Type) has a dedicated serializer. + +**Serialization format:** +- Fixed types: u32/u64/u16/i16/u8 in LE byte order +- Strings: `[u16 length (LE)] [UTF-8 bytes]` +- Arrays: `[u16 count (LE)] [N × element]` +- Enums: Converted to u8 via `to_u8()`/`from_u8()` + +Example — Database tuple: +``` +[db_oid:4][owner_len:2][owner:N][name_len:2][name:N][created_at:8][encoding:1] +``` + +This module ensures that complex Rust structs can be transparently stored in variable-length slots on catalog pages. + +### 3. Page Manager (`page_manager.rs`, 289 lines) + +Low-level CRUD interface for system catalogs, delegating all page I/O to the Buffer Manager. + +**Key methods:** + +- **`insert_catalog_tuple(bm, catalog_name, bytes) → (page, slot)`** + - Finds page with free space or allocates new page + - Appends tuple to page + - Returns exact slot ID computed from the page's `lower` pointer + +- **`scan_catalog(bm, catalog_name) → Vec>`** + - Iterates all pages in a catalog file + - Collects live tuples (skips deleted ones with length=0) + - Triggers one page read per page in the catalog + +- **`delete_catalog_tuple(bm, page, slot)`** + - Marks slot as deleted by zeroing its length field + - Does NOT compact the page (deferred via future `vacuum_catalog`) + +- **`find_catalog_tuple(bm, catalog_name, predicate) → (page, slot, bytes)`** + - Scans pages until predicate returns true + - Returns raw bytes without deserializing (efficient for lookups) + +- **`update_catalog_tuple(bm, page, slot, new_bytes) → (new_page, new_slot)`** + - Uses **delete-then-reinsert** strategy for variable-length updates + - Returns new location so callers update their cache + +### 4. OID System (`oid.rs`, 110 lines) + +Manages globally unique 32-bit Object Identifiers with crash-safe persistence. + +- Stores next OID as little-endian u32 in `pg_oid_counter.dat` +- Built-in types: OIDs 1–10 +- User objects: Start at OID 10,000 (`USER_OID_START`) +- **Critical:** Every `allocate_oid()` writes immediately to disk when `page_backend_active == true` + - Prevents OID reuse after crashes + - In legacy mode (no page backend), the counter is not persisted by `alloc_oid()` + +### 5. In-Memory Cache (`cache.rs`, 236 lines) + +LRU cache reducing disk I/O for frequently accessed metadata. Default capacity: 256 entries. + +**Cache entries:** +- `databases`: HashMap of `Database` structs by name +- `tables`: HashMap of `Table` structs by `(db_oid, table_name)` tuple +- `constraints`: HashMap of constraint vectors by `table_oid` +- `indexes`: HashMap of index vectors by `table_oid` +- `types`: HashMap of `DataType` structs by OID + +**Eviction strategy:** +- LRU: Oldest accessed entry evicted when capacity exceeded +- `access_order` vector tracks access ordering + +**Invalidation policy:** +- **Eager invalidation on every DDL operation** +- `create_table()` calls `cache.insert_table()` after creation +- `drop_table()` calls `invalidate_table()`, `invalidate_constraints()`, `invalidate_indexes()` +- `add_*_constraint()` and `drop_index()` call `invalidate_constraints()` / `invalidate_indexes()` +- Ensures stale data is never served to queries + +### 6. Constraint System (`constraints.rs`, 411 lines) + +High-level constraint management with enforcement infrastructure. + +**Constraint creation functions:** +- **`add_primary_key_constraint()`**: Validates no existing PK, sets columns NOT NULL, creates backing index, persists to pg_constraint +- **`add_foreign_key_constraint()`**: Validates referenced columns are PK/UNIQUE, stores with cascading actions +- **`add_unique_constraint()`**: Creates backing index, persists metadata +- **`add_not_null_constraint()`**: Updates column `is_nullable` field via delete-then-reinsert in pg_column, persists a NOT NULL constraint record to pg_constraint + +**Constraint validation:** +- **`validate_constraints(table_oid, values)`** — Runtime validation during INSERT/UPDATE + - NOT NULL: Checks values present + - UNIQUE: B-Tree index lookup (no duplicates) + - FK: Verifies referenced row exists (when enforcement enabled) + - CHECK: Expression evaluation (future) + +### 7. Index Operations (`indexes.rs`, 440 lines) + +B-Tree index management for constraint enforcement and query optimization. + +**Index operations:** +- **`create_index()`** + - Allocates index OID + - Creates B-Tree file: `database/base/{db}/{index_name}.idx` + - Writes initial root page + - Persists metadata to pg_index + - Associates with table + +- **`drop_index()`** + - Uses `find_catalog_tuple()` to get real `(page, slot)` (not fabricated values) + - Deletes tuple from pg_index + - Removes `.idx` file + +- **`index_lookup(bm, db_name, index_name, key_bytes) → bool`** + - B-Tree traversal: root → leaf → binary search + - Returns boolean (used by UNIQUE and FK constraint validation) + +- **`insert_index_entry(bm, db_name, index_name, key_bytes, page_num, slot_id)`** + - Called after INSERT passes constraints + - Updates B-Tree structure + +### 8. Core Orchestration (`catalog.rs`, 615 lines) + +High-level API coordinating all components. Main struct `Catalog` holds: + +- `oid_counter`: Next OID to allocate +- `page_backend_active`: Boolean flag indicating page backend is active +- `bootstrap_mode`: Flag for initialization phase +- `cache`: The LRU `CatalogCache` instance + +Databases and tables are **not stored in the `Catalog` struct**; they are stored in the system catalog pages and accessed on-demand via the cache. + +**Public operations:** +- **`create_database()`**: Allocate OID, create directory, insert to pg_database, invalidate cache +- **`drop_database()`**: Drop all tables, delete from pg_database, remove directory +- **`create_table()`**: Allocate table OID, create columns, apply constraints, initialize data file +- **`drop_table()`**: Check FK dependencies, drop indexes, delete constraints/columns, remove file +- **`alter_table_add_column()`**: Allocate column OID, insert to pg_column, return new OID + +--- + +## Catalog Cache & Lazy Loading + +The RookDB Catalog Manager employs a **lazy-loading** strategy to manage metadata. Unlike the legacy system, which loaded all metadata into memory at startup, the new system only loads metadata when it is explicitly requested (e.g., when opening a database or querying a table's schema). + +### Catalog Cache -The in-memory **LRU Catalog Cache** reduces disk I/O for frequently accessed metadata: +The in-memory **LRU Catalog Cache** is the central component of this strategy, reducing disk I/O for frequently accessed metadata: -### Cache Entries +- **Max size:** 256 entries (configurable). +- **Eviction:** LRU (Least Recently Used) — when capacity is reached, the oldest entry is removed. +- **Invalidation:** Every DDL operation (CREATE, ALTER, DROP) eagerly invalidates affected cache entries. +- **Lazy Population:** The cache is populated on-demand. If a request for `get_table()` misses the cache, the system scans the `pg_table` catalog on disk, populates the cache, and returns the entry. -| Entry Type | Key | Value | -|------------|-----|-------| -| Database | `db_name` | `Database` struct | -| Table | `(db_oid, table_name)` | `Table` struct | -| Constraints | `table_oid` | `Vec` | -| Indexes | `table_oid` | `Vec` | -| Types | `type_oid` | `DataType` struct | +### Decoupled Entity Architecture -### Cache Policy +To support lazy loading and scalability, the in-memory data structures are **normalized and decoupled**. -- **Max size:** 256 entries (configurable) -- **Eviction:** LRU (Least Recently Used) — when capacity is reached, the oldest entry is removed -- **Invalidation:** Every DDL operation (CREATE, ALTER, DROP) eagerly invalidates affected cache entries -- **Write-through:** Changes are always persisted to pages first; the cache is populated lazily on reads +- **No Nesting:** A `Database` struct does not contain a list of `Table` objects, and a `Table` struct does not contain its `Column` objects. +- **OID Linking:** Entities refer to each other via **OIDs**. For example, a `Table` record stores its parent `db_oid`. +- **Runtime Resolution:** When a full view of a table is needed (e.g., for query planning), the `get_table_metadata()` function orchestrates the resolution by looking up the table, its columns, its constraints, and its indexes as separate entities from the cache or disk. ### Invalidation Points @@ -146,12 +314,7 @@ The in-memory **LRU Catalog Cache** reduces disk I/O for frequently accessed met --- -## Dual-Mode Initialization - -The catalog system supports two storage backends for migration compatibility: - -1. **Page mode** — page-based storage under `database/global/catalog_pages/` -2. **Legacy mode** — JSON-based `database/global/catalog.json` +## Initialization ### Startup Flow @@ -159,7 +322,7 @@ The catalog system supports two storage backends for migration compatibility: init_catalog(bm) │ ├── catalog_pages/ exists? - │ └── YES → Page backend detected (load from pages) + │ └── YES → Load page backend (lazy; no catalog scan at startup) │ └── NO → Bootstrap ├── Create catalog_pages/ directory @@ -177,3 +340,183 @@ On a fresh install, `bootstrap_catalog()`: 3. Creates all six system catalog `.dat` files using `init_table()` 4. Registers all 10 built-in data types into `pg_type` 5. Inserts the system database record (`db_oid=1`, `name="system"`) into `pg_database` + +--- + +## Complete End-to-End Example: CREATE TABLE with PRIMARY KEY + +This example traces a complete DDL sequence through the catalog layers, demonstrating how components interact from SQL parsing to persistent storage. + +**SQL Statement:** +```sql +CREATE TABLE users ( + id INT PRIMARY KEY, + name TEXT NOT NULL +); +``` + +### Execution Flow + +``` +1. Frontend parses DDL → ColumnDefinitions & ConstraintDefinitions + + ColumnDefinitions: + ├── {name: "id", type: "INT", nullable: false} + └── {name: "name", type: "TEXT", nullable: false} + + ConstraintDefinitions: + ├── PrimaryKey(["id"]) + └── NotNull("name") + +2. catalog.rs::create_table() called + └─→ Validates database exists, table name unique + +3. Allocate OIDs: + ├── table_oid = catalog.alloc_oid() = 10000 + │ └─→ Persists to pg_oid_counter.dat + │ + ├── column_oid (id) = catalog.alloc_oid() = 10001 + │ └─→ Persists to pg_oid_counter.dat + │ + └── column_oid (name) = catalog.alloc_oid() = 10002 + └─→ Persists to pg_oid_counter.dat + +4. Process Columns: + + a) Column "id" (INT): + ├── DataType::from_name("INT") → DataType { oid: 1, category: Numeric, length: 4, align: 4 } + ├── serialize.rs::serialize_column_tuple() + │ └─→ [col_oid:4][table_oid:4][name_var][pos:2][type_oid:4][...] + │ └─→ (52 bytes) → raw_bytes_1 + │ + └── page_manager.rs::insert_catalog_tuple(bm, "pg_column", raw_bytes_1) + ├─→ Find page with free space in pg_column.dat (or create page) + ├─→ Append to page (update upper pointer, add slot) + └─→ Return (page=1, slot=0) + + b) Column "name" (TEXT): + ├── DataType::from_name("TEXT") → DataType { oid: 6, category: String, length: -1, align: 1 } + ├── serialize_column_tuple() + │ └─→ (48 bytes) → raw_bytes_2 + │ + └── insert_catalog_tuple(bm, "pg_column", raw_bytes_2) + └─→ Return (page=1, slot=1) + +5. Create Table File: + ├── Create: database/base/{db}/users.dat + └── init_table() via Buffer Manager + ├─→ Write page 0 (header with page_count=2) + └─→ Write page 1 (empty slotted page) + +6. Insert Table Metadata to pg_table: + ├── serialize_table_tuple() + │ └─→ [table_oid:4][name_var][db_oid:4][table_type:1][row_count:8][page_count:4][...] + │ └─→ (45 bytes) → raw_bytes_table + │ + └── insert_catalog_tuple(bm, "pg_table", raw_bytes_table) + ├─→ Find or create page + └─→ Return (page=1, slot=2) + +7. In-memory Cache Update: + └─→ catalog.cache.insert_table( + Table { + table_oid: 10000, + table_name: "users", + db_oid: , + columns: [Column{oid:10001, name:"id", ...}, Column{oid:10002, name:"name", ...}] + } + ) + +8. Process Constraint #1 — PrimaryKey(["id"]): + ├── constraints.rs::add_primary_key_constraint() + │ ├─→ Resolve "id" → column_oid=10001 + │ ├─→ Set column is_nullable = false + │ │ └─→ pg_column update: delete-then-reinsert (new page/slot) + │ │ + │ └─→ indexes.rs::create_index(unique=true, primary=true) + │ ├─→ index_oid = catalog.alloc_oid() = 10003 + │ │ └─→ Persists to pg_oid_counter.dat + │ │ + │ ├─→ Create: database/base/{db}/indexes/pk_users_id.idx + │ │ └─→ B-Tree root page initialized + │ │ + │ ├─→ serialize_index_tuple() + │ │ └─→ [index_oid:4][name_var][table_oid:4][type:1][cols_var][unique:1][primary:1][pages:4] + │ │ └─→ (42 bytes) → raw_bytes_idx + │ │ + │ └─→ insert_catalog_tuple(bm, "pg_index", raw_bytes_idx) + │ └─→ Return (page=1, slot=0) + │ + └─→ serialize_constraint_tuple() + └─→ [constr_oid:4][name_var][type:1][table_oid:4][cols_var][pk_index_oid:4] + └─→ (38 bytes) → raw_bytes_constr + + insert_catalog_tuple(bm, "pg_constraint", raw_bytes_constr) + └─→ Return (page=1, slot=0) + +9. Process Constraint #2 — NotNull("name"): + └─→ constraints.rs::add_not_null_constraint() + ├─→ Set column is_nullable = false (already false, no-op) + └─→ Serialize and insert to pg_constraint (similar to above) + +10. Cache Invalidation: + └─→ catalog.cache.invalidate_database(db_name) + └─→ Clears entry from cache (will reload on next access) + +11. Return Result: + └─→ Ok(table_oid=10000) +``` + +### Disk State After CREATE TABLE + +``` +database/global/catalog_pages/ +├── pg_database.dat +│ ├── Page 0: [page_count=2][reserved] +│ └── Page 1: (no new database entries) +│ +├── pg_table.dat +│ ├── Page 0: [page_count=2][reserved] +│ └── Page 1: Slotted page with 1 tuple (users table metadata) +│ +├── pg_column.dat +│ ├── Page 0: [page_count=2][reserved] +│ └── Page 1: Slotted page with 2 tuples (id column, name column) +│ +├── pg_constraint.dat +│ ├── Page 0: [page_count=2][reserved] +│ └── Page 1: Slotted page with 2 tuples (PK constraint, NOT NULL constraint) +│ +├── pg_index.dat +│ ├── Page 0: [page_count=2][reserved] +│ └── Page 1: Slotted page with 1 tuple (pk_users_id index metadata) +│ +├── pg_type.dat +│ └── (unchanged, contains 10 built-in types) +│ +└── pg_oid_counter.dat + └── [00 00 00 0A] (10003 in LE = next OID to allocate) + +database/base/{db}/ +├── users.dat +│ ├── Page 0: [page_count=2][reserved] +│ └── Page 1: Empty slotted page (no rows yet) +│ +└── indexes/ + └── pk_users_id.idx + └── Page 0: B-Tree root page (leaf, no keys yet) +``` + +### Key Observations + +1. **OID Persistence**: Every `alloc_oid()` writes immediately to `pg_oid_counter.dat`, ensuring crash-safety. + +2. **Cascading Invalidation**: `create_table()` invalidates the database cache, forcing a reload on next access. + +3. **Constraint-Index Coupling**: PRIMARY KEY and UNIQUE constraints automatically create backing B-Tree indexes, stored separately in `database/base/{db}/indexes/`. + +4. **Variable-Length Encoding**: Names, column types, and expressions use `[u16 len][bytes]` format, enabling variable-length catalog tuples. + +5. **Buffer Manager Integration**: All six catalog page operations (pg_table, pg_column, pg_constraint, pg_index) use identical `pin_page`/`unpin_page` semantics. + +6. **Lazy Disk Reads**: Catalog tuples are only deserialized when explicitly requested (e.g., `get_table_metadata`). Disk scans happen at access time, not bootstrap. diff --git a/content/storage-engine/projects/catalog-manager/data-structures.md b/content/storage-engine/projects/catalog-manager/data-structures.md index d100768..b2e1a08 100644 --- a/content/storage-engine/projects/catalog-manager/data-structures.md +++ b/content/storage-engine/projects/catalog-manager/data-structures.md @@ -70,7 +70,28 @@ pub enum DefaultValue { --- -## 2. Column +## Built-In Types Table + +RookDB provides 10 built-in types aligned with PostgreSQL conventions. Each type has a unique OID and metadata: + +| OID | Name | Category | Length | Align | Aliases | +|-----|------|----------|--------|-------|---------| +| 1 | INT | Numeric | 4 | 4 | INTEGER, INT32 | +| 2 | BIGINT | Numeric | 8 | 8 | INT64 | +| 3 | FLOAT | Numeric | 4 | 4 | REAL, FLOAT32 | +| 4 | DOUBLE | Numeric | 8 | 8 | FLOAT64 | +| 5 | BOOL | Boolean | 1 | 1 | BOOLEAN | +| 6 | TEXT | String | -1 | 1 | STRING | +| 7 | VARCHAR | String | -1 | 1 | VARCHAR(n) | +| 8 | DATE | DateTime | 4 | 4 | — | +| 9 | TIMESTAMP | DateTime | 8 | 8 | — | +| 10 | BYTES | Binary | -1 | 1 | BYTEA, BLOB | + +**Note:** `Length = -1` indicates variable-length types stored with a 2-byte length prefix on disk. + +The type system is case-insensitive and supports aliases via `DataType::from_name()`. + +--- ### `Column` @@ -244,9 +265,6 @@ pub struct Table { pub table_oid: u32, pub table_name: String, pub db_oid: u32, - pub columns: Vec, - pub constraints: Vec, - pub indexes: Vec, // OIDs of indexes on this table pub table_type: TableType, pub statistics: TableStatistics, } @@ -302,22 +320,12 @@ A database entry, mirroring `pg_database`. pub struct Database { pub db_oid: u32, pub db_name: String, - pub tables: HashMap, pub owner: String, pub encoding: Encoding, pub created_at: u64, } ``` -### `Encoding` - -```rust -pub enum Encoding { - UTF8, // 1 - ASCII, // 2 -} -``` - --- ## 7. Catalog @@ -328,7 +336,6 @@ Top-level catalog: databases in memory plus infrastructure fields. ```rust pub struct Catalog { - pub databases: HashMap, pub oid_counter: u32, pub bootstrap_mode: bool, pub page_backend_active: bool, @@ -381,3 +388,5 @@ The catalog data structures are defined across the following files: | `indexes.rs` | Index creation, deletion, and B-Tree operations | | `catalog.rs` | High-level catalog operations (init, load, create/drop DB/table) | | `mod.rs` | Module declarations and re-exports | + + diff --git a/content/storage-engine/projects/catalog-manager/overview.md b/content/storage-engine/projects/catalog-manager/overview.md index c02ca6b..4018972 100644 --- a/content/storage-engine/projects/catalog-manager/overview.md +++ b/content/storage-engine/projects/catalog-manager/overview.md @@ -21,11 +21,12 @@ The Catalog Manager project addresses these limitations by introducing: 1. **Page-based catalog storage** — system catalogs are stored as slotted pages (identical format to user tables), enabling integration with the buffer manager and supporting large-scale metadata. 2. **Self-hosting architecture** — system catalog tables describe themselves, following PostgreSQL's proven design. -3. **OID system** — every database object (database, table, column, constraint, index, type) receives a persistent, unique 32-bit Object Identifier. -4. **Constraint system** — full support for PRIMARY KEY, FOREIGN KEY (with cascading actions), UNIQUE, NOT NULL, and CHECK constraints. -5. **Extended type system** — ten built-in types (INT, BIGINT, FLOAT, DOUBLE, BOOL, TEXT, VARCHAR, DATE, TIMESTAMP, BYTES) with alignment and length metadata. -6. **In-memory LRU cache** — reduces redundant page reads with automatic invalidation on every DDL operation. -7. **Dual-mode compatibility** — the system gracefully handles both the legacy JSON format and the new page-based format during migration. +3. **Normalized metadata** — database objects (databases, tables, columns, etc.) are decoupled in memory and storage, linked via OIDs for improved scalability. +4. **Lazy-loading backend** — metadata is loaded on-demand from disk and cached, rather than eager-loading the entire catalog into memory at startup. +5. **OID system** — every database object receives a persistent, unique 32-bit Object Identifier. +6. **Constraint system** — full support for PRIMARY KEY, FOREIGN KEY (with cascading actions), UNIQUE, NOT NULL, and CHECK constraints. +7. **Extended type system** — ten built-in types (INT, BIGINT, FLOAT, DOUBLE, BOOL, TEXT, VARCHAR, DATE, TIMESTAMP, BYTES) with alignment and length metadata. +8. **In-memory LRU cache** — reduces redundant page reads with automatic invalidation on every DDL operation. ## Design Principles @@ -33,7 +34,8 @@ The design is guided by the following principles: - **Consistency with RookDB internals** — catalog pages use the same 8 KB slotted-page layout as user tables, reusing existing page and disk infrastructure. - **PostgreSQL conventions** — system catalog naming (`pg_database`, `pg_table`, etc.), OID-based references, and constraint semantics follow PostgreSQL precedents. -- **Separation of concerns** — the catalog module is cleanly separated into sub-modules: types, serialization, page management, constraints, indexes, cache, and OID management. +- **Decoupled Entities** — metadata is stored in a normalized form. In-memory structs (`Table`, `Database`) do not hold nested collections; instead, relationships are resolved at runtime via the `Catalog` orchestration layer. +- **Lazy Load, Eager Invalidate** — data is loaded lazily into the cache to minimize startup time and memory footprint, but invalidated eagerly on DDL changes to ensure consistency. - **Write-through durability** — DDL changes are persisted immediately to the page backend; the OID counter is written to disk on every allocation to prevent reuse after a restart. ## Scope @@ -47,4 +49,49 @@ The Catalog Manager project modifies or creates files across the following areas | Executor (`src/backend/executor/`) | `load_csv.rs`, `seq_scan.rs` | | Frontend (`src/frontend/`) | `menu.rs`, `database_cmd.rs`, `table_cmd.rs`, `data_cmd.rs` | | Layout (`src/backend/`) | `layout.rs` | -| Tests (`tests/`) | `test_init_catalog.rs`, `test_load_catalog.rs`, `test_save_catalog.rs` | +| Tests (`tests/`) | `test_catalog_bootstrap.rs`, `test_catalog_cache.rs`, `test_catalog_operations.rs`, `test_constraints.rs`, `test_indexes.rs`, `test_serialization.rs`, `test_type_system.rs` | + +--- + +## Implementation + +The final implementation comprises **~3,000 lines of Rust** organized into **8 core modules**: + +| Module | Lines | Purpose | +|--------|-------|---------| +| `types.rs` | 485 | Data structures: types, columns, constraints, indexes, tables, databases, errors | +| `serialize.rs` | 416 | Binary serialization/deserialization for all catalog tuple types | +| `page_manager.rs` | 289 | Low-level CRUD on system catalog pages via Buffer Manager | +| `oid.rs` | 110 | Persistent OID allocation with crash-safety | +| `cache.rs` | 236 | LRU in-memory cache with DDL-triggered invalidation | +| `constraints.rs` | 411 | Constraint creation, validation, and enforcement | +| `indexes.rs` | 440 | B-Tree index management and lookup | +| `catalog.rs` | 615 | High-level API for catalog initialization and operations | +| **Total** | **~3,000** | — | + +### Capabilities + +- **6 system catalogs** stored using the same 8KB slotted-page format as user tables +- **10 built-in types** with PostgreSQL-compatible OID allocation (1–10 for built-ins, 10,000+ for user objects) +- **5 constraint types** — PRIMARY KEY, FOREIGN KEY, UNIQUE, NOT NULL, and CHECK — enforced via B-Tree indexes +- **Crash-safe OID allocation** with immediate persistence to `pg_oid_counter.dat` on every allocation +- **256-entry LRU cache** with DDL-triggered invalidation for consistent metadata reads +- **Lazy-loading** — metadata is loaded on-demand, minimising startup time and memory usage +- **Variable-length tuple support** — names, expressions, and type modifiers stored with length-prefixed encoding + +--- + +## Storage + +All metadata is stored in `database/global/catalog_pages/` as slotted-page `.dat` files. On first startup, RookDB bootstraps the page backend automatically — no manual setup is required. + +--- + +## Future Work + +- **CHECK constraint evaluation** — expression parser for runtime CHECK validation +- **Vacuum** — compact catalog pages by reclaiming logically deleted slots +- **Statistics collection** — table cardinality and column distributions for query planning +- **User-defined types** — extensible type system for composite and domain types +- **Partitioning** — metadata support for table partitions + diff --git a/sidebars.ts b/sidebars.ts index 46dd658..5480f25 100644 --- a/sidebars.ts +++ b/sidebars.ts @@ -61,11 +61,11 @@ const sidebars: SidebarsConfig = { "storage-engine/projects/catalog-manager/catalog-manager", "storage-engine/projects/catalog-manager/overview", "storage-engine/projects/catalog-manager/architecture", - "storage-engine/projects/catalog-manager/system-catalogs", "storage-engine/projects/catalog-manager/data-structures", + "storage-engine/projects/catalog-manager/system-catalogs", + "storage-engine/projects/catalog-manager/physical-storage", "storage-engine/projects/catalog-manager/api-reference", "storage-engine/projects/catalog-manager/implementation-notes", - "storage-engine/projects/catalog-manager/physical-storage", ], },