diff --git a/content/storage-engine/projects/indexing/bplustree.md b/content/storage-engine/projects/indexing/bplustree.md new file mode 100644 index 0000000..c5b081b --- /dev/null +++ b/content/storage-engine/projects/indexing/bplustree.md @@ -0,0 +1,184 @@ +--- +title: B+ Tree +sidebar_position: 8 +--- + +### Algorithm + +* B+ Tree with minimum degree `t` +* Internal nodes store only routing keys and child pointers +* Leaf nodes store `(key, Vec)` +* Leaves are linked using `next_leaf` pointer (singly linked forward) +* Nodes stored in an arena (`Vec`) with stable indices +* All leaves are at the same depth +* Node capacity: + + * Max keys: `2t - 1` + * Min keys (non-root): `t - 1` + +--- + +### Basic Idea + +* Search descends from root to leaf using binary partition (`partition_point`) +* Insert happens at leaf; overflow triggers split and upward propagation +* Delete removes `(key, rid)` and handles underflow via borrow/merge +* Range scan uses linked leaves after initial descent + +--- + +### Time Complexity + +* Search: `O(t · log_t n)` +* Insert: `O(t · log_t n)` +* Delete: `O(t · log_t n)` +* Range scan: `O(log n + k)` +* Index creation (bulk via inserts): same as repeated insert + +--- + +### Space Complexity + +* In-memory: + + * `O(n)` nodes stored in `Vec` +* On-disk: + + * One page per node (`PAGE_SIZE`) + * Total pages = number of nodes + 1 header page + +--- + +### Metadata File Storage Format + +* First page (header page): + + * Bytes 0–8: magic `"RDBIDXV1"` + * Bytes 8–10: version + * Bytes 10–12: header size + * Bytes 12–16: page size + * Bytes 16–20: root page + * Bytes 20–24: node page count + * Bytes 24–32: entry count + * Bytes 32–36: minimum degree `t` + +--- + +### Data Storage Format (Per Node Page) + +* Fixed-size page (`PAGE_SIZE`) +* Header (16 bytes): + + * Byte 0: `is_leaf` + * Byte 1: `dead` + * Bytes 2–4: key count + * Bytes 4–6: child count + * Bytes 8–12: `next_leaf_page` + * Bytes 12–16: payload size +* Payload: + + * Keys (encoded sequentially) + * If leaf: + + * For each key: + + * `rid_count` + * List of `(page_no, item_id)` + * If internal: + + * Child page numbers + +--- + +### Hashing / Modulo Function + +* Not used + +--- + +### Collision Handling + +* Multiple records per key handled via: + + * `values: Vec>` + * Same key maps to a vector of record IDs + +--- + +### Operations + +#### Search + +* Traverse from root using `partition_point` +* At leaf: + + * Binary position lookup + * Return `Vec` or empty + +#### Insert + +* Insert into leaf at sorted position +* If key exists: + + * Append `RecordId` if not duplicate +* If overflow (`>= 2t` keys): + + * Split leaf + * Push first key of right node upward + * Recursively split internal nodes if needed +* Root split creates new root + +#### Delete + +* Locate leaf and key +* Remove specific `RecordId` +* If key has no more records: + + * Remove key +* If underflow (`< t-1` keys): + + * Borrow from sibling OR merge +* Propagate fixes upward +* If root becomes empty: + + * Replace with single child + +#### Update + +* Not explicitly implemented +* Equivalent to: + + * `delete(key, old_rid)` + `insert(key, new_rid)` + +--- + +### Disk Operations + +* Save: + + * Header page + serialized node pages +* Load: + + * Parse header + * Load all node pages into memory +* Direct search: + + * `search_on_disk()` performs traversal without full load + +--- + +### Additional Notes + +* Node splitting: + + * Leaf: split at index `t` + * Internal: median key moves up +* Leaf linking maintained during split/merge +* Dead nodes marked but not reused +* Validation ensures: + + * Sorted keys + * Proper child counts + * No cycles in tree or leaf chain + +--- \ No newline at end of file diff --git a/content/storage-engine/projects/indexing/btree.md b/content/storage-engine/projects/indexing/btree.md new file mode 100644 index 0000000..9671107 --- /dev/null +++ b/content/storage-engine/projects/indexing/btree.md @@ -0,0 +1,185 @@ +--- +title: BTree +sidebar_position: 7 +--- + + +### Algorithm + +* Classic B-Tree (Knuth / CLRS definition). +* Minimum degree `t`. +* Each node: + + * Stores `keys: Vec` + * Stores `values: Vec>` (multiple RIDs per key) + * Stores `children: Vec` (indices into arena) + * `is_leaf`, `dead` flags +* Nodes stored in a flat arena: `Vec` +* Root tracked by index. +* All leaves at same depth. +* Internal node: `k` keys → `k+1` children. +* Node capacity: `[t−1, 2t−1]` keys (except root). + +--- + +### Basic Idea + +* Tree-based index storing keys in sorted order. +* Each key maps to one or more record IDs. +* Uses balanced multi-way tree. +* Insert splits full nodes. +* Delete uses merge/rotation (CLRS algorithm). +* Arena-based storage using indices instead of pointers. + +--- + +### Time Complexity + +(from code comments) + +| Operation | Complexity | +| ---------- | ------------------ | +| search | O(t · log_t n) | +| insert | O(t · log_t n) | +| delete | O(t · log_t n) | +| range_scan | O(t · log_t n + k) | + +--- + +### Space Complexity + +* Nodes stored in `Vec` (arena). +* Each node holds up to `2t−1` keys. +* Deleted nodes remain as tombstones (`dead = true`). +* Overall: **O(n)** space. + +--- + +### Metadata Storage Format + +* Entire structure serialized using `serde::{Serialize, Deserialize}`. +* Persistence: + + * `save()` → `paged_store::save_entries(...)` + * `load()` → `paged_store::load_entries_stream(...)` +* No explicit separate metadata file structure defined in code. +* Metadata implicitly: + + * `nodes` vector + * `root` index + * `t` + * `entry_count` + +--- + +### Data Storage Format + +* Stored as `(IndexKey, RecordId)` pairs via paged store. +* On load: + + * Entries streamed and inserted one-by-one. +* In-memory: + + * Keys stored in nodes + * Values stored as `Vec` per key +* Tree reconstructed from entries (no direct node serialization in load). + +--- + +### Hashing / Modulo Function + +* Not used. +* Tree-based index (ordered structure). + +--- + +### Collision Handling + +* Multiple records for same key stored as: + + * `values[i] = Vec` +* Duplicate keys not duplicated structurally. +* Record IDs appended if key exists. + +--- + +### Operations + +#### Search + +* Binary search using `partition_point`. +* If key found → return `values[pos]`. +* Else descend to correct child. +* Leaf → return empty. + +--- + +#### Insert + +* If root full → split. +* Use `insert_non_full`: + + * If key exists → append RID if not present. + * Else insert key + new RID. +* Child split done before descending if full. + +--- + +#### Delete + +* Two-step: + + 1. Remove specific `(key, rid)` from values. + 2. If values empty → remove key structurally. +* Structural deletion: + + * Uses CLRS algorithm: + + * Replace with predecessor/successor + * Rotate (left/right) + * Merge children +* Nodes merged → right node marked `dead`. + +--- + +#### Update + +* Not explicitly implemented. +* Equivalent to: + + * delete(old key, rid) + * insert(new key, rid) + +--- + +#### Range Scan + +* In-order traversal: + + * Visit children and keys in sorted order. + * Collect RIDs within `[start, end]`. + +--- + +#### Validation + +* Ensures: + + * Keys sorted + * Values match keys + * No empty RID lists + * Correct child count + * No cycles + * No reachable dead nodes + +--- + +#### Additional Notes + +* Arena-based node storage using indices. +* Tombstoned nodes not reclaimed. +* Supports: + + * `min_key()` + * `max_key()` + * full traversal (`all_entries`) diff --git a/content/storage-engine/projects/indexing/catalogintegeration.md b/content/storage-engine/projects/indexing/catalogintegeration.md new file mode 100644 index 0000000..fdaac4e --- /dev/null +++ b/content/storage-engine/projects/indexing/catalogintegeration.md @@ -0,0 +1,67 @@ +--- +title: Catalog Integration +sidebar_position: 12 +--- + + +## 1. Catalog Extension + +Each table in the catalog maintains index metadata alongside column definitions. + +* Indexes are stored per table and keyed by index name. +* Metadata is persisted and loaded at startup. + +### Index Metadata Includes + +* Indexed columns (supports multi-column indexes) +* Index algorithm (hash/tree variants) +* Uniqueness flag +* Clustered flag (only one allowed per table) +* Included columns (for covering indexes) +* Index file location (for physical storage) + +--- + +## 2. Lifecycle Integration + +### Creation + +When an index is created: + +1. Validate columns and constraints (existence, duplicates, clustered rules) +2. Add entry to catalog +3. Persist catalog to disk +4. Trigger index build from table data +5. Create and link physical index file + +### Deletion + +* Remove index metadata from catalog +* Delete or detach corresponding index file + +### Startup + +* Catalog is loaded from disk +* Index metadata is used to locate and initialize index structures +* Enables immediate query-time usage + +--- + +## 3. Query Integration + +* Query planner consults catalog to discover available indexes +* Selects appropriate index based on: + + * Indexed columns + * Algorithm type + * Query predicates +* Execution layer uses catalog metadata to route lookups to the correct index structure + +--- + +## 4. Design Principles + +* Catalog is the **single source of truth** for index metadata +* Physical index structures are **decoupled but referenced** +* Validation is enforced at catalog level before index creation +* Supports extensibility for new index types and configurations diff --git a/content/storage-engine/projects/indexing/chainhash.md b/content/storage-engine/projects/indexing/chainhash.md new file mode 100644 index 0000000..9061f84 --- /dev/null +++ b/content/storage-engine/projects/indexing/chainhash.md @@ -0,0 +1,185 @@ +--- +title: Chain Hash +sidebar_position: 4 +--- + + +### Algorithm + +The implementation is a **chained hash index** using: + +* Fixed number of buckets (`bucket_count`) +* Each bucket is a `Vec` +* Each `ChainEntry` contains: + + * `key: IndexKey` + * `records: Vec` + +--- + +### Basic Idea + +* Keys are hashed using `key.hash_code()` +* Bucket index is computed using modulo operation +* Each bucket stores a list of entries (separate chaining) +* Each key maps to multiple `RecordId`s + +--- + +### Time Complexity + +* **Index Creation (load)** + Uses `paged_store::load_entries_stream` and calls `insert` per entry + → **O(N × bucket_scan)** + +* **Insert** + + * Bucket lookup: O(1) + * Scan within bucket: O(k) + → **O(k)** + +* **Search** + + * Bucket lookup: O(1) + * Scan within bucket: O(k) + → **O(k)** + +* **Delete** + + * Bucket lookup: O(1) + * Scan + retain: O(k + r) + → **O(k + r)** + +Where: + +* `k` = number of entries in a bucket +* `r` = number of record IDs in an entry + +--- + +### Space Complexity + +* Buckets: `O(bucket_count)` +* Entries: `O(number_of_keys)` +* Record storage: `O(total_record_ids)` +* Total: **O(bucket_count + keys + records)** + +--- + +### Metadata File Storage Format + +* Uses `paged_store::save_entries` +* Data is written as a stream of `(IndexKey, RecordId)` pairs +* No explicit metadata structure in this code +* Serialization derives: + + * `Serialize`, `Deserialize` on structs + +--- + +### Data Storage Format + +* Internally: + + ``` + buckets: Vec> + ``` +* Flattened during save: + + ``` + Vec<(IndexKey, RecordId)> + ``` +* Each `(key, record_id)` pair stored independently + +--- + +### Hashing / Modulo Function + +``` +bucket_index = (key.hash_code() as usize) % bucket_count +``` + +--- + +### Collision Handling + +* **Separate chaining** +* Each bucket is a `Vec` +* Multiple keys in same bucket handled via linear scan + +--- + +### Operations + +#### Search + +* Compute bucket index +* Find entry with matching key +* Return cloned `records` +* If not found → empty vector + +--- + +#### Insert + +* Compute bucket index +* If key exists: + + * Add `record_id` if not already present +* Else: + + * Create new `ChainEntry` + * Append to bucket + +--- + +#### Delete + +* Compute bucket index +* Find matching entry +* Remove `record_id` using `retain` +* If no records remain: + + * Remove entire `ChainEntry` +* Returns `true` if deletion occurred + +--- + +#### Update + +* Not explicitly implemented +* Can be inferred as: + + * Delete old `(key, record_id)` + * Insert new `(key, record_id)` + +--- + +### Additional Functions + +* **save** + Writes all `(key, record_id)` pairs using paged store + +* **all_entries** + Flattens buckets into vector of pairs + +* **entry_count** + Total number of record IDs across all entries + +* **validate_structure** + + * `bucket_count > 0` + * `bucket_count == buckets.len()` + * No entry has empty `records` + +* **load_factor** + + ``` + entry_count / bucket_count + ``` + +* **index_type_name** + + ``` + "chained_hash" + ``` diff --git a/content/storage-engine/projects/indexing/clustered_index.md b/content/storage-engine/projects/indexing/clustered_index.md new file mode 100644 index 0000000..77b65bf --- /dev/null +++ b/content/storage-engine/projects/indexing/clustered_index.md @@ -0,0 +1,71 @@ +--- +title: Clustered Indexing +sidebar_position: 14 +--- + +### **1. Supported Algorithms** +The system implements an `AnyIndex` enum dispatcher that encapsulates two primary families of indexing: +* **Hash-Based:** `StaticHash`, `ChainedHash`, `ExtendibleHash`, `LinearHash`. +* **Tree-Based:** `BTree`, `BPlusTree`, `RadixTree`, `SkipList`, `LsmTree`. + +### **2. Design Rationale** +The architecture uses an **Enum Dispatcher** pattern. Since Rust traits cannot easily support static factory methods (like `load`) via trait objects, `AnyIndex` acts as a concrete wrapper. It forwards all `IndexTrait` calls (insert, search, delete) to the underlying specialized implementations. + +--- + +### **3. Complexity Analysis** +*Based on the implementation logic in `manager.rs` and `executor.rs`:* + +| Operation | Complexity (General) | Notes | +| :--- | :--- | :--- | +| **Index Creation** | $O(N \cdot M)$ | $N$ is the number of table pages, $M$ is the number of tuples per page. | +| **Index Update** | $O(\log N)$ or $O(1)$ | Depends on whether the variant is Tree-based (Log) or Hash-based (Amortized constant). | +| **Space Complexity** | $O(K \cdot R)$ | $K$ is key size, $R$ is the number of records. Stored in specialized `.idx` files. | + +--- + +### **4. Storage and Persistence** +#### **Metadata Storage** +* **Format:** JSON. +* **Location:** `CATALOG_FILE` (typically `catalog.json`). +* **Structure:** A nested hierarchy: `Catalog` $\rightarrow$ `Database` $\rightarrow$ `Table` $\rightarrow$ `IndexEntry`. +* **Fields:** Stores `index_name`, `column_names` (supports composite keys), `algorithm` type, and `is_clustered` flags. + +#### **Data Storage** +* **Path:** `database/base/{db}/{table}_{index}.idx`. +* **Clustered vs Secondary:** * **Clustered:** The physical table data is reordered on disk to match the index key order. + * **Secondary:** Stores mappings of `IndexKey` to `RecordId` (Page Number + Item ID). +* **Serialization:** Uses a `paged_store` or algorithm-specific `save/load` methods to write to disk. + +--- + +### **5. Hashing and Collision Handling** +* **Hashing Function:** The code supports multiple strategies via `HashIndexType` (Static, Extendible, Linear). +* **Composite Keys:** For multi-column indices, components are encoded into bytes, with `INT` values transformed using XOR ($raw \oplus 0x8000\_0000$) to maintain bitwise sortability, then hex-encoded into a `TEXT` key. +* **Collision Handling:** * **ChainedHash:** Uses bucket chaining. + * **Extendible/Linear:** Uses dynamic resizing/splitting based on `LOAD_FACTOR_THRESHOLD`. + +--- + +### **6. Core Operations** + +#### **Search** +* **Point Lookup:** Dispatched via `search`. Supports `search_on_disk` for B+ Trees to allow traversal without loading the entire index into memory. +* **Range Scan:** Supported only by Tree-based variants. Hash-based variants return `io::ErrorKind::Unsupported`. + +#### **Insert / Update** +* When a tuple is added to a table, `add_tuple_to_all_indexes` triggers. +* It extracts the `IndexKey` from the raw tuple bytes (handling `INT`, `TEXT`, and `BOOL` types) and updates every registered index file. + +#### **Delete** +* `remove_tuple_from_all_indexes` locates the key and the specific `RecordId` to prune the index. + +#### **Clustering** +* `cluster_table_by_index` performs an out-of-place sort of the entire table's live tuples based on the index key and overwrites the table file to ensure physical contiguity. + + + +--- + +### **7. Data Consistency** +The system includes a validation utility (`validate_index_consistency`) that performs a **Full Table Scan** to build an "expected" key-map and compares it against the "actual" entries stored in the index file, reporting missing or stale entries. diff --git a/content/storage-engine/projects/indexing/extendiblehashing.md b/content/storage-engine/projects/indexing/extendiblehashing.md new file mode 100644 index 0000000..3b19232 --- /dev/null +++ b/content/storage-engine/projects/indexing/extendiblehashing.md @@ -0,0 +1,156 @@ +--- +title: Extendible Hash +sidebar_position: 5 +--- + +### Algorithm + +* Extendible hashing with: + + * **Global depth (`global_depth`)** + * **Directory (`Vec`)** mapping to buckets + * **Buckets (`Vec`)** each with **local depth (`local_depth`)** +* On overflow: + + * If `local_depth < global_depth` → split bucket + * If `local_depth == global_depth` → double directory, then split +* Directory slots may point to the same bucket (shared buckets) + +--- + +### Basic Idea + +* Use lower `global_depth` bits of `hash_code()` to index directory +* Directory points to buckets +* Buckets store `(key → Vec)` +* Growth handled by: + + * Directory doubling + * Bucket splitting +* Entries redistributed after split using updated hash mapping + +--- + +### Time Complexity + +* **Lookup:** `O(1)` +* **Insert:** `O(1)` amortised, `O(n)` during rare directory doubling +* **Delete:** `O(1)` +* **Update (insert existing key):** `O(1)` + + +--- + +### Space Complexity + +* `O(n)` +* Includes: + + * Directory (`2^global_depth`) + * Buckets + * Entries and record lists + + +--- + +### Metadata Storage Format + +* Entire structure is **serialized/deserialized using `serde`** +* Stored fields: + + * `global_depth: u32` + * `directory: Vec` + * `buckets: Vec` +* Each `EHBucket`: + + * `local_depth: u32` + * `entries: Vec` +* Each `EHEntry`: + + * `key: IndexKey` + * `records: Vec` + +--- + +### Data Storage (Persistent) + +* Uses: + + * `paged_store::save_entries(...)` + * `paged_store::load_entries_stream(...)` +* Storage format: + + * Flat stream of `(IndexKey, RecordId)` +* On load: + + * Reconstructed via repeated `insert()` calls + +--- + +### Hashing / Modulo Function + +* Directory index: + + ```rust + (key.hash_code() as usize) & ((1 << global_depth) - 1) + ``` +* Uses **bit masking (modulo by power of 2)** on hash + +--- + +### Collision Handling + +* Multiple keys in same bucket → stored in `entries: Vec` +* Same key: + + * Multiple `RecordId`s stored in `records: Vec` +* Overflow handled by: + + * Bucket splitting + * Directory expansion + +--- + +### Operations + +#### Search + +* Compute directory index +* Lookup bucket +* Linear scan in bucket (`find`) +* Return matching record list + +#### Insert + +* Compute directory index +* If key exists → append `record_id` (no duplicates) +* If bucket not full → insert +* If full: + + * Split bucket + * Retry insert (loop) + +#### Delete + +* Locate bucket +* Find key +* Remove matching `record_id` using `retain` +* Returns success if deletion happened + +#### Update + +* Same as insert for existing key: + + * Adds new `record_id` if not present + +--- + +### Additional Structural Rules + +* Directory size = `2^global_depth` +* Bucket `local_depth <= global_depth` +* Bucket capacity enforced via `EXTENDIBLE_HASH_BUCKET_CAPACITY` +* Directory may have duplicate bucket references +* Entry must have non-empty `records` + +--- diff --git a/content/storage-engine/projects/indexing/indexing_api_endpoints.md b/content/storage-engine/projects/indexing/indexing_api_endpoints.md new file mode 100644 index 0000000..3515755 --- /dev/null +++ b/content/storage-engine/projects/indexing/indexing_api_endpoints.md @@ -0,0 +1,436 @@ +--- +title: API Endpoints +sidebar_position: 13 +--- + +## 1. Module Import Map + +The crate root re-exports backend modules, so callers can use these paths directly: + +- `storage_manager::catalog::*` for catalog metadata operations +- `storage_manager::index::*` for index build, load, search, rebuild, validation, and maintenance helpers +- `storage_manager::executor::{index_scan, index_scan_by_column}` for tuple fetch via index +- `storage_manager::heap::{insert_tuple_with_index_maintenance, delete_tuple_with_index_maintenance}` for write path maintenance + +## 2. Core Data Contracts + +### `IndexAlgorithm` + +Supported values: + +- `StaticHash` +- `ChainedHash` +- `ExtendibleHash` +- `LinearHash` +- `BTree` +- `BPlusTree` +- `RadixTree` +- `SkipList` +- `LsmTree` + +Helpers: + +- `is_hash()` and `is_tree()` +- `from_str(...)` supports aliases like `btree`, `bplus_tree`, `linear_hash`, etc. + +### `IndexEntry` + +Catalog metadata for an index: + +- `index_name: String` +- `column_name: Vec` (single or composite) +- `algorithm: IndexAlgorithm` +- `is_clustered: bool` +- `include_columns: Vec` + +Useful helper: + +- `is_secondary()` is `!is_clustered` + +### `IndexKey` and `RecordId` + +- `IndexKey` variants: `Int(i64)`, `Float(f64)`, `Text(String)` +- `RecordId`: `{ page_no: u32, item_id: u32 }` + +## 3. Catalog-Level Index Endpoints (Metadata) + +These APIs register/remove index metadata in catalog and persist `catalog.json`. + +### `create_index` + +```rust +pub fn create_index( + catalog: &mut Catalog, + db_name: &str, + table_name: &str, + index_name: &str, + column_names: &[String], + algorithm: IndexAlgorithm, + is_clustered: bool, + include_columns: Vec, +) -> bool +``` + +Behavior: + +- Validates DB/table existence +- Validates indexed and include columns +- Prevents duplicate index names per table +- Enforces max one clustered index per table +- Appends `IndexEntry` and saves catalog + +Important: + +- This does **not** build or save an `.idx` file. Build step must be called separately. + +### `create_secondary_index` + +```rust +pub fn create_secondary_index( + catalog: &mut Catalog, + db_name: &str, + table_name: &str, + index_name: &str, + column_names: &[String], + algorithm: IndexAlgorithm, +) -> io::Result<()> +``` + +Behavior: + +- Wrapper over `create_index(..., is_clustered = false, include_columns = vec![])` + +### `drop_index` + +```rust +pub fn drop_index( + catalog: &mut Catalog, + db_name: &str, + table_name: &str, + index_name: &str, +) -> bool +``` + +Behavior: + +- Removes index metadata entry from table and saves catalog + +Important: + +- Does **not** delete index file from disk; remove `.idx` manually if desired. + +### `drop_secondary_index` + +```rust +pub fn drop_secondary_index( + catalog: &mut Catalog, + db_name: &str, + table_name: &str, + index_name: &str, +) -> io::Result<()> +``` + +Behavior: + +- Ensures index exists and is non-clustered +- Delegates metadata removal to `drop_index` + +Important: + +- Also does **not** delete index file from disk. + +### Listing/lookup helpers + +```rust +pub fn list_indexes<'a>( + catalog: &'a Catalog, + db_name: &str, + table_name: &str, +) -> Option<&'a Vec> + +pub fn list_secondary_indices( + catalog: &Catalog, + db_name: &str, + table_name: &str, +) -> io::Result> +``` + +### Catalog I/O used by integrations + +```rust +pub fn load_catalog() -> Catalog +pub fn save_catalog(catalog: &Catalog) +``` + +## 4. Index Build, Persistence, and Lookup Endpoints + +### `AnyIndex` constructors/loaders + +```rust +pub fn new_empty(algorithm: &IndexAlgorithm) -> AnyIndex +pub fn new_default(family: &str) -> AnyIndex +pub fn load(path: &str, algorithm: &IndexAlgorithm) -> io::Result +``` + +### Build from existing table data + +```rust +pub fn build_from_table( + catalog: &Catalog, + db_name: &str, + table_name: &str, + column_name: &str, + algorithm: &IndexAlgorithm, +) -> io::Result + +pub fn build_from_table_columns( + catalog: &Catalog, + db_name: &str, + table_name: &str, + column_names: &[String], + algorithm: &IndexAlgorithm, +) -> io::Result + +pub fn build_secondary_index( + catalog: &Catalog, + db_name: &str, + table_name: &str, + index_entry: &IndexEntry, +) -> io::Result +``` + +Notes: + +- Supports composite index keys using `column_names` +- Reads tuple bytes from table pages and extracts key(s) +- Key extraction supports `INT`, `TEXT`, `BOOL/BOOLEAN` + +### Runtime index operations + +```rust +pub fn insert(&mut self, key: IndexKey, record_id: RecordId) -> io::Result<()> +pub fn search(&self, key: &IndexKey) -> io::Result> +pub fn delete(&mut self, key: &IndexKey, record_id: &RecordId) -> io::Result +pub fn save(&self, path: &str) -> io::Result<()> +``` + +### On-disk point search (without fully materializing index in caller) + +```rust +pub fn search_on_disk( + path: &str, + algorithm: &IndexAlgorithm, + key: &IndexKey, +) -> io::Result> +``` + +Implementation detail: + +- `BPlusTree` uses dedicated on-disk traversal +- Other algorithms use paged-store search path + +### Tree-only range scan + +```rust +pub fn range_scan(&self, start: &IndexKey, end: &IndexKey) -> io::Result> +pub fn supports_range_scan(&self) -> bool +``` + +- Hash indexes return `Unsupported` for range scan + +## 5. Path and Key Utility Endpoints + +### Key conversion for lookup inputs + +```rust +pub fn index_key_from_values( + columns: &[Column], + index_columns: &[String], + values: &[String], +) -> io::Result +``` + +Use this for all user/API-provided search keys, especially composite indexes. + +Important composite-key note: + +- Multi-column keys are encoded into a sortable hex string and represented as `IndexKey::Text(...)` +- Do not handcraft composite keys; use this helper to avoid ordering/encoding mismatch + +### Canonical index file paths + +```rust +pub fn index_file_path(db_name: &str, table_name: &str, index_name: &str) -> String +pub fn secondary_index_file_path(db_name: &str, table_name: &str, index_name: &str) -> String +``` + +Current behavior: + +- Both helpers resolve to the same path shape: `database/base/{db}/{table}_{index}.idx` + +## 6. Rebuild, Validation, and Layout Endpoints + +### Rebuild APIs + +```rust +pub fn rebuild_table_indexes(catalog: &Catalog, db_name: &str, table_name: &str) -> io::Result +pub fn rebuild_secondary_index( + catalog: &Catalog, + db_name: &str, + table_name: &str, + index_name: &str, +) -> io::Result<()> +``` + +- `rebuild_table_indexes` rebuilds every registered index on the table +- `rebuild_secondary_index` refuses clustered indexes + +### Consistency validation APIs + +```rust +pub fn validate_index_consistency( + catalog: &Catalog, + db_name: &str, + table_name: &str, + index_name: &str, +) -> io::Result<()> + +pub fn validate_all_table_indexes( + catalog: &Catalog, + db_name: &str, + table_name: &str, +) -> io::Result +``` + +Validation checks: + +- Algorithm structure invariants via `validate_structure()` +- Full tuple scan vs index entries +- Missing/stale entries and entry-count mismatch + +### Clustered physical layout API + +```rust +pub fn maintain_clustered_index_layout( + catalog: &Catalog, + db_name: &str, + table_name: &str, +) -> io::Result + +pub fn cluster_table_by_index( + catalog: &Catalog, + db_name: &str, + table_name: &str, + index_name: &str, +) -> io::Result<()> +``` + +- `maintain_clustered_index_layout` auto-detects clustered index and reorders table if present +- `cluster_table_by_index` rewrites table file in key order, then rebuilds all indexes + +## 7. Write-Path Maintenance Endpoints + +### Low-level index maintenance helpers + +```rust +pub fn add_tuple_to_all_indexes( + catalog: &Catalog, + db_name: &str, + table_name: &str, + tuple: &[u8], + record_id: RecordId, +) -> io::Result + +pub fn remove_tuple_from_all_indexes( + catalog: &Catalog, + db_name: &str, + table_name: &str, + tuple: &[u8], + record_id: RecordId, +) -> io::Result +``` + +### Preferred heap APIs for inserts/deletes with index sync + +```rust +pub fn insert_tuple_with_index_maintenance( + catalog: &Catalog, + db_name: &str, + table_name: &str, + file: &mut File, + data: &[u8], +) -> io::Result + +pub fn delete_tuple_with_index_maintenance( + catalog: &Catalog, + db_name: &str, + table_name: &str, + file: &mut File, + record_id: RecordId, +) -> io::Result<()> +``` + +These call the heap operation and index maintenance in one place. + +## 8. Query Endpoints Returning Tuples + +```rust +pub fn index_scan( + catalog: &Catalog, + db_name: &str, + table_name: &str, + index_name: &str, + key: &IndexKey, +) -> io::Result>> + +pub fn index_scan_by_column( + catalog: &Catalog, + db_name: &str, + table_name: &str, + column_name: &str, + key: &IndexKey, +) -> io::Result>> +``` + +Behavior: + +- Uses catalog metadata to resolve algorithm and index file path +- Probes index via `AnyIndex::search_on_disk` +- Fetches tuples by `RecordId` +- Prefers clustered index when multiple indexes exist on same column (`index_scan_by_column`) + +## 9. Integration Playbooks + +### A. Create and build a new index + +1. Load catalog with `load_catalog()` +2. Register metadata using `create_index(...)` or `create_secondary_index(...)` +3. Build using `AnyIndex::build_from_table_columns(...)` +4. Persist with `idx.save(index_file_path(...))` +5. If clustered, call `cluster_table_by_index(...)` + +### B. Bulk load then restore index correctness + +1. Load data into heap pages +2. Call `rebuild_table_indexes(...)` +3. Call `maintain_clustered_index_layout(...)` + +This is the same post-load pattern used by CSV load paths. + +### C. OLTP insert/delete with automatic index updates + +- Use `insert_tuple_with_index_maintenance(...)` for inserts +- Use `delete_tuple_with_index_maintenance(...)` for deletes + +### D. Drop an index cleanly + +1. Remove catalog metadata with `drop_index(...)` or `drop_secondary_index(...)` +2. Delete `.idx` file on disk using `std::fs::remove_file(...)` + +## 10. Practical Caveats for Cross-Team Integration + +- Metadata registration and physical index build are intentionally separate steps. +- `include_columns` is stored in catalog metadata, but there is currently no dedicated covering-index read path. +- Composite index lookup should always use `index_key_from_values(...)`. +- If you bypass heap maintenance wrappers during writes, schedule explicit rebuild/validation. +- For post-integration checks, use `validate_index_consistency(...)` or `validate_all_table_indexes(...)` in tests or startup health checks. diff --git a/content/storage-engine/projects/indexing/linearhash.md b/content/storage-engine/projects/indexing/linearhash.md new file mode 100644 index 0000000..6638fed --- /dev/null +++ b/content/storage-engine/projects/indexing/linearhash.md @@ -0,0 +1,228 @@ +--- +title: Chain Hash +sidebar_position: 6 +--- + + +### Basic Idea + +Incremental hash-based indexing using **linear hashing**. +Buckets are split one at a time using a **split pointer (`split_ptr`)** and a **level (`level`)**, avoiding full rehashing. + +* Initial buckets: `N₀` +* Buckets grow dynamically by splitting +* Split triggered when load factor exceeds threshold +* Entries redistributed only for the split bucket + +--- + +### Hashing / Modulo Function + +Two hash levels are used: + +* Level `l`: + + ``` + h_l(k) = hash(k) mod (N₀ · 2^l) + ``` +* If bucket index `< split_ptr`, use: + + ``` + h_{l+1}(k) = hash(k) mod (N₀ · 2^{l+1}) + ``` + +Bucket selection: + +``` +if h_l(k) < split_ptr: + use h_{l+1}(k) +else: + use h_l(k) +``` + + + +--- + +### Time Complexity + +* **Insert**: + Average: O(1) + Worst: O(n) (during split + overflow traversal) + +* **Search**: + Average: O(1) + Worst: O(n) (overflow chains) + +* **Delete**: + Average: O(1) + Worst: O(n) + +* **Index Creation (load)**: + O(n) (sequential inserts from paged store) + +* **Update**: + Same as insert/delete + +--- + +### Space Complexity + +* Buckets + overflow segments store all entries +* Total space: **O(n)** +* Additional overhead: + + * Overflow segments + * Dynamic bucket growth + +--- + +### Metadata Storage Format + +Stored implicitly via struct serialization: + +``` +LinearHashIndex { + level: u32 + split_ptr: usize + initial_buckets: usize + buckets: Vec + load_factor_threshold: f64 +} +``` + +Saved using: + +``` +paged_store::save_entries(...) +``` + +Loaded using: + +``` +paged_store::load_entries_stream(...) +``` + + + +--- + +### Data Storage Format (Buckets) + +Each bucket: + +``` +LHBucket { + entries: Vec + overflow: Vec +} +``` + +Entry: + +``` +LHEntry { + key: IndexKey + records: Vec +} +``` + +Overflow: + +``` +OverflowSegment { + entries: Vec +} +``` + + + +--- + +### Collision Handling + +* Primary bucket stores up to `STATIC_HASH_BUCKET_CAPACITY` +* On overflow: + + * Entries added to last overflow segment if space exists + * Else new overflow segment created +* Multiple overflow segments form a chain + +--- + +### Splitting Mechanism + +* Trigger: `load_factor() > threshold` +* Steps: + + 1. Create new bucket + 2. Drain entries from bucket at `split_ptr` + 3. Rehash using `h_{l+1}` + 4. Redistribute entries + 5. Increment `split_ptr` + 6. If end of round: + + * `level += 1` + * `split_ptr = 0` + +--- + +### Operations + +#### Search + +* Compute bucket index using `bucket_for` +* Search in: + + * primary entries + * overflow segments +* Return matching `RecordId`s + +--- + +#### Insert + +* Compute bucket index +* Insert into: + + * existing entry if key exists + * else new entry +* Handle overflow if needed +* Trigger split if load factor exceeded + +--- + +#### Delete + +* Locate entry via bucket +* Remove `RecordId` from entry +* Returns success if deletion occurred + +--- + +#### Update + +* Not explicitly implemented +* Achieved via: + + * delete + insert + +--- + +### Load Factor + +``` +load_factor = total_records / (bucket_count × bucket_capacity) +``` + +--- + +### Additional Notes from Code + +* Duplicate `RecordId`s are avoided per key +* Overflow segments must not exceed capacity +* Structure validation ensures: + + * valid split pointer + * non-empty records + * capacity constraints diff --git a/content/storage-engine/projects/indexing/lsmtree.md b/content/storage-engine/projects/indexing/lsmtree.md new file mode 100644 index 0000000..e47ab8b --- /dev/null +++ b/content/storage-engine/projects/indexing/lsmtree.md @@ -0,0 +1,100 @@ +--- +title: LSM Tree +sidebar_position: 11 +--- + +**Algorithm:** + +* In-memory **memtable**: `BTreeMap>` +* Immutable **runs**: `Vec` where each run stores a `BTreeMap>` +* Insert into memtable → flush to runs when `memtable_limit` reached → optional full compaction when runs > 8 + +--- + +**Basic Idea:** + +* Writes go to memtable (sorted BTreeMap). +* When full, memtable is flushed as a new run (immutable). +* Reads check memtable first, then runs (newest to oldest). +* Compaction merges all runs into one, keeping latest values. + +--- + +**Time Complexity:** + +* Insert: + + * `O(log n)` (BTreeMap insert) + occasional flush `O(n)` +* Search: + + * `O(log n)` (memtable) + `O(R * log n)` worst-case over runs +* Delete: + + * Same as search + `O(k)` for filtering record IDs +* Compaction: + + * `O(total_entries)` + +--- + +**Space Complexity:** + +* `O(N)` total across memtable + runs +* Temporary `O(N)` during compaction + +--- + +**Metadata Storage Format:** + +* No explicit separate metadata file +* Entire index reconstructed using: + + * `paged_store::load_entries_stream(path, |key, rid| insert(...))` + +--- + +**Data Storage Format:** + +* Stored as flat `(IndexKey, RecordId)` pairs via: + + * `paged_store::save_entries(path, iterator)` +* Logical structure (memtable + runs) is rebuilt on load + +--- + +**Hashing / Modulo:** + +* Not used +* Uses ordered `BTreeMap` + +--- + +**Collision Handling:** + +* Multiple `RecordId`s per key stored as `Vec` +* Duplicate prevention via `if !list.contains(&record_id)` + +--- + +**Operations:** + +* **Search:** + + * Check memtable → then runs in order → return first match + +* **Insert:** + + * Add to memtable + * Avoid duplicate record IDs + * Trigger flush if limit reached + +* **Delete:** + + * If in memtable → remove directly + * Else → fetch current values → rewrite updated version into memtable + +* **Update:** + + * Not explicit; achieved via insert/delete combination + +--- diff --git a/content/storage-engine/projects/indexing/overview.md b/content/storage-engine/projects/indexing/overview.md new file mode 100644 index 0000000..162f059 --- /dev/null +++ b/content/storage-engine/projects/indexing/overview.md @@ -0,0 +1,43 @@ +--- +title: Indexing Overview +sidebar_position: 2 +--- + +# Indexing Overview + +## Purpose +RookDB currently relies on sequential scans for most lookups. +This project introduces an indexing subsystem to support faster access paths for point and range queries. +The design keeps compatibility with the existing storage stack, including page layout, disk I/O, and buffer handling. + +## Objectives +1. Reduce lookup latency on large tables. +2. Support exact match queries through hash and tree indexes. +3. Support range queries through tree indexes. +4. Persist index metadata in the catalog for recovery and startup loading. +5. Keep indexes synchronized with table inserts and deletes. +6. Expose index lifecycle operations through the CLI. + +## Initial Scope +The first implementation includes: +1. Hash indexes with static, chain, extendible, and linear variants. +2. B, B+ tree, Radix Tree, Skip List and LSM Tree indexes with point lookup and range scan. +3. Primary key and secondary index metadata support. +4. Covering index metadata support with included columns. + reordering for clustered storage. + +## Storage Model + +### Index File Naming +Each index is stored as a separate file. + +```text +database/base/{database_name}/{table_name}_{index_name}.idx +``` + +This separation isolates index lifecycle operations from table data files. + +### Page Size and Common Layout +Index files use the same 8 KiB page size as table files. +This allows reuse of existing page read and write paths. + diff --git a/content/storage-engine/projects/indexing/radixtree.md b/content/storage-engine/projects/indexing/radixtree.md new file mode 100644 index 0000000..edb9c7c --- /dev/null +++ b/content/storage-engine/projects/indexing/radixtree.md @@ -0,0 +1,147 @@ +--- +title: Radix Tree +sidebar_position: 9 +--- + + +### Algorithm + +A radix tree (compressed trie) stores keys as byte sequences. Consecutive single-child nodes are merged into a single node with a multi-byte prefix. Each node represents a prefix of keys and branches based on the next byte. + +Insertion uses longest common prefix (LCP) comparison: + +* If partial match → node split. +* If full match → descend or mark terminal. + +Search and delete traverse using prefix matching. + +--- + +### Basic Idea + +* Keys are converted to byte arrays using `IndexKey::as_bytes()` +* Tree stores compressed prefixes instead of single characters +* Each node: + + * `prefix`: compressed edge label + * `children`: sorted map (byte → node) + * `terminal`: optional `(key, [record_ids])` + +--- + +### Time Complexity + +Let `k = key length in bytes` + +| Operation | Complexity | +| -------------- | ----------------- | +| Index Creation | O(n · k) | +| Insert | O(k) | +| Search | O(k) | +| Delete | O(k) | +| Range Scan | O(k + k · output) | + +--- + +### Space Complexity + +* O(total bytes of all keys) +* Prefix compression reduces redundancy +* Additional overhead: + + * `BTreeMap` per node + * RecordId vectors at terminals + +--- + +### Metadata Storage Format + +* No explicit metadata file structure defined +* Tree is reconstructed using: + + * `paged_store::load_entries_stream(path, ...)` +* Entries are streamed and inserted into the tree + +--- + +### Data Storage Format + +* Stored via: + + * `paged_store::save_entries(path, iterator)` +* Format: + + * Sequence of `(IndexKey, RecordId)` pairs +* Tree structure itself is **not serialized directly** +* Rebuilt by replaying inserts + +--- + +### Hashing / Modulo Function + +* Not used +* Structure is tree-based, not hash-based + +--- + +### Collision Handling + +* Not applicable (no hashing) +* Multiple records per key handled via: + + * `terminal: Option<(IndexKey, Vec)>` + +--- + +### Operations + +#### Search + +* Traverse using prefix matching +* If full match and terminal exists → return record IDs +* Else → return empty + +#### Insert + +* Compute LCP with node prefix +* Cases: + + * Partial match → split node + * Full match → descend or create child + * Exact match → append RecordId (no duplicates) + +#### Delete + +* Traverse to terminal node +* Remove specific RecordId +* If empty: + + * Remove terminal + * If node becomes empty → prune + * If single child → compress (merge) + +#### Update + +* Not explicitly implemented +* Equivalent to: + + * Delete(old_key, rid) + * Insert(new_key, rid) + +--- + +### Additional Behavior + +* Children stored in `BTreeMap` → maintains sorted order +* Enables correct lexicographic range scans +* Range scan: + + * Recursively collects keys within `[start, end]` +* Supports: + + * `min_key()` + * `max_key()` + * `entry_count()` + * structural validation + +--- diff --git a/content/storage-engine/projects/indexing/skiplist.md b/content/storage-engine/projects/indexing/skiplist.md new file mode 100644 index 0000000..99a5973 --- /dev/null +++ b/content/storage-engine/projects/indexing/skiplist.md @@ -0,0 +1,108 @@ +--- +title: Skip List +sidebar_position: 10 +--- + +**Basic Idea** + +* Implements an ordered index using an in-memory `BTreeMap>`. +* Each key maps to a list of record IDs to support duplicate keys. +* Despite the name, it is not an actual probabilistic skip list; ordering and range behavior are provided by `BTreeMap`. + +--- + +**Time Complexity** + +* **Insert:** `O(log N + K)` + + * `log N` for map insertion, `K` for checking duplicates in vector. +* **Search:** `O(log N + K)` + + * `log N` for lookup, `K` for returning all record IDs. +* **Delete:** `O(log N + K)` + + * `log N` for lookup, `K` for filtering vector. +* **Range Scan:** `O(log N + M)` + + * `log N` to locate range start, `M` total elements in range. +* **Index Creation (load):** `O(T log N)` + + * Inserts all entries sequentially using `insert`. + +--- + +**Space Complexity** + +* `O(N + T)` + + * `N` = number of unique keys + * `T` = total number of `(key, record_id)` pairs + +--- + +**Metadata Storage Format** + +* No explicit metadata structure is defined. +* No separate metadata file is created or managed. + +--- + +**Data Storage Format** + +* Stored using `paged_store::save_entries` as a stream of `(IndexKey, RecordId)` pairs. +* Loading is performed via `paged_store::load_entries_stream`, reconstructing the index by reinserting entries. +* Data is not stored as a tree or levels; it is serialized as flat key–record pairs. + +--- + +**Hashing / Modulo Function** + +* Not used. +* Index is tree-based (`BTreeMap`), not hash-based. + +--- + +**Collision Handling** + +* Multiple records for the same key are stored in a `Vec`. +* Duplicate `(key, record_id)` pairs are prevented by checking `list.contains()` before insertion. + +--- + +**Operations** + +* **Insert** + + * Adds `record_id` to the vector for a key. + * Avoids duplicates. + +* **Search** + + * Returns all record IDs for a given key. + * Returns empty vector if key is absent. + +* **Delete** + + * Removes specific `record_id` from the key’s vector. + * Removes key entirely if no record IDs remain. + +* **Update** + + * Not explicitly implemented. + * Achieved via delete + insert. + +* **Range Scan** + + * Uses ordered traversal via `BTreeMap::range`. + * Returns concatenated record IDs within `[start, end]`. + +* **Min / Max Key** + + * Retrieved using ordered key iterators (`next`, `next_back`). + +--- + +**Validation** + +* Ensures no key has an empty `Vec`. +* Returns error if such a case is found. diff --git a/content/storage-engine/projects/indexing/statichash.md b/content/storage-engine/projects/indexing/statichash.md new file mode 100644 index 0000000..eb28843 --- /dev/null +++ b/content/storage-engine/projects/indexing/statichash.md @@ -0,0 +1,242 @@ +--- +title: Static Hash +sidebar_position: 3 +--- + +### 1. Algorithm Overview + +The implementation is a **static hash-based index** with: + +* Fixed number of buckets (`num_buckets`) +* Each bucket contains: + + * Primary storage (`entries`) + * Overflow storage (`overflow` → list of segments) + +Each **key maps to a bucket** using a hash function, and each bucket stores: + +* `BucketEntry { key, records: Vec }` + +--- + +### 2. Basic Idea + +* Compute bucket index using hash modulo. +* Store `(key → multiple record_ids)` inside bucket entries. +* If primary bucket is full: + + * Use overflow segments. + * Each segment has the same capacity as primary. +* Duplicate `(key, record_id)` pairs are avoided. + +--- + +### 3. Time Complexity + +Let: + +* `B = STATIC_HASH_BUCKET_CAPACITY` +* `O = number of overflow segments in a bucket` + +**Index Creation (load):** + +* Inserts each entry via streaming + → **O(N × (B + O·B))** + +**Insert:** + +* Search existing key: scan primary + overflow + → **O(B + O·B)** +* Insert is O(1) after location found + +**Search:** +→ **O(B + O·B)** + +**Delete:** +→ **O(B + O·B)** + +**Update (not explicitly implemented):** + +* Would be delete + insert + → **O(B + O·B)** + +--- + +### 4. Space Complexity + +* Buckets: `num_buckets` +* Each bucket: + + * Primary capacity: `B` + * Overflow grows dynamically + +Total: +→ **O(N)** (records stored across buckets and overflow) + +--- + +### 5. Metadata Storage Format + +* Struct: + + ```rust + pub struct StaticHashIndex { + num_buckets: usize, + buckets: Vec, + } + ``` +* Uses `serde::{Serialize, Deserialize}` + → Metadata is serializable (format depends on `paged_store` usage, not explicitly defined here) + +--- + +### 6. Data Storage Format + +Each bucket: + +```rust +struct Bucket { + entries: Vec, + overflow: Vec, +} +``` + +Overflow: + +```rust +struct OverflowSegment { + entries: Vec, +} +``` + +Entry: + +```rust +struct BucketEntry { + key: IndexKey, + records: Vec, +} +``` + +* Primary entries stored first +* Overflow stored as chained segments +* Each entry stores **one key → multiple record IDs** + +--- + +### 7. Hashing / Modulo Function + +```rust +(key.hash_code() as usize) % self.num_buckets +``` + +* Uses `IndexKey.hash_code()` +* Modulo determines bucket index + +--- + +### 8. Collision Handling + +* Collisions handled via: + + * Multiple entries per bucket + * Overflow segments when capacity exceeded + +Flow: + +1. Try primary bucket +2. If full → check last overflow segment +3. If that is full → create new overflow segment + +--- + +### 9. Operations + +#### Search + +```rust +bucket.find(key) +``` + +* Scan primary entries +* Then scan overflow segments +* Return cloned `Vec` + +--- + +#### Insert + +```rust +bucket.insert(key, record_id) +``` + +Steps: + +1. If key exists: + + * Append record_id (if not duplicate) +2. Else: + + * Insert into primary if space + * Else into last overflow segment if space + * Else create new overflow segment + +--- + +#### Delete + +```rust +entry.records.retain(|r| r != record_id) +``` + +* Removes record_id from matching key +* Returns whether deletion happened +* Does NOT remove empty entries + +--- + +#### Update + +* Not explicitly implemented +* Would require: + + * delete(old) + * insert(new) + +--- + +### 10. Persistence + +* Load: + +```rust +paged_store::load_entries_stream(path, |key, rid| index.insert(key, rid)) +``` + +* Save: + +```rust +paged_store::save_entries(path, self.all_entries()?.into_iter()) +``` + +* Storage is **entry-wise (key, record_id)** + +--- + +### 11. Additional Notes + +* Bucket capacity strictly enforced for: + + * Primary entries + * Each overflow segment +* Structure validation checks: + + * Bucket size constraints + * Non-empty record lists +* Load factor: + +```rust +entry_count / (num_buckets × capacity) +``` + +--- \ No newline at end of file