Indexing Documentation

content/storage-engine/projects/indexing/bplustree.md

@@ -0,0 +1,184 @@
+---
+title: B+ Tree 
+sidebar_position: 8
+---
+
+### Algorithm
+
+* B+ Tree with minimum degree `t`
+* Internal nodes store only routing keys and child pointers
+* Leaf nodes store `(key, Vec<RecordId>)`
+* Leaves are linked using `next_leaf` pointer (singly linked forward)
+* Nodes stored in an arena (`Vec<BPlusNode>`) with stable indices
+* All leaves are at the same depth
+* Node capacity:
+
+  * Max keys: `2t - 1`
+  * Min keys (non-root): `t - 1`
+
+---
+
+### Basic Idea
+
+* Search descends from root to leaf using binary partition (`partition_point`)
+* Insert happens at leaf; overflow triggers split and upward propagation
+* Delete removes `(key, rid)` and handles underflow via borrow/merge
+* Range scan uses linked leaves after initial descent
+
+---
+
+### Time Complexity
+
+* Search: `O(t · log_t n)`
+* Insert: `O(t · log_t n)`
+* Delete: `O(t · log_t n)`
+* Range scan: `O(log n + k)`
+* Index creation (bulk via inserts): same as repeated insert
+
+---
+
+### Space Complexity
+
+* In-memory:
+
+  * `O(n)` nodes stored in `Vec<BPlusNode>`
+* On-disk:
+
+  * One page per node (`PAGE_SIZE`)
+  * Total pages = number of nodes + 1 header page
+
+---
+
+### Metadata File Storage Format
+
+* First page (header page):
+
+  * Bytes 0–8: magic `"RDBIDXV1"`
+  * Bytes 8–10: version
+  * Bytes 10–12: header size
+  * Bytes 12–16: page size
+  * Bytes 16–20: root page
+  * Bytes 20–24: node page count
+  * Bytes 24–32: entry count
+  * Bytes 32–36: minimum degree `t`
+
+---
+
+### Data Storage Format (Per Node Page)
+
+* Fixed-size page (`PAGE_SIZE`)
+* Header (16 bytes):
+
+  * Byte 0: `is_leaf`
+  * Byte 1: `dead`
+  * Bytes 2–4: key count
+  * Bytes 4–6: child count
+  * Bytes 8–12: `next_leaf_page`
+  * Bytes 12–16: payload size
+* Payload:
+
+  * Keys (encoded sequentially)
+  * If leaf:
+
+    * For each key:
+
+      * `rid_count`
+      * List of `(page_no, item_id)`
+  * If internal:
+
+    * Child page numbers
+
+---
+
+### Hashing / Modulo Function
+
+* Not used
+
+---
+
+### Collision Handling
+
+* Multiple records per key handled via:
+
+  * `values: Vec<Vec<RecordId>>`
+  * Same key maps to a vector of record IDs
+
+---
+
+### Operations
+
+#### Search
+
+* Traverse from root using `partition_point`
+* At leaf:
+
+  * Binary position lookup
+  * Return `Vec<RecordId>` or empty
+
+#### Insert
+
+* Insert into leaf at sorted position
+* If key exists:
+
+  * Append `RecordId` if not duplicate
+* If overflow (`>= 2t` keys):
+
+  * Split leaf
+  * Push first key of right node upward
+  * Recursively split internal nodes if needed
+* Root split creates new root
+
+#### Delete
+
+* Locate leaf and key
+* Remove specific `RecordId`
+* If key has no more records:
+
+  * Remove key
+* If underflow (`< t-1` keys):
+
+  * Borrow from sibling OR merge
+* Propagate fixes upward
+* If root becomes empty:
+
+  * Replace with single child
+
+#### Update
+
+* Not explicitly implemented
+* Equivalent to:
+
+  * `delete(key, old_rid)` + `insert(key, new_rid)`
+
+---
+
+### Disk Operations
+
+* Save:
+
+  * Header page + serialized node pages
+* Load:
+
+  * Parse header
+  * Load all node pages into memory
+* Direct search:
+
+  * `search_on_disk()` performs traversal without full load
+
+---
+
+### Additional Notes
+
+* Node splitting:
+
+  * Leaf: split at index `t`
+  * Internal: median key moves up
+* Leaf linking maintained during split/merge
+* Dead nodes marked but not reused
+* Validation ensures:
+
+  * Sorted keys
+  * Proper child counts
+  * No cycles in tree or leaf chain
+
+---

content/storage-engine/projects/indexing/btree.md

@@ -0,0 +1,185 @@
+---
+title: BTree 
+sidebar_position: 7
+---
+
+
+### Algorithm
+
+* Classic B-Tree (Knuth / CLRS definition).
+* Minimum degree `t`.
+* Each node:
+
+  * Stores `keys: Vec<IndexKey>`
+  * Stores `values: Vec<Vec<RecordId>>` (multiple RIDs per key)
+  * Stores `children: Vec<usize>` (indices into arena)
+  * `is_leaf`, `dead` flags
+* Nodes stored in a flat arena: `Vec<BTreeNode>`
+* Root tracked by index.
+* All leaves at same depth.
+* Internal node: `k` keys → `k+1` children.
+* Node capacity: `[t−1, 2t−1]` keys (except root).
+
+---
+
+### Basic Idea
+
+* Tree-based index storing keys in sorted order.
+* Each key maps to one or more record IDs.
+* Uses balanced multi-way tree.
+* Insert splits full nodes.
+* Delete uses merge/rotation (CLRS algorithm).
+* Arena-based storage using indices instead of pointers. 
+
+---
+
+### Time Complexity
+
+(from code comments)
+
+| Operation  | Complexity         |
+| ---------- | ------------------ |
+| search     | O(t · log_t n)     |
+| insert     | O(t · log_t n)     |
+| delete     | O(t · log_t n)     |
+| range_scan | O(t · log_t n + k) |
+
+---
+
+### Space Complexity
+
+* Nodes stored in `Vec<BTreeNode>` (arena).
+* Each node holds up to `2t−1` keys.
+* Deleted nodes remain as tombstones (`dead = true`).
+* Overall: **O(n)** space.
+
+---
+
+### Metadata Storage Format
+
+* Entire structure serialized using `serde::{Serialize, Deserialize}`.
+* Persistence:
+
+  * `save()` → `paged_store::save_entries(...)`
+  * `load()` → `paged_store::load_entries_stream(...)`
+* No explicit separate metadata file structure defined in code.
+* Metadata implicitly:
+
+  * `nodes` vector
+  * `root` index
+  * `t`
+  * `entry_count`
+
+---
+
+### Data Storage Format
+
+* Stored as `(IndexKey, RecordId)` pairs via paged store.
+* On load:
+
+  * Entries streamed and inserted one-by-one.
+* In-memory:
+
+  * Keys stored in nodes
+  * Values stored as `Vec<RecordId>` per key
+* Tree reconstructed from entries (no direct node serialization in load).
+
+---
+
+### Hashing / Modulo Function
+
+* Not used.
+* Tree-based index (ordered structure).
+
+---
+
+### Collision Handling
+
+* Multiple records for same key stored as:
+
+  * `values[i] = Vec<RecordId>`
+* Duplicate keys not duplicated structurally.
+* Record IDs appended if key exists.
+
+---
+
+### Operations
+
+#### Search
+
+* Binary search using `partition_point`.
+* If key found → return `values[pos]`.
+* Else descend to correct child.
+* Leaf → return empty.
+
+---
+
+#### Insert
+
+* If root full → split.
+* Use `insert_non_full`:
+
+  * If key exists → append RID if not present.
+  * Else insert key + new RID.
+* Child split done before descending if full.
+
+---
+
+#### Delete
+
+* Two-step:
+
+  1. Remove specific `(key, rid)` from values.
+  2. If values empty → remove key structurally.
+* Structural deletion:
+
+  * Uses CLRS algorithm:
+
+    * Replace with predecessor/successor
+    * Rotate (left/right)
+    * Merge children
+* Nodes merged → right node marked `dead`.
+
+---
+
+#### Update
+
+* Not explicitly implemented.
+* Equivalent to:
+
+  * delete(old key, rid)
+  * insert(new key, rid)
+
+---
+
+#### Range Scan
+
+* In-order traversal:
+
+  * Visit children and keys in sorted order.
+  * Collect RIDs within `[start, end]`.
+
+---
+
+#### Validation
+
+* Ensures:
+
+  * Keys sorted
+  * Values match keys
+  * No empty RID lists
+  * Correct child count
+  * No cycles
+  * No reachable dead nodes
+
+---
+
+#### Additional Notes
+
+* Arena-based node storage using indices.
+* Tombstoned nodes not reclaimed.
+* Supports:
+
+  * `min_key()`
+  * `max_key()`
+  * full traversal (`all_entries`)