From 58c86042fca0f22bdaf2966cbd69e9a8f11690c8 Mon Sep 17 00:00:00 2001
From: George Rahul <75750164+georgerahul24@users.noreply.github.com>
Date: Fri, 24 Apr 2026 03:13:10 +0530
Subject: [PATCH 1/8] Adding overview.md

---
 .../projects/indexing/catalogintegeration.md  |  4 ++
 .../projects/indexing/overview.md             | 43 +++++++++++++++++++
 2 files changed, 47 insertions(+)
 create mode 100644 content/storage-engine/projects/indexing/catalogintegeration.md
 create mode 100644 content/storage-engine/projects/indexing/overview.md

diff --git a/content/storage-engine/projects/indexing/catalogintegeration.md b/content/storage-engine/projects/indexing/catalogintegeration.md
new file mode 100644
index 0000000..ff99131
--- /dev/null
+++ b/content/storage-engine/projects/indexing/catalogintegeration.md
@@ -0,0 +1,4 @@
+---
+title: Catalog Integration
+sidebar_position: 2
+---
diff --git a/content/storage-engine/projects/indexing/overview.md b/content/storage-engine/projects/indexing/overview.md
new file mode 100644
index 0000000..e7c76d9
--- /dev/null
+++ b/content/storage-engine/projects/indexing/overview.md
@@ -0,0 +1,43 @@
+---
+title: Indexing Overview
+sidebar_position: 2
+---
+
+# Indexing Overview
+
+## Purpose
+RookDB currently relies on sequential scans for most lookups.
+This project introduces an indexing subsystem to support faster access paths for point and range queries.
+The design keeps compatibility with the existing storage stack, including page layout, disk I/O, and buffer handling.
+
+## Objectives
+1. Reduce lookup latency on large tables.
+2. Support exact match queries through hash and B+ tree indexes.
+3. Support range queries through B+ tree indexes.
+4. Persist index metadata in the catalog for recovery and startup loading.
+5. Keep indexes synchronized with table inserts and deletes.
+6. Expose index lifecycle operations through the CLI.
+
+## Initial Scope
+The first implementation includes:
+1. Hash indexes with static, chain, extendible, and linear variants.
+2. B, B+ tree, Radix Tree, Skip List and LSM Tree indexes with point lookup and range scan.
+3. Primary key and secondary index metadata support.
+4. Covering index metadata support with included columns.
+ reordering for clustered storage.
+
+## Storage Model
+
+### Index File Naming
+Each index is stored as a separate file.
+
+```text
+database/base/{database_name}/{table_name}_{index_name}.idx
+```
+
+This separation isolates index lifecycle operations from table data files.
+
+### Page Size and Common Layout
+Index files use the same 8 KiB page size as table files.
+This allows reuse of existing page read and write paths.
+

From e0557fd3ac31c785d6faeddd28e571f69f9e392f Mon Sep 17 00:00:00 2001
From: George Rahul <75750164+georgerahul24@users.noreply.github.com>
Date: Fri, 24 Apr 2026 03:33:56 +0530
Subject: [PATCH 2/8] Added static and chain has doc

---
 .../projects/indexing/catalogintegeration.md  |  65 ++++-
 .../projects/indexing/chainhash.md            | 185 +++++++++++++
 .../projects/indexing/overview.md             |   4 +-
 .../projects/indexing/statichash.md           | 243 ++++++++++++++++++
 4 files changed, 494 insertions(+), 3 deletions(-)
 create mode 100644 content/storage-engine/projects/indexing/chainhash.md
 create mode 100644 content/storage-engine/projects/indexing/statichash.md

diff --git a/content/storage-engine/projects/indexing/catalogintegeration.md b/content/storage-engine/projects/indexing/catalogintegeration.md
index ff99131..38ec031 100644
--- a/content/storage-engine/projects/indexing/catalogintegeration.md
+++ b/content/storage-engine/projects/indexing/catalogintegeration.md
@@ -1,4 +1,67 @@
 ---
 title: Catalog Integration
-sidebar_position: 2
+sidebar_position: 3
 ---
+
+
+## 1. Catalog Extension
+
+Each table in the catalog maintains index metadata alongside column definitions.
+
+* Indexes are stored per table and keyed by index name.
+* Metadata is persisted and loaded at startup.
+
+### Index Metadata Includes
+
+* Indexed columns (supports multi-column indexes)
+* Index algorithm (hash/tree variants)
+* Uniqueness flag
+* Clustered flag (only one allowed per table)
+* Included columns (for covering indexes)
+* Index file location (for physical storage)
+
+---
+
+## 2. Lifecycle Integration
+
+### Creation
+
+When an index is created:
+
+1. Validate columns and constraints (existence, duplicates, clustered rules)
+2. Add entry to catalog
+3. Persist catalog to disk
+4. Trigger index build from table data
+5. Create and link physical index file
+
+### Deletion
+
+* Remove index metadata from catalog
+* Delete or detach corresponding index file
+
+### Startup
+
+* Catalog is loaded from disk
+* Index metadata is used to locate and initialize index structures
+* Enables immediate query-time usage
+
+---
+
+## 3. Query Integration
+
+* Query planner consults catalog to discover available indexes
+* Selects appropriate index based on:
+
+  * Indexed columns
+  * Algorithm type
+  * Query predicates
+* Execution layer uses catalog metadata to route lookups to the correct index structure
+
+---
+
+## 4. Design Principles
+
+* Catalog is the **single source of truth** for index metadata
+* Physical index structures are **decoupled but referenced**
+* Validation is enforced at catalog level before index creation
+* Supports extensibility for new index types and configurations
diff --git a/content/storage-engine/projects/indexing/chainhash.md b/content/storage-engine/projects/indexing/chainhash.md
new file mode 100644
index 0000000..d702b7c
--- /dev/null
+++ b/content/storage-engine/projects/indexing/chainhash.md
@@ -0,0 +1,185 @@
+---
+title: Chain Hash
+sidebar_position: 5
+---
+## Chained Hash Index — Code-Based Documentation
+
+### Algorithm
+
+The implementation is a **chained hash index** using:
+
+* Fixed number of buckets (`bucket_count`)
+* Each bucket is a `Vec<ChainEntry>`
+* Each `ChainEntry` contains:
+
+  * `key: IndexKey`
+  * `records: Vec<RecordId>`
+
+---
+
+### Basic Idea
+
+* Keys are hashed using `key.hash_code()`
+* Bucket index is computed using modulo operation
+* Each bucket stores a list of entries (separate chaining)
+* Each key maps to multiple `RecordId`s
+
+---
+
+### Time Complexity
+
+* **Index Creation (load)**
+  Uses `paged_store::load_entries_stream` and calls `insert` per entry
+  → **O(N × bucket_scan)**
+
+* **Insert**
+
+  * Bucket lookup: O(1)
+  * Scan within bucket: O(k)
+    → **O(k)**
+
+* **Search**
+
+  * Bucket lookup: O(1)
+  * Scan within bucket: O(k)
+    → **O(k)**
+
+* **Delete**
+
+  * Bucket lookup: O(1)
+  * Scan + retain: O(k + r)
+    → **O(k + r)**
+
+Where:
+
+* `k` = number of entries in a bucket
+* `r` = number of record IDs in an entry
+
+---
+
+### Space Complexity
+
+* Buckets: `O(bucket_count)`
+* Entries: `O(number_of_keys)`
+* Record storage: `O(total_record_ids)`
+* Total: **O(bucket_count + keys + records)**
+
+---
+
+### Metadata File Storage Format
+
+* Uses `paged_store::save_entries`
+* Data is written as a stream of `(IndexKey, RecordId)` pairs
+* No explicit metadata structure in this code
+* Serialization derives:
+
+  * `Serialize`, `Deserialize` on structs
+
+---
+
+### Data Storage Format
+
+* Internally:
+
+  ```
+  buckets: Vec<Vec<ChainEntry>>
+  ```
+* Flattened during save:
+
+  ```
+  Vec<(IndexKey, RecordId)>
+  ```
+* Each `(key, record_id)` pair stored independently
+
+---
+
+### Hashing / Modulo Function
+
+```
+bucket_index = (key.hash_code() as usize) % bucket_count
+```
+
+---
+
+### Collision Handling
+
+* **Separate chaining**
+* Each bucket is a `Vec<ChainEntry>`
+* Multiple keys in same bucket handled via linear scan
+
+---
+
+### Operations
+
+#### Search
+
+* Compute bucket index
+* Find entry with matching key
+* Return cloned `records`
+* If not found → empty vector
+
+---
+
+#### Insert
+
+* Compute bucket index
+* If key exists:
+
+  * Add `record_id` if not already present
+* Else:
+
+  * Create new `ChainEntry`
+  * Append to bucket
+
+---
+
+#### Delete
+
+* Compute bucket index
+* Find matching entry
+* Remove `record_id` using `retain`
+* If no records remain:
+
+  * Remove entire `ChainEntry`
+* Returns `true` if deletion occurred
+
+---
+
+#### Update
+
+* Not explicitly implemented
+* Can be inferred as:
+
+  * Delete old `(key, record_id)`
+  * Insert new `(key, record_id)`
+
+---
+
+### Additional Functions
+
+* **save**
+  Writes all `(key, record_id)` pairs using paged store
+
+* **all_entries**
+  Flattens buckets into vector of pairs
+
+* **entry_count**
+  Total number of record IDs across all entries
+
+* **validate_structure**
+
+  * `bucket_count > 0`
+  * `bucket_count == buckets.len()`
+  * No entry has empty `records`
+
+* **load_factor**
+
+  ```
+  entry_count / bucket_count
+  ```
+
+* **index_type_name**
+
+  ```
+  "chained_hash"
+  ```
diff --git a/content/storage-engine/projects/indexing/overview.md b/content/storage-engine/projects/indexing/overview.md
index e7c76d9..162f059 100644
--- a/content/storage-engine/projects/indexing/overview.md
+++ b/content/storage-engine/projects/indexing/overview.md
@@ -12,8 +12,8 @@ The design keeps compatibility with the existing storage stack, including page l
 
 ## Objectives
 1. Reduce lookup latency on large tables.
-2. Support exact match queries through hash and B+ tree indexes.
-3. Support range queries through B+ tree indexes.
+2. Support exact match queries through hash and tree indexes.
+3. Support range queries through tree indexes.
 4. Persist index metadata in the catalog for recovery and startup loading.
 5. Keep indexes synchronized with table inserts and deletes.
 6. Expose index lifecycle operations through the CLI.
diff --git a/content/storage-engine/projects/indexing/statichash.md b/content/storage-engine/projects/indexing/statichash.md
new file mode 100644
index 0000000..45a657f
--- /dev/null
+++ b/content/storage-engine/projects/indexing/statichash.md
@@ -0,0 +1,243 @@
+---
+title: Static Hash
+sidebar_position: 4
+---
+## Static Hash Index – Concise Documentation (from code)
+
+### 1. Algorithm Overview
+
+The implementation is a **static hash-based index** with:
+
+* Fixed number of buckets (`num_buckets`)
+* Each bucket contains:
+
+  * Primary storage (`entries`)
+  * Overflow storage (`overflow` → list of segments)
+
+Each **key maps to a bucket** using a hash function, and each bucket stores:
+
+* `BucketEntry { key, records: Vec<RecordId> }`
+
+---
+
+### 2. Basic Idea
+
+* Compute bucket index using hash modulo.
+* Store `(key → multiple record_ids)` inside bucket entries.
+* If primary bucket is full:
+
+  * Use overflow segments.
+  * Each segment has the same capacity as primary.
+* Duplicate `(key, record_id)` pairs are avoided.
+
+---
+
+### 3. Time Complexity
+
+Let:
+
+* `B = STATIC_HASH_BUCKET_CAPACITY`
+* `O = number of overflow segments in a bucket`
+
+**Index Creation (load):**
+
+* Inserts each entry via streaming
+  → **O(N × (B + O·B))**
+
+**Insert:**
+
+* Search existing key: scan primary + overflow
+  → **O(B + O·B)**
+* Insert is O(1) after location found
+
+**Search:**
+→ **O(B + O·B)**
+
+**Delete:**
+→ **O(B + O·B)**
+
+**Update (not explicitly implemented):**
+
+* Would be delete + insert
+  → **O(B + O·B)**
+
+---
+
+### 4. Space Complexity
+
+* Buckets: `num_buckets`
+* Each bucket:
+
+  * Primary capacity: `B`
+  * Overflow grows dynamically
+
+Total:
+→ **O(N)** (records stored across buckets and overflow)
+
+---
+
+### 5. Metadata Storage Format
+
+* Struct:
+
+  ```rust
+  pub struct StaticHashIndex {
+      num_buckets: usize,
+      buckets: Vec<Bucket>,
+  }
+  ```
+* Uses `serde::{Serialize, Deserialize}`
+  → Metadata is serializable (format depends on `paged_store` usage, not explicitly defined here)
+
+---
+
+### 6. Data Storage Format
+
+Each bucket:
+
+```rust
+struct Bucket {
+    entries: Vec<BucketEntry>,
+    overflow: Vec<OverflowSegment>,
+}
+```
+
+Overflow:
+
+```rust
+struct OverflowSegment {
+    entries: Vec<BucketEntry>,
+}
+```
+
+Entry:
+
+```rust
+struct BucketEntry {
+    key: IndexKey,
+    records: Vec<RecordId>,
+}
+```
+
+* Primary entries stored first
+* Overflow stored as chained segments
+* Each entry stores **one key → multiple record IDs**
+
+---
+
+### 7. Hashing / Modulo Function
+
+```rust
+(key.hash_code() as usize) % self.num_buckets
+```
+
+* Uses `IndexKey.hash_code()`
+* Modulo determines bucket index
+
+---
+
+### 8. Collision Handling
+
+* Collisions handled via:
+
+  * Multiple entries per bucket
+  * Overflow segments when capacity exceeded
+
+Flow:
+
+1. Try primary bucket
+2. If full → check last overflow segment
+3. If that is full → create new overflow segment
+
+---
+
+### 9. Operations
+
+#### Search
+
+```rust
+bucket.find(key)
+```
+
+* Scan primary entries
+* Then scan overflow segments
+* Return cloned `Vec<RecordId>`
+
+---
+
+#### Insert
+
+```rust
+bucket.insert(key, record_id)
+```
+
+Steps:
+
+1. If key exists:
+
+   * Append record_id (if not duplicate)
+2. Else:
+
+   * Insert into primary if space
+   * Else into last overflow segment if space
+   * Else create new overflow segment
+
+---
+
+#### Delete
+
+```rust
+entry.records.retain(|r| r != record_id)
+```
+
+* Removes record_id from matching key
+* Returns whether deletion happened
+* Does NOT remove empty entries
+
+---
+
+#### Update
+
+* Not explicitly implemented
+* Would require:
+
+  * delete(old)
+  * insert(new)
+
+---
+
+### 10. Persistence
+
+* Load:
+
+```rust
+paged_store::load_entries_stream(path, |key, rid| index.insert(key, rid))
+```
+
+* Save:
+
+```rust
+paged_store::save_entries(path, self.all_entries()?.into_iter())
+```
+
+* Storage is **entry-wise (key, record_id)**
+
+---
+
+### 11. Additional Notes
+
+* Bucket capacity strictly enforced for:
+
+  * Primary entries
+  * Each overflow segment
+* Structure validation checks:
+
+  * Bucket size constraints
+  * Non-empty record lists
+* Load factor:
+
+```rust
+entry_count / (num_buckets × capacity)
+```
+
+---
\ No newline at end of file

From 27ac36036e96d5361a56d7f419d133f6dd53387f Mon Sep 17 00:00:00 2001
From: George Rahul <75750164+georgerahul24@users.noreply.github.com>
Date: Fri, 24 Apr 2026 03:38:20 +0530
Subject: [PATCH 3/8] Completed the hashing indexes. Left with the tree ones

---
 .../projects/indexing/chainhash.md            |   2 +-
 .../projects/indexing/extendiblehashing.md    | 156 ++++++++++++
 .../projects/indexing/linearhash.md           | 228 ++++++++++++++++++
 .../projects/indexing/statichash.md           |   1 -
 4 files changed, 385 insertions(+), 2 deletions(-)
 create mode 100644 content/storage-engine/projects/indexing/extendiblehashing.md
 create mode 100644 content/storage-engine/projects/indexing/linearhash.md

diff --git a/content/storage-engine/projects/indexing/chainhash.md b/content/storage-engine/projects/indexing/chainhash.md
index d702b7c..584594b 100644
--- a/content/storage-engine/projects/indexing/chainhash.md
+++ b/content/storage-engine/projects/indexing/chainhash.md
@@ -2,7 +2,7 @@
 title: Chain Hash
 sidebar_position: 5
 ---
-## Chained Hash Index — Code-Based Documentation
+
 
 ### Algorithm
 
diff --git a/content/storage-engine/projects/indexing/extendiblehashing.md b/content/storage-engine/projects/indexing/extendiblehashing.md
new file mode 100644
index 0000000..dbe0210
--- /dev/null
+++ b/content/storage-engine/projects/indexing/extendiblehashing.md
@@ -0,0 +1,156 @@
+---
+title: Extendible Hash
+sidebar_position: 6
+---
+
+### Algorithm
+
+* Extendible hashing with:
+
+  * **Global depth (`global_depth`)**
+  * **Directory (`Vec<usize>`)** mapping to buckets
+  * **Buckets (`Vec<EHBucket>`)** each with **local depth (`local_depth`)**
+* On overflow:
+
+  * If `local_depth < global_depth` → split bucket
+  * If `local_depth == global_depth` → double directory, then split
+* Directory slots may point to the same bucket (shared buckets)
+
+---
+
+### Basic Idea
+
+* Use lower `global_depth` bits of `hash_code()` to index directory
+* Directory points to buckets
+* Buckets store `(key → Vec<RecordId>)`
+* Growth handled by:
+
+  * Directory doubling
+  * Bucket splitting
+* Entries redistributed after split using updated hash mapping
+
+---
+
+### Time Complexity
+
+* **Lookup:** `O(1)`
+* **Insert:** `O(1)` amortised, `O(n)` during rare directory doubling
+* **Delete:** `O(1)`
+* **Update (insert existing key):** `O(1)`
+
+
+---
+
+### Space Complexity
+
+* `O(n)`
+* Includes:
+
+  * Directory (`2^global_depth`)
+  * Buckets
+  * Entries and record lists
+
+
+---
+
+### Metadata Storage Format
+
+* Entire structure is **serialized/deserialized using `serde`**
+* Stored fields:
+
+  * `global_depth: u32`
+  * `directory: Vec<usize>`
+  * `buckets: Vec<EHBucket>`
+* Each `EHBucket`:
+
+  * `local_depth: u32`
+  * `entries: Vec<EHEntry>`
+* Each `EHEntry`:
+
+  * `key: IndexKey`
+  * `records: Vec<RecordId>`
+
+---
+
+### Data Storage (Persistent)
+
+* Uses:
+
+  * `paged_store::save_entries(...)`
+  * `paged_store::load_entries_stream(...)`
+* Storage format:
+
+  * Flat stream of `(IndexKey, RecordId)`
+* On load:
+
+  * Reconstructed via repeated `insert()` calls
+
+---
+
+### Hashing / Modulo Function
+
+* Directory index:
+
+  ```rust
+  (key.hash_code() as usize) & ((1 << global_depth) - 1)
+  ```
+* Uses **bit masking (modulo by power of 2)** on hash
+
+---
+
+### Collision Handling
+
+* Multiple keys in same bucket → stored in `entries: Vec<EHEntry>`
+* Same key:
+
+  * Multiple `RecordId`s stored in `records: Vec<RecordId>`
+* Overflow handled by:
+
+  * Bucket splitting
+  * Directory expansion
+
+---
+
+### Operations
+
+#### Search
+
+* Compute directory index
+* Lookup bucket
+* Linear scan in bucket (`find`)
+* Return matching record list
+
+#### Insert
+
+* Compute directory index
+* If key exists → append `record_id` (no duplicates)
+* If bucket not full → insert
+* If full:
+
+  * Split bucket
+  * Retry insert (loop)
+
+#### Delete
+
+* Locate bucket
+* Find key
+* Remove matching `record_id` using `retain`
+* Returns success if deletion happened
+
+#### Update
+
+* Same as insert for existing key:
+
+  * Adds new `record_id` if not present
+
+---
+
+### Additional Structural Rules
+
+* Directory size = `2^global_depth`
+* Bucket `local_depth <= global_depth`
+* Bucket capacity enforced via `EXTENDIBLE_HASH_BUCKET_CAPACITY`
+* Directory may have duplicate bucket references
+* Entry must have non-empty `records`
+
+---
diff --git a/content/storage-engine/projects/indexing/linearhash.md b/content/storage-engine/projects/indexing/linearhash.md
new file mode 100644
index 0000000..6638fed
--- /dev/null
+++ b/content/storage-engine/projects/indexing/linearhash.md
@@ -0,0 +1,228 @@
+---
+title: Chain Hash
+sidebar_position: 6
+---
+
+
+### Basic Idea
+
+Incremental hash-based indexing using **linear hashing**.
+Buckets are split one at a time using a **split pointer (`split_ptr`)** and a **level (`level`)**, avoiding full rehashing. 
+
+* Initial buckets: `N₀`
+* Buckets grow dynamically by splitting
+* Split triggered when load factor exceeds threshold
+* Entries redistributed only for the split bucket
+
+---
+
+### Hashing / Modulo Function
+
+Two hash levels are used:
+
+* Level `l`:
+
+  ```
+  h_l(k) = hash(k) mod (N₀ · 2^l)
+  ```
+* If bucket index `< split_ptr`, use:
+
+  ```
+  h_{l+1}(k) = hash(k) mod (N₀ · 2^{l+1})
+  ```
+
+Bucket selection:
+
+```
+if h_l(k) < split_ptr:
+    use h_{l+1}(k)
+else:
+    use h_l(k)
+```
+
+
+
+---
+
+### Time Complexity
+
+* **Insert**:
+  Average: O(1)
+  Worst: O(n) (during split + overflow traversal)
+
+* **Search**:
+  Average: O(1)
+  Worst: O(n) (overflow chains)
+
+* **Delete**:
+  Average: O(1)
+  Worst: O(n)
+
+* **Index Creation (load)**:
+  O(n) (sequential inserts from paged store)
+
+* **Update**:
+  Same as insert/delete
+
+---
+
+### Space Complexity
+
+* Buckets + overflow segments store all entries
+* Total space: **O(n)**
+* Additional overhead:
+
+  * Overflow segments
+  * Dynamic bucket growth
+
+---
+
+### Metadata Storage Format
+
+Stored implicitly via struct serialization:
+
+```
+LinearHashIndex {
+    level: u32
+    split_ptr: usize
+    initial_buckets: usize
+    buckets: Vec<LHBucket>
+    load_factor_threshold: f64
+}
+```
+
+Saved using:
+
+```
+paged_store::save_entries(...)
+```
+
+Loaded using:
+
+```
+paged_store::load_entries_stream(...)
+```
+
+
+
+---
+
+### Data Storage Format (Buckets)
+
+Each bucket:
+
+```
+LHBucket {
+    entries: Vec<LHEntry>
+    overflow: Vec<OverflowSegment>
+}
+```
+
+Entry:
+
+```
+LHEntry {
+    key: IndexKey
+    records: Vec<RecordId>
+}
+```
+
+Overflow:
+
+```
+OverflowSegment {
+    entries: Vec<LHEntry>
+}
+```
+
+
+
+---
+
+### Collision Handling
+
+* Primary bucket stores up to `STATIC_HASH_BUCKET_CAPACITY`
+* On overflow:
+
+  * Entries added to last overflow segment if space exists
+  * Else new overflow segment created
+* Multiple overflow segments form a chain
+
+---
+
+### Splitting Mechanism
+
+* Trigger: `load_factor() > threshold`
+* Steps:
+
+  1. Create new bucket
+  2. Drain entries from bucket at `split_ptr`
+  3. Rehash using `h_{l+1}`
+  4. Redistribute entries
+  5. Increment `split_ptr`
+  6. If end of round:
+
+     * `level += 1`
+     * `split_ptr = 0` 
+
+---
+
+### Operations
+
+#### Search
+
+* Compute bucket index using `bucket_for`
+* Search in:
+
+  * primary entries
+  * overflow segments
+* Return matching `RecordId`s
+
+---
+
+#### Insert
+
+* Compute bucket index
+* Insert into:
+
+  * existing entry if key exists
+  * else new entry
+* Handle overflow if needed
+* Trigger split if load factor exceeded
+
+---
+
+#### Delete
+
+* Locate entry via bucket
+* Remove `RecordId` from entry
+* Returns success if deletion occurred
+
+---
+
+#### Update
+
+* Not explicitly implemented
+* Achieved via:
+
+  * delete + insert
+
+---
+
+### Load Factor
+
+```
+load_factor = total_records / (bucket_count × bucket_capacity)
+```
+
+---
+
+### Additional Notes from Code
+
+* Duplicate `RecordId`s are avoided per key
+* Overflow segments must not exceed capacity
+* Structure validation ensures:
+
+  * valid split pointer
+  * non-empty records
+  * capacity constraints 
diff --git a/content/storage-engine/projects/indexing/statichash.md b/content/storage-engine/projects/indexing/statichash.md
index 45a657f..d866275 100644
--- a/content/storage-engine/projects/indexing/statichash.md
+++ b/content/storage-engine/projects/indexing/statichash.md
@@ -2,7 +2,6 @@
 title: Static Hash
 sidebar_position: 4
 ---
-## Static Hash Index – Concise Documentation (from code)
 
 ### 1. Algorithm Overview
 

From e1bd9c6e52f5b49dd42a86fd6d197dcf175ea35a Mon Sep 17 00:00:00 2001
From: George Rahul <75750164+georgerahul24@users.noreply.github.com>
Date: Fri, 24 Apr 2026 03:55:08 +0530
Subject: [PATCH 4/8] Added btree bplus tree and radix tree

---
 .../projects/indexing/bplustree.md            | 184 +++++++++++++++++
 .../storage-engine/projects/indexing/btree.md | 185 ++++++++++++++++++
 .../projects/indexing/catalogintegeration.md  |   2 +-
 .../projects/indexing/chainhash.md            |   2 +-
 .../projects/indexing/extendiblehashing.md    |   2 +-
 .../projects/indexing/lsmtree.md              |   4 +
 .../projects/indexing/radixtree.md            | 147 ++++++++++++++
 .../projects/indexing/skiplist.md             |   4 +
 .../projects/indexing/statichash.md           |   2 +-
 9 files changed, 528 insertions(+), 4 deletions(-)
 create mode 100644 content/storage-engine/projects/indexing/bplustree.md
 create mode 100644 content/storage-engine/projects/indexing/btree.md
 create mode 100644 content/storage-engine/projects/indexing/lsmtree.md
 create mode 100644 content/storage-engine/projects/indexing/radixtree.md
 create mode 100644 content/storage-engine/projects/indexing/skiplist.md

diff --git a/content/storage-engine/projects/indexing/bplustree.md b/content/storage-engine/projects/indexing/bplustree.md
new file mode 100644
index 0000000..c5b081b
--- /dev/null
+++ b/content/storage-engine/projects/indexing/bplustree.md
@@ -0,0 +1,184 @@
+---
+title: B+ Tree 
+sidebar_position: 8
+---
+
+### Algorithm
+
+* B+ Tree with minimum degree `t`
+* Internal nodes store only routing keys and child pointers
+* Leaf nodes store `(key, Vec<RecordId>)`
+* Leaves are linked using `next_leaf` pointer (singly linked forward)
+* Nodes stored in an arena (`Vec<BPlusNode>`) with stable indices
+* All leaves are at the same depth
+* Node capacity:
+
+  * Max keys: `2t - 1`
+  * Min keys (non-root): `t - 1`
+
+---
+
+### Basic Idea
+
+* Search descends from root to leaf using binary partition (`partition_point`)
+* Insert happens at leaf; overflow triggers split and upward propagation
+* Delete removes `(key, rid)` and handles underflow via borrow/merge
+* Range scan uses linked leaves after initial descent
+
+---
+
+### Time Complexity
+
+* Search: `O(t · log_t n)`
+* Insert: `O(t · log_t n)`
+* Delete: `O(t · log_t n)`
+* Range scan: `O(log n + k)`
+* Index creation (bulk via inserts): same as repeated insert
+
+---
+
+### Space Complexity
+
+* In-memory:
+
+  * `O(n)` nodes stored in `Vec<BPlusNode>`
+* On-disk:
+
+  * One page per node (`PAGE_SIZE`)
+  * Total pages = number of nodes + 1 header page
+
+---
+
+### Metadata File Storage Format
+
+* First page (header page):
+
+  * Bytes 0–8: magic `"RDBIDXV1"`
+  * Bytes 8–10: version
+  * Bytes 10–12: header size
+  * Bytes 12–16: page size
+  * Bytes 16–20: root page
+  * Bytes 20–24: node page count
+  * Bytes 24–32: entry count
+  * Bytes 32–36: minimum degree `t`
+
+---
+
+### Data Storage Format (Per Node Page)
+
+* Fixed-size page (`PAGE_SIZE`)
+* Header (16 bytes):
+
+  * Byte 0: `is_leaf`
+  * Byte 1: `dead`
+  * Bytes 2–4: key count
+  * Bytes 4–6: child count
+  * Bytes 8–12: `next_leaf_page`
+  * Bytes 12–16: payload size
+* Payload:
+
+  * Keys (encoded sequentially)
+  * If leaf:
+
+    * For each key:
+
+      * `rid_count`
+      * List of `(page_no, item_id)`
+  * If internal:
+
+    * Child page numbers
+
+---
+
+### Hashing / Modulo Function
+
+* Not used
+
+---
+
+### Collision Handling
+
+* Multiple records per key handled via:
+
+  * `values: Vec<Vec<RecordId>>`
+  * Same key maps to a vector of record IDs
+
+---
+
+### Operations
+
+#### Search
+
+* Traverse from root using `partition_point`
+* At leaf:
+
+  * Binary position lookup
+  * Return `Vec<RecordId>` or empty
+
+#### Insert
+
+* Insert into leaf at sorted position
+* If key exists:
+
+  * Append `RecordId` if not duplicate
+* If overflow (`>= 2t` keys):
+
+  * Split leaf
+  * Push first key of right node upward
+  * Recursively split internal nodes if needed
+* Root split creates new root
+
+#### Delete
+
+* Locate leaf and key
+* Remove specific `RecordId`
+* If key has no more records:
+
+  * Remove key
+* If underflow (`< t-1` keys):
+
+  * Borrow from sibling OR merge
+* Propagate fixes upward
+* If root becomes empty:
+
+  * Replace with single child
+
+#### Update
+
+* Not explicitly implemented
+* Equivalent to:
+
+  * `delete(key, old_rid)` + `insert(key, new_rid)`
+
+---
+
+### Disk Operations
+
+* Save:
+
+  * Header page + serialized node pages
+* Load:
+
+  * Parse header
+  * Load all node pages into memory
+* Direct search:
+
+  * `search_on_disk()` performs traversal without full load
+
+---
+
+### Additional Notes
+
+* Node splitting:
+
+  * Leaf: split at index `t`
+  * Internal: median key moves up
+* Leaf linking maintained during split/merge
+* Dead nodes marked but not reused
+* Validation ensures:
+
+  * Sorted keys
+  * Proper child counts
+  * No cycles in tree or leaf chain
+
+---
\ No newline at end of file
diff --git a/content/storage-engine/projects/indexing/btree.md b/content/storage-engine/projects/indexing/btree.md
new file mode 100644
index 0000000..9671107
--- /dev/null
+++ b/content/storage-engine/projects/indexing/btree.md
@@ -0,0 +1,185 @@
+---
+title: BTree 
+sidebar_position: 7
+---
+
+
+### Algorithm
+
+* Classic B-Tree (Knuth / CLRS definition).
+* Minimum degree `t`.
+* Each node:
+
+  * Stores `keys: Vec<IndexKey>`
+  * Stores `values: Vec<Vec<RecordId>>` (multiple RIDs per key)
+  * Stores `children: Vec<usize>` (indices into arena)
+  * `is_leaf`, `dead` flags
+* Nodes stored in a flat arena: `Vec<BTreeNode>`
+* Root tracked by index.
+* All leaves at same depth.
+* Internal node: `k` keys → `k+1` children.
+* Node capacity: `[t−1, 2t−1]` keys (except root).
+
+---
+
+### Basic Idea
+
+* Tree-based index storing keys in sorted order.
+* Each key maps to one or more record IDs.
+* Uses balanced multi-way tree.
+* Insert splits full nodes.
+* Delete uses merge/rotation (CLRS algorithm).
+* Arena-based storage using indices instead of pointers. 
+
+---
+
+### Time Complexity
+
+(from code comments)
+
+| Operation  | Complexity         |
+| ---------- | ------------------ |
+| search     | O(t · log_t n)     |
+| insert     | O(t · log_t n)     |
+| delete     | O(t · log_t n)     |
+| range_scan | O(t · log_t n + k) |
+
+---
+
+### Space Complexity
+
+* Nodes stored in `Vec<BTreeNode>` (arena).
+* Each node holds up to `2t−1` keys.
+* Deleted nodes remain as tombstones (`dead = true`).
+* Overall: **O(n)** space.
+
+---
+
+### Metadata Storage Format
+
+* Entire structure serialized using `serde::{Serialize, Deserialize}`.
+* Persistence:
+
+  * `save()` → `paged_store::save_entries(...)`
+  * `load()` → `paged_store::load_entries_stream(...)`
+* No explicit separate metadata file structure defined in code.
+* Metadata implicitly:
+
+  * `nodes` vector
+  * `root` index
+  * `t`
+  * `entry_count`
+
+---
+
+### Data Storage Format
+
+* Stored as `(IndexKey, RecordId)` pairs via paged store.
+* On load:
+
+  * Entries streamed and inserted one-by-one.
+* In-memory:
+
+  * Keys stored in nodes
+  * Values stored as `Vec<RecordId>` per key
+* Tree reconstructed from entries (no direct node serialization in load).
+
+---
+
+### Hashing / Modulo Function
+
+* Not used.
+* Tree-based index (ordered structure).
+
+---
+
+### Collision Handling
+
+* Multiple records for same key stored as:
+
+  * `values[i] = Vec<RecordId>`
+* Duplicate keys not duplicated structurally.
+* Record IDs appended if key exists.
+
+---
+
+### Operations
+
+#### Search
+
+* Binary search using `partition_point`.
+* If key found → return `values[pos]`.
+* Else descend to correct child.
+* Leaf → return empty.
+
+---
+
+#### Insert
+
+* If root full → split.
+* Use `insert_non_full`:
+
+  * If key exists → append RID if not present.
+  * Else insert key + new RID.
+* Child split done before descending if full.
+
+---
+
+#### Delete
+
+* Two-step:
+
+  1. Remove specific `(key, rid)` from values.
+  2. If values empty → remove key structurally.
+* Structural deletion:
+
+  * Uses CLRS algorithm:
+
+    * Replace with predecessor/successor
+    * Rotate (left/right)
+    * Merge children
+* Nodes merged → right node marked `dead`.
+
+---
+
+#### Update
+
+* Not explicitly implemented.
+* Equivalent to:
+
+  * delete(old key, rid)
+  * insert(new key, rid)
+
+---
+
+#### Range Scan
+
+* In-order traversal:
+
+  * Visit children and keys in sorted order.
+  * Collect RIDs within `[start, end]`.
+
+---
+
+#### Validation
+
+* Ensures:
+
+  * Keys sorted
+  * Values match keys
+  * No empty RID lists
+  * Correct child count
+  * No cycles
+  * No reachable dead nodes
+
+---
+
+#### Additional Notes
+
+* Arena-based node storage using indices.
+* Tombstoned nodes not reclaimed.
+* Supports:
+
+  * `min_key()`
+  * `max_key()`
+  * full traversal (`all_entries`)
diff --git a/content/storage-engine/projects/indexing/catalogintegeration.md b/content/storage-engine/projects/indexing/catalogintegeration.md
index 38ec031..e269b2a 100644
--- a/content/storage-engine/projects/indexing/catalogintegeration.md
+++ b/content/storage-engine/projects/indexing/catalogintegeration.md
@@ -1,6 +1,6 @@
 ---
 title: Catalog Integration
-sidebar_position: 3
+sidebar_position: 10
 ---
 
 
diff --git a/content/storage-engine/projects/indexing/chainhash.md b/content/storage-engine/projects/indexing/chainhash.md
index 584594b..9061f84 100644
--- a/content/storage-engine/projects/indexing/chainhash.md
+++ b/content/storage-engine/projects/indexing/chainhash.md
@@ -1,6 +1,6 @@
 ---
 title: Chain Hash
-sidebar_position: 5
+sidebar_position: 4
 ---
 
 
diff --git a/content/storage-engine/projects/indexing/extendiblehashing.md b/content/storage-engine/projects/indexing/extendiblehashing.md
index dbe0210..3b19232 100644
--- a/content/storage-engine/projects/indexing/extendiblehashing.md
+++ b/content/storage-engine/projects/indexing/extendiblehashing.md
@@ -1,6 +1,6 @@
 ---
 title: Extendible Hash
-sidebar_position: 6
+sidebar_position: 5
 ---
 
 ### Algorithm
diff --git a/content/storage-engine/projects/indexing/lsmtree.md b/content/storage-engine/projects/indexing/lsmtree.md
new file mode 100644
index 0000000..590e020
--- /dev/null
+++ b/content/storage-engine/projects/indexing/lsmtree.md
@@ -0,0 +1,4 @@
+---
+title: LSM Tree
+sidebar_position: 11
+---
\ No newline at end of file
diff --git a/content/storage-engine/projects/indexing/radixtree.md b/content/storage-engine/projects/indexing/radixtree.md
new file mode 100644
index 0000000..edb9c7c
--- /dev/null
+++ b/content/storage-engine/projects/indexing/radixtree.md
@@ -0,0 +1,147 @@
+---
+title: Radix Tree
+sidebar_position: 9
+---
+
+
+### Algorithm
+
+A radix tree (compressed trie) stores keys as byte sequences. Consecutive single-child nodes are merged into a single node with a multi-byte prefix. Each node represents a prefix of keys and branches based on the next byte.
+
+Insertion uses longest common prefix (LCP) comparison:
+
+* If partial match → node split.
+* If full match → descend or mark terminal.
+
+Search and delete traverse using prefix matching.
+
+---
+
+### Basic Idea
+
+* Keys are converted to byte arrays using `IndexKey::as_bytes()` 
+* Tree stores compressed prefixes instead of single characters
+* Each node:
+
+  * `prefix`: compressed edge label
+  * `children`: sorted map (byte → node)
+  * `terminal`: optional `(key, [record_ids])`
+
+---
+
+### Time Complexity
+
+Let `k = key length in bytes`
+
+| Operation      | Complexity        |
+| -------------- | ----------------- |
+| Index Creation | O(n · k)          |
+| Insert         | O(k)              |
+| Search         | O(k)              |
+| Delete         | O(k)              |
+| Range Scan     | O(k + k · output) |
+
+---
+
+### Space Complexity
+
+* O(total bytes of all keys)
+* Prefix compression reduces redundancy
+* Additional overhead:
+
+  * `BTreeMap` per node
+  * RecordId vectors at terminals
+
+---
+
+### Metadata Storage Format
+
+* No explicit metadata file structure defined
+* Tree is reconstructed using:
+
+  * `paged_store::load_entries_stream(path, ...)` 
+* Entries are streamed and inserted into the tree
+
+---
+
+### Data Storage Format
+
+* Stored via:
+
+  * `paged_store::save_entries(path, iterator)` 
+* Format:
+
+  * Sequence of `(IndexKey, RecordId)` pairs
+* Tree structure itself is **not serialized directly**
+* Rebuilt by replaying inserts
+
+---
+
+### Hashing / Modulo Function
+
+* Not used
+* Structure is tree-based, not hash-based
+
+---
+
+### Collision Handling
+
+* Not applicable (no hashing)
+* Multiple records per key handled via:
+
+  * `terminal: Option<(IndexKey, Vec<RecordId>)>`
+
+---
+
+### Operations
+
+#### Search
+
+* Traverse using prefix matching
+* If full match and terminal exists → return record IDs
+* Else → return empty
+
+#### Insert
+
+* Compute LCP with node prefix
+* Cases:
+
+  * Partial match → split node
+  * Full match → descend or create child
+  * Exact match → append RecordId (no duplicates)
+
+#### Delete
+
+* Traverse to terminal node
+* Remove specific RecordId
+* If empty:
+
+  * Remove terminal
+  * If node becomes empty → prune
+  * If single child → compress (merge)
+
+#### Update
+
+* Not explicitly implemented
+* Equivalent to:
+
+  * Delete(old_key, rid)
+  * Insert(new_key, rid)
+
+---
+
+### Additional Behavior
+
+* Children stored in `BTreeMap` → maintains sorted order
+* Enables correct lexicographic range scans
+* Range scan:
+
+  * Recursively collects keys within `[start, end]`
+* Supports:
+
+  * `min_key()`
+  * `max_key()`
+  * `entry_count()`
+  * structural validation
+
+---
diff --git a/content/storage-engine/projects/indexing/skiplist.md b/content/storage-engine/projects/indexing/skiplist.md
new file mode 100644
index 0000000..3190ad1
--- /dev/null
+++ b/content/storage-engine/projects/indexing/skiplist.md
@@ -0,0 +1,4 @@
+---
+title: Skip List
+sidebar_position: 10
+---
\ No newline at end of file
diff --git a/content/storage-engine/projects/indexing/statichash.md b/content/storage-engine/projects/indexing/statichash.md
index d866275..eb28843 100644
--- a/content/storage-engine/projects/indexing/statichash.md
+++ b/content/storage-engine/projects/indexing/statichash.md
@@ -1,6 +1,6 @@
 ---
 title: Static Hash
-sidebar_position: 4
+sidebar_position: 3
 ---
 
 ### 1. Algorithm Overview

From 26848f8846244f2e1712604ebc076872abd23269 Mon Sep 17 00:00:00 2001
From: George Rahul <75750164+georgerahul24@users.noreply.github.com>
Date: Fri, 24 Apr 2026 03:58:40 +0530
Subject: [PATCH 5/8] The tree ones are also done for hashing

---
 .../projects/indexing/catalogintegeration.md  |   2 +-
 .../projects/indexing/lsmtree.md              |  98 +++++++++++++++-
 .../projects/indexing/skiplist.md             | 106 +++++++++++++++++-
 3 files changed, 203 insertions(+), 3 deletions(-)

diff --git a/content/storage-engine/projects/indexing/catalogintegeration.md b/content/storage-engine/projects/indexing/catalogintegeration.md
index e269b2a..fdaac4e 100644
--- a/content/storage-engine/projects/indexing/catalogintegeration.md
+++ b/content/storage-engine/projects/indexing/catalogintegeration.md
@@ -1,6 +1,6 @@
 ---
 title: Catalog Integration
-sidebar_position: 10
+sidebar_position: 12
 ---
 
 
diff --git a/content/storage-engine/projects/indexing/lsmtree.md b/content/storage-engine/projects/indexing/lsmtree.md
index 590e020..e47ab8b 100644
--- a/content/storage-engine/projects/indexing/lsmtree.md
+++ b/content/storage-engine/projects/indexing/lsmtree.md
@@ -1,4 +1,100 @@
 ---
 title: LSM Tree
 sidebar_position: 11
----
\ No newline at end of file
+---
+
+**Algorithm:**
+
+* In-memory **memtable**: `BTreeMap<IndexKey, Vec<RecordId>>`
+* Immutable **runs**: `Vec<LsmRun>` where each run stores a `BTreeMap<IndexKey, Vec<RecordId>>` 
+* Insert into memtable → flush to runs when `memtable_limit` reached → optional full compaction when runs > 8 
+
+---
+
+**Basic Idea:**
+
+* Writes go to memtable (sorted BTreeMap).
+* When full, memtable is flushed as a new run (immutable).
+* Reads check memtable first, then runs (newest to oldest).
+* Compaction merges all runs into one, keeping latest values. 
+
+---
+
+**Time Complexity:**
+
+* Insert:
+
+  * `O(log n)` (BTreeMap insert) + occasional flush `O(n)`
+* Search:
+
+  * `O(log n)` (memtable) + `O(R * log n)` worst-case over runs
+* Delete:
+
+  * Same as search + `O(k)` for filtering record IDs
+* Compaction:
+
+  * `O(total_entries)`
+
+---
+
+**Space Complexity:**
+
+* `O(N)` total across memtable + runs
+* Temporary `O(N)` during compaction
+
+---
+
+**Metadata Storage Format:**
+
+* No explicit separate metadata file
+* Entire index reconstructed using:
+
+  * `paged_store::load_entries_stream(path, |key, rid| insert(...))` 
+
+---
+
+**Data Storage Format:**
+
+* Stored as flat `(IndexKey, RecordId)` pairs via:
+
+  * `paged_store::save_entries(path, iterator)` 
+* Logical structure (memtable + runs) is rebuilt on load
+
+---
+
+**Hashing / Modulo:**
+
+* Not used
+* Uses ordered `BTreeMap`
+
+---
+
+**Collision Handling:**
+
+* Multiple `RecordId`s per key stored as `Vec<RecordId>`
+* Duplicate prevention via `if !list.contains(&record_id)` 
+
+---
+
+**Operations:**
+
+* **Search:**
+
+  * Check memtable → then runs in order → return first match
+
+* **Insert:**
+
+  * Add to memtable
+  * Avoid duplicate record IDs
+  * Trigger flush if limit reached
+
+* **Delete:**
+
+  * If in memtable → remove directly
+  * Else → fetch current values → rewrite updated version into memtable
+
+* **Update:**
+
+  * Not explicit; achieved via insert/delete combination
+
+---
diff --git a/content/storage-engine/projects/indexing/skiplist.md b/content/storage-engine/projects/indexing/skiplist.md
index 3190ad1..99a5973 100644
--- a/content/storage-engine/projects/indexing/skiplist.md
+++ b/content/storage-engine/projects/indexing/skiplist.md
@@ -1,4 +1,108 @@
 ---
 title: Skip List
 sidebar_position: 10
----
\ No newline at end of file
+---
+
+**Basic Idea**
+
+* Implements an ordered index using an in-memory `BTreeMap<IndexKey, Vec<RecordId>>`.
+* Each key maps to a list of record IDs to support duplicate keys.
+* Despite the name, it is not an actual probabilistic skip list; ordering and range behavior are provided by `BTreeMap`.
+
+---
+
+**Time Complexity**
+
+* **Insert:** `O(log N + K)`
+
+  * `log N` for map insertion, `K` for checking duplicates in vector.
+* **Search:** `O(log N + K)`
+
+  * `log N` for lookup, `K` for returning all record IDs.
+* **Delete:** `O(log N + K)`
+
+  * `log N` for lookup, `K` for filtering vector.
+* **Range Scan:** `O(log N + M)`
+
+  * `log N` to locate range start, `M` total elements in range.
+* **Index Creation (load):** `O(T log N)`
+
+  * Inserts all entries sequentially using `insert`.
+
+---
+
+**Space Complexity**
+
+* `O(N + T)`
+
+  * `N` = number of unique keys
+  * `T` = total number of `(key, record_id)` pairs
+
+---
+
+**Metadata Storage Format**
+
+* No explicit metadata structure is defined.
+* No separate metadata file is created or managed.
+
+---
+
+**Data Storage Format**
+
+* Stored using `paged_store::save_entries` as a stream of `(IndexKey, RecordId)` pairs.
+* Loading is performed via `paged_store::load_entries_stream`, reconstructing the index by reinserting entries.
+* Data is not stored as a tree or levels; it is serialized as flat key–record pairs.
+
+---
+
+**Hashing / Modulo Function**
+
+* Not used.
+* Index is tree-based (`BTreeMap`), not hash-based.
+
+---
+
+**Collision Handling**
+
+* Multiple records for the same key are stored in a `Vec<RecordId>`.
+* Duplicate `(key, record_id)` pairs are prevented by checking `list.contains()` before insertion.
+
+---
+
+**Operations**
+
+* **Insert**
+
+  * Adds `record_id` to the vector for a key.
+  * Avoids duplicates.
+
+* **Search**
+
+  * Returns all record IDs for a given key.
+  * Returns empty vector if key is absent.
+
+* **Delete**
+
+  * Removes specific `record_id` from the key’s vector.
+  * Removes key entirely if no record IDs remain.
+
+* **Update**
+
+  * Not explicitly implemented.
+  * Achieved via delete + insert.
+
+* **Range Scan**
+
+  * Uses ordered traversal via `BTreeMap::range`.
+  * Returns concatenated record IDs within `[start, end]`.
+
+* **Min / Max Key**
+
+  * Retrieved using ordered key iterators (`next`, `next_back`).
+
+---
+
+**Validation**
+
+* Ensures no key has an empty `Vec<RecordId>`.
+* Returns error if such a case is found.

From 1eb36bf0f61b1e0c3b88c2d43e6a396600f245f7 Mon Sep 17 00:00:00 2001
From: George Rahul <75750164+georgerahul24@users.noreply.github.com>
Date: Sat, 25 Apr 2026 00:01:08 +0530
Subject: [PATCH 6/8] Add files via upload

---
 .../indexing/indexing_api_endpoints.md        | 435 ++++++++++++++++++
 1 file changed, 435 insertions(+)
 create mode 100644 content/storage-engine/projects/indexing/indexing_api_endpoints.md

diff --git a/content/storage-engine/projects/indexing/indexing_api_endpoints.md b/content/storage-engine/projects/indexing/indexing_api_endpoints.md
new file mode 100644
index 0000000..af05015
--- /dev/null
+++ b/content/storage-engine/projects/indexing/indexing_api_endpoints.md
@@ -0,0 +1,435 @@
+# RookDB Indexing API Endpoints (Integration Reference)
+
+This document lists the important indexing-related APIs that other teams can call directly from the `storage_manager` crate, based on the current codebase.
+
+## 1. Module Import Map
+
+The crate root re-exports backend modules, so callers can use these paths directly:
+
+- `storage_manager::catalog::*` for catalog metadata operations
+- `storage_manager::index::*` for index build, load, search, rebuild, validation, and maintenance helpers
+- `storage_manager::executor::{index_scan, index_scan_by_column}` for tuple fetch via index
+- `storage_manager::heap::{insert_tuple_with_index_maintenance, delete_tuple_with_index_maintenance}` for write path maintenance
+
+## 2. Core Data Contracts
+
+### `IndexAlgorithm`
+
+Supported values:
+
+- `StaticHash`
+- `ChainedHash`
+- `ExtendibleHash`
+- `LinearHash`
+- `BTree`
+- `BPlusTree`
+- `RadixTree`
+- `SkipList`
+- `LsmTree`
+
+Helpers:
+
+- `is_hash()` and `is_tree()`
+- `from_str(...)` supports aliases like `btree`, `bplus_tree`, `linear_hash`, etc.
+
+### `IndexEntry`
+
+Catalog metadata for an index:
+
+- `index_name: String`
+- `column_name: Vec<String>` (single or composite)
+- `algorithm: IndexAlgorithm`
+- `is_clustered: bool`
+- `include_columns: Vec<String>`
+
+Useful helper:
+
+- `is_secondary()` is `!is_clustered`
+
+### `IndexKey` and `RecordId`
+
+- `IndexKey` variants: `Int(i64)`, `Float(f64)`, `Text(String)`
+- `RecordId`: `{ page_no: u32, item_id: u32 }`
+
+## 3. Catalog-Level Index Endpoints (Metadata)
+
+These APIs register/remove index metadata in catalog and persist `catalog.json`.
+
+### `create_index`
+
+```rust
+pub fn create_index(
+    catalog: &mut Catalog,
+    db_name: &str,
+    table_name: &str,
+    index_name: &str,
+    column_names: &[String],
+    algorithm: IndexAlgorithm,
+    is_clustered: bool,
+    include_columns: Vec<String>,
+) -> bool
+```
+
+Behavior:
+
+- Validates DB/table existence
+- Validates indexed and include columns
+- Prevents duplicate index names per table
+- Enforces max one clustered index per table
+- Appends `IndexEntry` and saves catalog
+
+Important:
+
+- This does **not** build or save an `.idx` file. Build step must be called separately.
+
+### `create_secondary_index`
+
+```rust
+pub fn create_secondary_index(
+    catalog: &mut Catalog,
+    db_name: &str,
+    table_name: &str,
+    index_name: &str,
+    column_names: &[String],
+    algorithm: IndexAlgorithm,
+) -> io::Result<()>
+```
+
+Behavior:
+
+- Wrapper over `create_index(..., is_clustered = false, include_columns = vec![])`
+
+### `drop_index`
+
+```rust
+pub fn drop_index(
+    catalog: &mut Catalog,
+    db_name: &str,
+    table_name: &str,
+    index_name: &str,
+) -> bool
+```
+
+Behavior:
+
+- Removes index metadata entry from table and saves catalog
+
+Important:
+
+- Does **not** delete index file from disk; remove `.idx` manually if desired.
+
+### `drop_secondary_index`
+
+```rust
+pub fn drop_secondary_index(
+    catalog: &mut Catalog,
+    db_name: &str,
+    table_name: &str,
+    index_name: &str,
+) -> io::Result<()>
+```
+
+Behavior:
+
+- Ensures index exists and is non-clustered
+- Delegates metadata removal to `drop_index`
+
+Important:
+
+- Also does **not** delete index file from disk.
+
+### Listing/lookup helpers
+
+```rust
+pub fn list_indexes<'a>(
+    catalog: &'a Catalog,
+    db_name: &str,
+    table_name: &str,
+) -> Option<&'a Vec<IndexEntry>>
+
+pub fn list_secondary_indices(
+    catalog: &Catalog,
+    db_name: &str,
+    table_name: &str,
+) -> io::Result<Vec<IndexEntry>>
+```
+
+### Catalog I/O used by integrations
+
+```rust
+pub fn load_catalog() -> Catalog
+pub fn save_catalog(catalog: &Catalog)
+```
+
+## 4. Index Build, Persistence, and Lookup Endpoints
+
+### `AnyIndex` constructors/loaders
+
+```rust
+pub fn new_empty(algorithm: &IndexAlgorithm) -> AnyIndex
+pub fn new_default(family: &str) -> AnyIndex
+pub fn load(path: &str, algorithm: &IndexAlgorithm) -> io::Result<AnyIndex>
+```
+
+### Build from existing table data
+
+```rust
+pub fn build_from_table(
+    catalog: &Catalog,
+    db_name: &str,
+    table_name: &str,
+    column_name: &str,
+    algorithm: &IndexAlgorithm,
+) -> io::Result<AnyIndex>
+
+pub fn build_from_table_columns(
+    catalog: &Catalog,
+    db_name: &str,
+    table_name: &str,
+    column_names: &[String],
+    algorithm: &IndexAlgorithm,
+) -> io::Result<AnyIndex>
+
+pub fn build_secondary_index(
+    catalog: &Catalog,
+    db_name: &str,
+    table_name: &str,
+    index_entry: &IndexEntry,
+) -> io::Result<AnyIndex>
+```
+
+Notes:
+
+- Supports composite index keys using `column_names`
+- Reads tuple bytes from table pages and extracts key(s)
+- Key extraction supports `INT`, `TEXT`, `BOOL/BOOLEAN`
+
+### Runtime index operations
+
+```rust
+pub fn insert(&mut self, key: IndexKey, record_id: RecordId) -> io::Result<()>
+pub fn search(&self, key: &IndexKey) -> io::Result<Vec<RecordId>>
+pub fn delete(&mut self, key: &IndexKey, record_id: &RecordId) -> io::Result<bool>
+pub fn save(&self, path: &str) -> io::Result<()>
+```
+
+### On-disk point search (without fully materializing index in caller)
+
+```rust
+pub fn search_on_disk(
+    path: &str,
+    algorithm: &IndexAlgorithm,
+    key: &IndexKey,
+) -> io::Result<Vec<RecordId>>
+```
+
+Implementation detail:
+
+- `BPlusTree` uses dedicated on-disk traversal
+- Other algorithms use paged-store search path
+
+### Tree-only range scan
+
+```rust
+pub fn range_scan(&self, start: &IndexKey, end: &IndexKey) -> io::Result<Vec<RecordId>>
+pub fn supports_range_scan(&self) -> bool
+```
+
+- Hash indexes return `Unsupported` for range scan
+
+## 5. Path and Key Utility Endpoints
+
+### Key conversion for lookup inputs
+
+```rust
+pub fn index_key_from_values(
+    columns: &[Column],
+    index_columns: &[String],
+    values: &[String],
+) -> io::Result<IndexKey>
+```
+
+Use this for all user/API-provided search keys, especially composite indexes.
+
+Important composite-key note:
+
+- Multi-column keys are encoded into a sortable hex string and represented as `IndexKey::Text(...)`
+- Do not handcraft composite keys; use this helper to avoid ordering/encoding mismatch
+
+### Canonical index file paths
+
+```rust
+pub fn index_file_path(db_name: &str, table_name: &str, index_name: &str) -> String
+pub fn secondary_index_file_path(db_name: &str, table_name: &str, index_name: &str) -> String
+```
+
+Current behavior:
+
+- Both helpers resolve to the same path shape: `database/base/{db}/{table}_{index}.idx`
+
+## 6. Rebuild, Validation, and Layout Endpoints
+
+### Rebuild APIs
+
+```rust
+pub fn rebuild_table_indexes(catalog: &Catalog, db_name: &str, table_name: &str) -> io::Result<usize>
+pub fn rebuild_secondary_index(
+    catalog: &Catalog,
+    db_name: &str,
+    table_name: &str,
+    index_name: &str,
+) -> io::Result<()>
+```
+
+- `rebuild_table_indexes` rebuilds every registered index on the table
+- `rebuild_secondary_index` refuses clustered indexes
+
+### Consistency validation APIs
+
+```rust
+pub fn validate_index_consistency(
+    catalog: &Catalog,
+    db_name: &str,
+    table_name: &str,
+    index_name: &str,
+) -> io::Result<()>
+
+pub fn validate_all_table_indexes(
+    catalog: &Catalog,
+    db_name: &str,
+    table_name: &str,
+) -> io::Result<usize>
+```
+
+Validation checks:
+
+- Algorithm structure invariants via `validate_structure()`
+- Full tuple scan vs index entries
+- Missing/stale entries and entry-count mismatch
+
+### Clustered physical layout API
+
+```rust
+pub fn maintain_clustered_index_layout(
+    catalog: &Catalog,
+    db_name: &str,
+    table_name: &str,
+) -> io::Result<bool>
+
+pub fn cluster_table_by_index(
+    catalog: &Catalog,
+    db_name: &str,
+    table_name: &str,
+    index_name: &str,
+) -> io::Result<()>
+```
+
+- `maintain_clustered_index_layout` auto-detects clustered index and reorders table if present
+- `cluster_table_by_index` rewrites table file in key order, then rebuilds all indexes
+
+## 7. Write-Path Maintenance Endpoints
+
+### Low-level index maintenance helpers
+
+```rust
+pub fn add_tuple_to_all_indexes(
+    catalog: &Catalog,
+    db_name: &str,
+    table_name: &str,
+    tuple: &[u8],
+    record_id: RecordId,
+) -> io::Result<usize>
+
+pub fn remove_tuple_from_all_indexes(
+    catalog: &Catalog,
+    db_name: &str,
+    table_name: &str,
+    tuple: &[u8],
+    record_id: RecordId,
+) -> io::Result<usize>
+```
+
+### Preferred heap APIs for inserts/deletes with index sync
+
+```rust
+pub fn insert_tuple_with_index_maintenance(
+    catalog: &Catalog,
+    db_name: &str,
+    table_name: &str,
+    file: &mut File,
+    data: &[u8],
+) -> io::Result<RecordId>
+
+pub fn delete_tuple_with_index_maintenance(
+    catalog: &Catalog,
+    db_name: &str,
+    table_name: &str,
+    file: &mut File,
+    record_id: RecordId,
+) -> io::Result<()>
+```
+
+These call the heap operation and index maintenance in one place.
+
+## 8. Query Endpoints Returning Tuples
+
+```rust
+pub fn index_scan(
+    catalog: &Catalog,
+    db_name: &str,
+    table_name: &str,
+    index_name: &str,
+    key: &IndexKey,
+) -> io::Result<Vec<Vec<u8>>>
+
+pub fn index_scan_by_column(
+    catalog: &Catalog,
+    db_name: &str,
+    table_name: &str,
+    column_name: &str,
+    key: &IndexKey,
+) -> io::Result<Vec<Vec<u8>>>
+```
+
+Behavior:
+
+- Uses catalog metadata to resolve algorithm and index file path
+- Probes index via `AnyIndex::search_on_disk`
+- Fetches tuples by `RecordId`
+- Prefers clustered index when multiple indexes exist on same column (`index_scan_by_column`)
+
+## 9. Integration Playbooks
+
+### A. Create and build a new index
+
+1. Load catalog with `load_catalog()`
+2. Register metadata using `create_index(...)` or `create_secondary_index(...)`
+3. Build using `AnyIndex::build_from_table_columns(...)`
+4. Persist with `idx.save(index_file_path(...))`
+5. If clustered, call `cluster_table_by_index(...)`
+
+### B. Bulk load then restore index correctness
+
+1. Load data into heap pages
+2. Call `rebuild_table_indexes(...)`
+3. Call `maintain_clustered_index_layout(...)`
+
+This is the same post-load pattern used by CSV load paths.
+
+### C. OLTP insert/delete with automatic index updates
+
+- Use `insert_tuple_with_index_maintenance(...)` for inserts
+- Use `delete_tuple_with_index_maintenance(...)` for deletes
+
+### D. Drop an index cleanly
+
+1. Remove catalog metadata with `drop_index(...)` or `drop_secondary_index(...)`
+2. Delete `.idx` file on disk using `std::fs::remove_file(...)`
+
+## 10. Practical Caveats for Cross-Team Integration
+
+- Metadata registration and physical index build are intentionally separate steps.
+- `include_columns` is stored in catalog metadata, but there is currently no dedicated covering-index read path.
+- Composite index lookup should always use `index_key_from_values(...)`.
+- If you bypass heap maintenance wrappers during writes, schedule explicit rebuild/validation.
+- For post-integration checks, use `validate_index_consistency(...)` or `validate_all_table_indexes(...)` in tests or startup health checks.

From 69f6afc3956664f89a3288ae0904e1239c82ac77 Mon Sep 17 00:00:00 2001
From: George Rahul <75750164+georgerahul24@users.noreply.github.com>
Date: Sat, 25 Apr 2026 00:02:54 +0530
Subject: [PATCH 7/8] Revise title and sidebar position in indexing API docs

Updated the title and sidebar position for the indexing API endpoints documentation.
---
 .../projects/indexing/indexing_api_endpoints.md            | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/content/storage-engine/projects/indexing/indexing_api_endpoints.md b/content/storage-engine/projects/indexing/indexing_api_endpoints.md
index af05015..3515755 100644
--- a/content/storage-engine/projects/indexing/indexing_api_endpoints.md
+++ b/content/storage-engine/projects/indexing/indexing_api_endpoints.md
@@ -1,6 +1,7 @@
-# RookDB Indexing API Endpoints (Integration Reference)
-
-This document lists the important indexing-related APIs that other teams can call directly from the `storage_manager` crate, based on the current codebase.
+---
+title: API Endpoints
+sidebar_position: 13
+---
 
 ## 1. Module Import Map
 

From 9b2e988a86b834a500130d5b6dc969e8421c5e76 Mon Sep 17 00:00:00 2001
From: George Rahul <75750164+georgerahul24@users.noreply.github.com>
Date: Sat, 25 Apr 2026 00:13:51 +0530
Subject: [PATCH 8/8] Create clustered_index.md

---
 .../projects/indexing/clustered_index.md      | 71 +++++++++++++++++++
 1 file changed, 71 insertions(+)
 create mode 100644 content/storage-engine/projects/indexing/clustered_index.md

diff --git a/content/storage-engine/projects/indexing/clustered_index.md b/content/storage-engine/projects/indexing/clustered_index.md
new file mode 100644
index 0000000..77b65bf
--- /dev/null
+++ b/content/storage-engine/projects/indexing/clustered_index.md
@@ -0,0 +1,71 @@
+---
+title: Clustered Indexing
+sidebar_position: 14
+---
+
+### **1. Supported Algorithms**
+The system implements an `AnyIndex` enum dispatcher that encapsulates two primary families of indexing:
+* **Hash-Based:** `StaticHash`, `ChainedHash`, `ExtendibleHash`, `LinearHash`.
+* **Tree-Based:** `BTree`, `BPlusTree`, `RadixTree`, `SkipList`, `LsmTree`.
+
+### **2. Design Rationale**
+The architecture uses an **Enum Dispatcher** pattern. Since Rust traits cannot easily support static factory methods (like `load`) via trait objects, `AnyIndex` acts as a concrete wrapper. It forwards all `IndexTrait` calls (insert, search, delete) to the underlying specialized implementations.
+
+---
+
+### **3. Complexity Analysis**
+*Based on the implementation logic in `manager.rs` and `executor.rs`:*
+
+| Operation | Complexity (General) | Notes |
+| :--- | :--- | :--- |
+| **Index Creation** | $O(N \cdot M)$ | $N$ is the number of table pages, $M$ is the number of tuples per page. |
+| **Index Update** | $O(\log N)$ or $O(1)$ | Depends on whether the variant is Tree-based (Log) or Hash-based (Amortized constant). |
+| **Space Complexity** | $O(K \cdot R)$ | $K$ is key size, $R$ is the number of records. Stored in specialized `.idx` files. |
+
+---
+
+### **4. Storage and Persistence**
+#### **Metadata Storage**
+* **Format:** JSON.
+* **Location:** `CATALOG_FILE` (typically `catalog.json`).
+* **Structure:** A nested hierarchy: `Catalog` $\rightarrow$ `Database` $\rightarrow$ `Table` $\rightarrow$ `IndexEntry`.
+* **Fields:** Stores `index_name`, `column_names` (supports composite keys), `algorithm` type, and `is_clustered` flags.
+
+#### **Data Storage**
+* **Path:** `database/base/{db}/{table}_{index}.idx`.
+* **Clustered vs Secondary:** * **Clustered:** The physical table data is reordered on disk to match the index key order.
+    * **Secondary:** Stores mappings of `IndexKey` to `RecordId` (Page Number + Item ID).
+* **Serialization:** Uses a `paged_store` or algorithm-specific `save/load` methods to write to disk.
+
+---
+
+### **5. Hashing and Collision Handling**
+* **Hashing Function:** The code supports multiple strategies via `HashIndexType` (Static, Extendible, Linear). 
+* **Composite Keys:** For multi-column indices, components are encoded into bytes, with `INT` values transformed using XOR ($raw \oplus 0x8000\_0000$) to maintain bitwise sortability, then hex-encoded into a `TEXT` key.
+* **Collision Handling:** * **ChainedHash:** Uses bucket chaining.
+    * **Extendible/Linear:** Uses dynamic resizing/splitting based on `LOAD_FACTOR_THRESHOLD`.
+
+---
+
+### **6. Core Operations**
+
+#### **Search**
+* **Point Lookup:** Dispatched via `search`. Supports `search_on_disk` for B+ Trees to allow traversal without loading the entire index into memory.
+* **Range Scan:** Supported only by Tree-based variants. Hash-based variants return `io::ErrorKind::Unsupported`.
+
+#### **Insert / Update**
+* When a tuple is added to a table, `add_tuple_to_all_indexes` triggers. 
+* It extracts the `IndexKey` from the raw tuple bytes (handling `INT`, `TEXT`, and `BOOL` types) and updates every registered index file.
+
+#### **Delete**
+* `remove_tuple_from_all_indexes` locates the key and the specific `RecordId` to prune the index.
+
+#### **Clustering**
+* `cluster_table_by_index` performs an out-of-place sort of the entire table's live tuples based on the index key and overwrites the table file to ensure physical contiguity.
+
+
+
+---
+
+### **7. Data Consistency**
+The system includes a validation utility (`validate_index_consistency`) that performs a **Full Table Scan** to build an "expected" key-map and compares it against the "actual" entries stored in the index file, reporting missing or stale entries.