From e0d1d91efc307a35723f7ba819aa9f904ca261e3 Mon Sep 17 00:00:00 2001 From: An Nguyen Date: Mon, 29 Jun 2026 19:49:45 +0000 Subject: [PATCH] docs: add RFC draft for Cloud Bigtable storage backend Proposes the design of a production-grade Cloud Bigtable storage backend for ExtendDB. Key items include a Lock-then-Read 2PC transaction flow, sharded TTL indexing shadow table, GSI projection enforcement, and secure GCP authentication config. --- docs/rfcs/0000-bigtable-backend.md | 168 +++++++++++++++++++++++++++++ 1 file changed, 168 insertions(+) create mode 100644 docs/rfcs/0000-bigtable-backend.md diff --git a/docs/rfcs/0000-bigtable-backend.md b/docs/rfcs/0000-bigtable-backend.md new file mode 100644 index 00000000..a7f4c172 --- /dev/null +++ b/docs/rfcs/0000-bigtable-backend.md @@ -0,0 +1,168 @@ +# RFC-0003: Cloud Bigtable Storage Backend for ExtendDB + +- Status: Draft +- Author: @annguy3n +- Created: 2026-06-29 +- Tracking issue: #185 + +## Summary + +This RFC defines the design and implementation of a production-grade Cloud Bigtable storage backend for ExtendDB. It addresses critical architectural requirements for this new backend: transaction isolation, stream emissions in transactions, Global Secondary Index (GSI) projections, GSI write consistency, Time-to-Live (TTL) eviction scalability, and secure Google Cloud Platform (GCP) connectivity. + +## Motivation + +A naive mapping of DynamoDB structures to Bigtable introduces several architectural challenges that must be addressed to ensure production-readiness: +1. **Transaction Isolation Leak:** Non-transactional (single-row) writes bypass 2-Phase Commit (2PC) locks (intents), enabling dirty writes and lost updates. +2. **2PC Race Condition (Read-then-Lock):** The 2PC coordinator reads row data before placing locks, exposing the system to Time-of-Check to Time-of-Use (TOCTOU) races. +3. **Missing Transaction Streams:** Multi-row transactions (`TransactWriteItems`) do not emit DynamoDB Stream records. +4. **Inefficient TTL Eviction:** The TTL worker performs full-table scans to identify expired items, which degrades performance on large datasets. +5. **GSI Consistency Gap:** GSI shadow table write failures are logged without retry or repair, causing permanent base-to-shadow divergence. +6. **Auth Configuration Constraints:** Lack of support for explicit credentials file configuration. + +This RFC provides the technical specification to resolve these challenges. + +## Detailed Design + +### 1. Table Mapping & Naming Constraints + +* **Catalog:** System metadata, accounts, and policies are stored in a single table named `__extenddb_catalog__`. +* **Data Tables:** DynamoDB tables map to Bigtable tables named `t` (where `table_id_hex` is a 32-character hex UUID). Data attributes are stored in column family `d`. +* **GSIs:** Shadow tables are named `t_g` (where `idx_hash` is an 8-character hash of the index name). +* **Transaction Intents:** Column family `m` in data tables stores lock/intent cells. +* **TTL Markers:** Column family `t` (reserved). + +> [!NOTE] +> **Naming Constraints:** Table IDs in Cloud Bigtable are limited to 50 characters. To prevent overflow, GSIs are named `t_g` (43 characters total) rather than reproducing the full index name. + +### 2. Transaction Isolation & 2PC Refactoring + +#### A. Lock-then-Read 2PC Flow +Refactor the `TransactWriteItems` coordinator to acquire locks *before* reading data and evaluating condition expressions. + +```mermaid +sequenceDiagram + participant C as Coordinator (ExtendDB) + participant B as Bigtable + + C->>B: Phase 1: Open Transaction (Write PENDING to __extenddb_txn_log__) + Note over C,B: Place Intents (Locks) + loop For each participant row + C->>B: Place Intent (CheckAndMutateRow: fail if other fresh intent exists) + end + alt Any Lock Fails + C->>B: Rollback (Clear placed intents, drop txn row) + Note over C: Return TransactionConflict + end + Note over C,B: Read & Verify + C->>B: Read participant rows (now locked by us) + Note over C: Evaluate Condition Expressions + alt Any Condition Fails + C->>B: Rollback (Clear intents, drop txn row) + Note over C: Return TransactionCanceled + end + C->>B: Phase 2: Commit Transaction (Write COMMITTED to __extenddb_txn_log__) + Note over C,B: Apply Mutations + loop For each participant row + C->>B: Apply Data Mutations & Clear Intent (Atomic MutateRow) + end + C->>B: Phase 3: Clean up (Write CLEANED to log, then drop txn row) +``` + +#### B. Guarding Single-Row Writes +All single-row mutations (`PutItem`, `UpdateItem`, `DeleteItem`) must run via `CheckAndMutateRow` to prevent overwriting active 2PC locks: +* **Predicate:** Matches if a cell exists in family `m` with qualifier `intent:*` and a timestamp $\ge$ `now - intent_timeout`. +* **False Mutations (No Lock):** Execute the mutation. +* **True Mutations (Locked):** None. Returns `TransactionConflict` (client retries). +* **Deletes:** Use `DeleteFromFamily(d)` instead of `DeleteFromRow` to preserve the intent family `m`. + +#### C. Recovery Sweeper +A background worker periodically scans `__extenddb_txn_log__` for transactions older than `intent_timeout`: +* **`PENDING` State:** Clear intents on participant rows and delete the coordinator row (rollback). +* **`COMMITTED` State:** Re-apply mutations to participant rows, write stream records, clear intents, and delete the coordinator row (roll-forward). + +### 3. Stream Emissions in Transactions + +To ensure stream record atomicity: +1. **Phase 1 Resolution:** Fetch the full `TableDescription` (including `StreamSpecification` and `latest_stream_arn`) and pre-generate the `StreamRecord` payloads. +2. **Log Enrichment:** Store the pre-generated stream record payloads in the coordinator row of `__extenddb_txn_log__` in Phase 2. +3. **Phase 5 Commit:** Write the pre-generated stream records to the stream table alongside the data mutations. +4. **Recovery:** The recovery sweeper uses the payloads stored in the log to roll forward stream writes on crash. + +### 4. Scalable TTL Indexing + +Avoid full-table scans by introducing a sharded TTL index table. + +#### A. TTL Index Table Schema +* **Table Name:** `__extenddb_ttl_index__` +* **Row Key format (Binary):** `[shard_id:1] [expiry_timestamp_be:8] [account_id_len:1] [account_id] [table_name_len:1] [table_name] [encoded_base_row_key]` + * `shard_id` (1 byte): `hash(base_row_key) % 16` (prevents write hotspotting). + * `expiry_timestamp_be` (8 bytes): Big-Endian `u64` representing the TTL epoch second. + * `encoded_base_row_key` (variable): Raw row key of the target item. +* **Payload:** None (empty values). + +#### B. Maintenance +* **Writes:** Insert an empty row in `__extenddb_ttl_index__` when writing an item with a TTL attribute. +* **Updates/Deletes:** Delete the old index entry using the item's prior image. + +#### C. Sweeper Flow +1. Perform parallel range scans across all 16 shards from `start_key = [S] [0]` to `end_key = [S] [current_time_epoch_s + 1]`. +2. For each expired entry, read the base row and verify its current TTL matches the index entry. +3. If valid, execute `delete_item` on the base table. +4. If invalid (stale index), delete the index entry. + +### 5. GSI Projections & Consistency + +#### A. Projection Validation +Enforce DynamoDB GSI projection behavior: +* **Read Path:** Queries on GSIs with `KEYS_ONLY` or `INCLUDE` projections return only the projected attributes. +* **Validation:** If the client requests `Select: ALL_ATTRIBUTES` (or requests non-projected attributes) on a non-`ALL` GSI, return `ValidationException`. Do not perform base table fetches. +* **Defaulting:** If `Select` is omitted in a GSI query, default to `Select::AllProjectedAttributes`. + +#### B. Background Reconciler +Implement a reconciler as an internal background worker spawned via `RuntimeHooks`. The worker scans GSIs, compares them with the base tables, and repairs missing, orphaned, or mismatched shadow rows. + +### 6. GCP Credentials Configuration + +Extend `BigtableStorageConfig` to support explicit service account files: + +```toml +[storage.bigtable] +project_id = "my-project" +instance_id = "my-instance" +data_instance_id = "my-data-instance" # Optional +credentials_path = "/path/to/sa-key.json" # Optional +pool_size = 20 +``` + +* **Behavior:** If `credentials_path` is specified, the server programmatically sets the `GOOGLE_APPLICATION_CREDENTIALS` environment variable before client initialization. Configure `tonic` channels with `ClientTlsConfig` using native system roots. + +### 7. Implementation Sketch + +* **`config.rs`**: Add `credentials_path` and `data_instance_id` to config. +* **`client.rs` & `admin.rs`**: Configure TLS and inject bearer tokens. +* **`item_ops.rs`**: Update mutations to use `CheckAndMutateRow` and target deletes to the `d` family. +* **`transact.rs`**: Refactor `TxnCoordinator` to Lock-then-Read and implement the recovery sweeper. +* **`engine.rs`**: Update `transact_write_items`, GSI retry tasks, and stream record writes. +* **`ttl_worker.rs`**: Implement sharded TTL sweeps. +* **`gsi_reconciler.rs`**: Implement background GSI reconciliation. + +## Testing Strategy + +1. **Unit Tests (Emulator-based):** + * Verify type encoding/decoding and lexicographical sorting of numeric keys. + * Mock Bigtable client to verify 2PC coordinator state transitions. + * Assert `ValidationException` is returned for invalid GSI attribute requests. +2. **Failure Integration Tests (Real Bigtable & Emulator):** + * Inject crashes during the 2PC flow and assert that the recovery sweeper rolls forward/back and preserves stream record writes. + * Run concurrent mixed workloads to verify transactional serializability. + * Verify GSI reconciler heals base-to-shadow mismatches. + * Verify TTL sweeper ignores updated items with old index entries. +3. **E2E & Chaos Tests:** + * Run the Python integration suite (`devtools/run-tests --extenddb --pytest`) against production Bigtable. + * Inject network partitions during `TransactWriteItems` to verify 2PC resilience. + +## Alternatives Considered + +* **Unprotected Single-Row Writes:** Bypassing intent checks for single-row writes allows dirty writes, violating serializability. Rejected. +* **Scan-Based TTL Sweeper:** Scanning the entire base table to evict items degrades performance at scale. Rejected. +* **Cloud Spanner Backend:** Cloud Spanner supports multi-row transactions natively, eliminating the need for application-layer 2PC. This is a viable alternative to be explored as a separate backend connector.