[RFC] Cloud Bigtable Storage Backend

[annguy3n]

Summary

Defines the design and implementation of a production-grade Cloud Bigtable storage backend for ExtendDB. It specifies the mapping of the DynamoDB data model, query engine, transaction controls, and streams onto Google Cloud Bigtable. The design focuses on ensuring that workloads utilizing the DynamoDB API can run on GCP with high fidelity, guaranteeing transaction isolation, serializability, and scalable Time-to-Live (TTL) eviction.

Motivation

ExtendDB's goal is to allow developers to use the DynamoDB programming model in environments where the managed DynamoDB service is unavailable. Implementing a production-ready Cloud Bigtable connector supports this mission by:

1. Enabling Multi-Cloud Portability
2. Supporting Hybrid Deployments
3. Ensuring API Fidelity

Proposed design

Describe the design in enough detail that someone familiar with ExtendDB could implement it. Include:

• API surface (new operations, changed behavior, wire format)
  • Implements the full mandatory subset of the StorageEngine trait.
  • Enforces strict GSI projection checks: queries requesting non-projected attributes on a GSI return
  • ValidationException (matching DynamoDB behavior) instead of performing slow base-table fetches.
• Storage implications (schema changes, new tables, migration path)
  • Catalog Metadata: System metadata, accounts, and policies are stored in a single table named __extenddb_catalog__.
  • Data Tables: DynamoDB tables map to Bigtable tables named t<table_id_hex> (where table_id_hex is a 32-character hex UUID). Data attributes are stored in column family d.
  • GSIs: Shadow tables are named t<table_id_hex>_g<idx_hash> (where idx_hash is an 8-character hash of the index name) to respect Bigtable's 50-character limit.
  • Transaction Intents: Column family m in data tables stores locks.
  • TTL Index Table: Introduces a dedicated, sharded shadow table __extenddb_ttl_index__ with key encoding [shard_id:1] [expiry_timestamp_be:8] [account_id_len:1] [account_id] [table_name_len:1] [table_name] [encoded_base_row_key]. This allows targeted range scans for expired items rather than scanning entire data tables.
• Configuration (new extenddb.toml sections or flags)
  • Adds the following [storage.bigtable] block to extenddb.toml:

[storage]
backend = "bigtable"

[storage.bigtable]
project_id = "my-project"
instance_id = "my-instance"
data_instance_id = "my-data-instance"      # Optional (defaults to instance_id)
credentials_path = "/path/to/sa-key.json" # Optional (falls back to ADC)
pool_size = 20

DynamoDB compatibility

How does this relate to the real DynamoDB API? Is this:

• Matching existing DynamoDB behavior
• Extending beyond DynamoDB (ExtendDB-specific)
• Deliberately diverging from DynamoDB (explain why)

Alternatives considered

1. Unprotected Single-Row Writes: Bypassing intent locks for non-transactional writes simplifies implementation but permits dirty writes/overwrites on active transactions, breaking serializability. Rejected to preserve API correctness.
2. Scan-Based TTL Sweeper: Scanning the entire base table to evict expired items is simple to implement but does not scale beyond small tables and creates massive read overhead. Rejected in favor of the sharded TTL index table.
3. Cloud Spanner Backend: Cloud Spanner natively supports multi-row transactions. This is a very viable alternative, but it has a different performance, indexing, and pricing profile compared to Bigtable. A Spanner connector should be built as a separate, distinct backend.

Breaking changes

None

Open questions

List anything unresolved that needs discussion before implementation.

1. Tombstone Management on TTL Index: The extenddb_ttl_index table will accumulate empty row tombstones as keys expire. We need to define optimal Bigtable Garbage Collection (GC) policies to ensure compaction cleans them up efficiently.
2. GSI Reconciliation Frequency: How often should the background GSI reconciler scan for mismatches? It should be configurable to allow operators to balance consistency latency against node load.