Skip to content

docs: add RFC draft for Cloud Bigtable storage backend - #186

Draft
annguy3n wants to merge 1 commit into
ExtendDB:mainfrom
annguy3n:rfc/bigtable-backend
Draft

docs: add RFC draft for Cloud Bigtable storage backend#186
annguy3n wants to merge 1 commit into
ExtendDB:mainfrom
annguy3n:rfc/bigtable-backend

Conversation

@annguy3n

Copy link
Copy Markdown

What

This PR introduces the draft design for the Cloud Bigtable storage backend (RFC-0003).

Key design highlights include:

  • Lock-then-Read 2PC Flow
  • Single-Row Write Guarding
  • Transactional Streams
  • Sharded TTL Index shadow table
  • GSI Validation and Reconciler
  • Secure Credentials Path

Why

Cloud Bigtable is GCP's managed wide-column store and serves as the natural target for DynamoDB workloads migrating to GCP. A robust connector allows AWS-to-GCP migration for high-throughput, low-latency applications with zero code changes.

Checklist

ADR / RFC: #185


By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.

@annguy3n
annguy3n force-pushed the rfc/bigtable-backend branch from 4c0a377 to 88190c4 Compare June 29, 2026 20:05
Proposes the design of a production-grade Cloud Bigtable storage backend for ExtendDB. Key items include a Lock-then-Read 2PC transaction flow, sharded TTL indexing shadow table, GSI projection enforcement, and secure GCP authentication config.
@annguy3n
annguy3n force-pushed the rfc/bigtable-backend branch from 88190c4 to e0d1d91 Compare June 29, 2026 20:07
@LeeroyHannigan

Copy link
Copy Markdown
Collaborator

Thank you for this RFC. At a quick glance, this looks solid. Let me do a thorough review and circle back to you early next week (week commencing 6 July).

@LeeroyHannigan LeeroyHannigan added the RFC Request for Comments, a proposal open for discussion before implementation label Jul 3, 2026
@LeeroyHannigan

Copy link
Copy Markdown
Collaborator

Thanks for the detailed RFC, @annguy3n this is a strong design and we're happy to accept a Cloud Bigtable backend on these lines. You've clearly thought hard about the DynamoDB semantics: the lock-then-read 2PC, single-row write guarding, stream replay via the txn log, and the sharded TTL index are exactly the right calls given Bigtable's single-row atomicity.

A few DynamoDB-parity points to fold in, split into what must match vs. what's fine to differ if documented:

Must match DynamoDB

  • TTL + Streams: the TTL sweeper's deletes must emit DynamoDB-compatible REMOVE stream records (with the TTL principal) when streams are enabled, please make that explicit in the design, since silently dropping them is a breaking behavior gap.

  • Key encoding: DynamoDB sorts Number keys numerically (exact to 38 significant digits) and String/Binary keys by byte order. In a byte-lexicographic store that means numeric keys need an order-preserving byte encoding, and numbers must round-trip as exact decimals (not floats). This currently only shows up under Testing but it's a core design detail and should be promoted into the row-key format and GSI index-key sections.

  • Transaction rules: TransactWriteItems needs ClientRequestToken idempotency (10-minute window) and must reject two operations on the same item, worth noting alongside the 2PC flow.

  • Read isolation: DynamoDB gives read-committed for GetItem/Query/Scan/BatchGetItem and serializable for TransactGetItems. Two things fall out of the 2PC apply window: (1) a row's new value must not be visible until the transaction's commit point, applying mutations before the commit decision is durable would expose a write that can still roll back, which is a dirty read and breaks read-committed; and (2) TransactGetItems must observe the committed set atomically (read through the coordinator / at the commit boundary), otherwise it can see partial state and violate serializable. Please spell out how both are guaranteed.

Fine to differ, just call it out

  • GSI consistency: shadow tables make GSIs eventually consistent, which matches DynamoDB, good. Please still reject ConsistentRead=true on a GSI (API parity), and make sure the reconciler keeps GSI reads from returning orphaned/phantom rows during the divergence window.

  • Cross-row visibility for standard reads: during the apply window a Query/Scan may see some rows of a committed transaction updated and others not. That's within read-committed (each row still returns a committed value), so it's fine, just document it, and confirm the read-committed and TransactGetItems guarantees above hold.

  • Backup / PITR / export: not covered here, fine to defer to a follow-up RFC, but let's note that the effective PITR/export window will differ from DynamoDB's 35 days.

Overall: accepted in principle, let's tighten the TTL-stream path, key-encoding, and the read-isolation guarantees in the doc, then move to implementation behind the opt-in feature. Really nice write-up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

accepted RFC Request for Comments, a proposal open for discussion before implementation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants