tagdata

tagdata is an embedded, single-data-file, memory-mapped key/value database for Rust.

See acknowledgments for prior work that informed the project.

Engine

• ACID transactions: serializable and isolated transactions with explicit commit and automatic rollback on drop.
• Concurrent access: multiple snapshot readers and one writer across threads and processes.
• Single-file coordination: Linux, Android, macOS, and iOS keep gates and live-reader registrations in byte-range locks on the database file itself.
• Memory-mapped reads: values are read directly from the mapped database file.
• B+ tree storage: efficient random lookups and ordered sequential access.
• Nested buckets: byte-key/byte-value namespaces can form arbitrary trees.
• Ordered traversal: cursors, key/value iterators, bucket iterators, and ranges.
• Space reuse: freed pages are tracked and reused by later transactions.
• Configurable opening: page size, initial allocation, mmap population, write verification, and direct writes are available through OpenOptions.
• Read-only handles: existing databases can be opened from read-only files and shared safely by multiple reader processes.
• Runtime statistics: file, page, freelist, transaction, and reader state is available through DB::stats().
• Operational snapshots: validated backups and compact copies can be written without stopping concurrent writers.
• Checksummed format: every database authenticates each persisted page and overflow block and exposes full offline verification.

use tagdata::{DB, Error};

fn main() -> Result<(), Error> {
    let db = DB::open("my.db")?;

    let tx = db.tx(true)?;
    let names = tx.create_bucket("names")?;
    names.put("Kanan", "Jarrus")?;
    names.put("Ezra", "Bridger")?;
    tx.commit()?;

    let tx = db.tx(false)?;
    let names = tx.get_bucket("names")?;
    assert!(names
        .get_kv(b"Kanan")
        .is_some_and(|pair| pair.value() == b"Jarrus"));
    Ok(())
}

New code can use read_tx(), write_tx(), and nonblocking try_write_tx() instead of the lower-level tx(bool) method. view and update scope a synchronous closure; update commits only when the closure returns Ok and rolls back on errors or panics. Transactions are synchronous and must not cross an async suspension point.

Writable buckets provide serializable put_if_absent, compare_exchange, and delete_if_value operations. Their owned AtomicResult reports whether the mutation applied plus the observed and resulting values. For compare_exchange, an expected value of None matches a missing key. A missing key is a conflict for delete_if_value.

Recursive merges

DB::merge_from recursively combines every root and nested bucket in one source database with a destination database. The call commits one atomic write transaction. Existing destination entries that are absent from the source are retained, bucket sequence counters preserve the greater value, and the returned MergeReport counts inserted, updated, unchanged, skipped, created, and merged entries.

use tagdata::{DB, MergeConflictPolicy, MergeOptions};

# fn merge() -> Result<(), tagdata::Error> {
let source = DB::open("source.db")?;
let destination = DB::open("destination.db")?;
let report = destination.merge_from(
    &source,
    MergeOptions::new().conflict_policy(MergeConflictPolicy::Overwrite),
)?;
println!("inserted {} keys", report.keys_inserted);
# Ok(())
# }

Overwrite replaces conflicting values and key/value-versus-bucket shapes, KeepExisting retains destination entries, and Error rejects the first conflict with its binary bucket path while rolling back the database-level merge. Bucket::merge_from provides the same recursive behavior inside an existing caller-managed write transaction. TTL metadata follows the winning value, while durable journal history is not imported from the source.

Typed codecs

The raw byte API remains the default and adds no serialization dependency. Enable typed for KeyCodec, ValueCodec, and TypedBucket<K, V, C>:

tagdata = { version = "0.1", features = ["typed"] }

The built-in unsigned, sign-bit-adjusted signed, UTF-8 string, byte-vector, and compound-string key codecs preserve lexicographic ordering. Typed ranges reject codecs that do not declare ordering preservation. Decoding returns owned values; only the raw API claims mmap-backed zero-copy reads. serde-codec additionally enables the opt-in MessagePack value codec.

The default build contains the raw database, integrity verification, TTL consistency, and write-safety policies. Optional features keep unused APIs out of constrained applications:

• bytes-interop: accept bytes::Bytes without copying;
• changefeed: process-local watches and the durable journal;
• maintenance: backup and compaction APIs;
• operator: the operator CLI and salvage APIs (includes maintenance);
• typed: typed collections;
• serde-codec: MessagePack typed values (includes typed).

Codec selection is part of an application's schema. Store a schema/version key in the containing raw bucket (or use versioned bucket names), migrate values in a write transaction, and never change a live bucket's codec without rewriting all entries. Raw and typed views may coexist when they follow the same schema.

Change watches and TTL

With changefeed, DB::watch(capacity) receives ordered, process-local ChangeSet values after a transaction is durably committed. Sets preserve transaction boundaries and IDs and contain bucket paths, keys, and operation types—not values. Delivery is best-effort with no durable replay or cross-process transport. Commit never waits for a watcher; a subscriber whose bounded queue fills is disconnected. Large transactions cap tracking at 4,096 changes or 4 MiB and set truncated.

Bucket::put_with_ttl persists a Unix-millisecond expiration in a reserved nested index. get_live applies the current wall clock, while get_live_at accepts an explicit time. Cleanup is deliberately lazy and bounded through the deadline-ordered purge_expired; DB::purge_expired applies one global limit while walking nested buckets. No runtime or background thread is required. Raw get ignores TTL and expired bytes remain visible to raw access until cleanup. Wall-clock jumps affect expiry, and ordinary put does not clear an existing TTL—call clear_ttl when making a key persistent. Backup and compaction preserve TTL indexes. See docs/decisions/0002-changes-ttl-watches.md for delivery and time semantics.

Durable change journal

With changefeed, watches remain process-local and best-effort. Applications that need replay can opt into a journal stored atomically inside the user transaction:

use tagdata::{DB, JournalConfig};

# fn example() -> Result<(), tagdata::Error> {
let db = DB::open("my.db")?;
db.enable_journal(JournalConfig { max_transactions: 10_000 })?;
let replay = db.replay_journal(0, 100, None)?;
for transaction in replay.transactions {
    println!("transaction {}", transaction.transaction_id);
}
# Ok(())
# }

Replay preserves transaction boundaries and supports the same filters as watches. Retention is transaction-count based. Consumer checkpoints are durable but do not pin history; JournalReplay::gap reports when retention passed a requested transaction. Journal records contain keys and operation metadata, never values. Tracking remains bounded to 4,096 changes or 4 MiB per transaction, and oversized records carry truncated = true. Cross-process consumers poll replay by transaction ID, which also provides the base contract for secondary indexes, incremental backup, and replication adapters.

Storage layout

The format uses fixed-size pages:

meta pages | freelist | leaf and branch pages

Write transactions update a copy-on-write B+ tree and publish a new meta page when committed. Large nodes can span multiple pages, and the freelist makes released pages available to future writes.

Commits use two durability barriers: changed data and freelist pages are synced before the alternate meta page is published, then the meta page is synced before the commit returns. After an interrupted commit, reopening selects either the complete previous snapshot or the complete newly published snapshot.

The default WriteVerification::ReadBack policy rereads every dirty block from a separate buffered file descriptor after the data barrier and compares it with the bytes submitted to the kernel. Only a successful comparison permits the new metadata snapshot to be published; that metadata is also reread after its barrier. WriteVerification::Standard keeps checksums, copy-on-write, and both barriers without readback. WriteVerification::Full adds a complete structural walk before metadata publication. For example:

use tagdata::{OpenOptions, WriteVerification};

let db = OpenOptions::new()
    .write_verification(WriteVerification::Full)
    .open("important.db")?;
# Ok::<(), tagdata::Error>(())

Readback verifies the bytes visible through the operating system after sync; it does not replace drive power-loss protection or end-to-end storage hardware.

Use OpenOptions::new().read_only() for a handle that never creates, resizes, or writes the database. A read-only handle rejects writable transactions.

Writable handles use kernel-owned locks for writer gates and live-reader registrations. Registrations disappear automatically when their transaction is dropped or their process terminates.

With maintenance, use DB::backup_to for an atomically published snapshot, DB::backup_writer to stream a snapshot, and DB::compact_to to rewrite only live data into a smaller file. Maintenance operations never replace the source database.

Tagdata has one current on-disk format. Each allocated page block ends with a SHA3-256 checksum covering its header and payload. Page bounds, element counts, offsets, overflow spans, tree ordering, and reachability are checked by DB::verify(). OpenOptions::verify_on_open(true) performs that full walk while opening. Normal commits checksum only dirty blocks; reads retain the mmap-backed zero-copy path, so full verification remains an explicit policy choice.

The format persists each freed page's retirement transaction and only reclaims pages older than the oldest registered reader. There is no format-selection API or legacy-format compatibility path. Files whose format marker differs from the current FORMAT_VERSION are rejected.

Format version 4 uses open-file-description byte-range locks on the database file. Gate, reader-token, and transaction-payload locks live in a reserved range far beyond the data and never extend or modify the file. Kernel lock ownership removes registrations immediately when a process exits or crashes. Normal read and write gates are held per transaction; a handle opened with read_only() keeps its shared whole-file lock until the handle is dropped.

Offline verification uses a read-only handle:

cargo run --example verify -- data.db
# Supply the original page size when it differs from the host default:
cargo run --example verify -- data.db 8192

make benchmark reports full-verification time, long-reader/write-churn growth, deadline-index TTL cleanup latency, and compares the selected SHA3-256 checksum with FNV-1a-64 over the same file. SHA3-256 is used on disk for substantially stronger corruption detection; the benchmark keeps that cost visible rather than silently choosing the faster non-cryptographic hash.

make benchmark-compare compares Tagdata with jammdb 0.11.0 using hot point reads, one-lookup transactions, overlapping snapshots, ordered scans, reopen reads, three value sizes, and the same batched writes. It alternates execution order and reports medians across independent database files. The workload is controlled by TAGDATA_BENCH_ITEMS, TAGDATA_BENCH_READS, TAGDATA_BENCH_SHORT_READS, TAGDATA_BENCH_REOPEN_READS, TAGDATA_BENCH_SAMPLES, and comma-separated TAGDATA_BENCH_VALUE_BYTES.

make benchmark-write-verification reports median commit latency for Standard, ReadBack, and Full policies. TAGDATA_BENCH_ITEMS controls database size and TAGDATA_BENCH_SAMPLES controls repetitions.

make operator-size builds the operator with size optimization, whole-program LTO, one codegen unit, abort-on-panic, and stripped symbols. Applications that embed Tagdata must define equivalent profile settings in their own workspace; Cargo ignores release profiles declared by dependencies.

make benchmark-memory reports peak heap growth and allocation calls while reopening and updating a 10,000-record database, making bootstrap scanning and optional change-tracking overhead visible.

make benchmark-merge creates two 500,000-entry databases with randomized keys and values, applies a configurable overlap, merges both through DB::merge_from, and verifies the final key count, content fingerprint, and tree structure. TAGDATA_MERGE_OVERLAP_PERCENT defaults to 12.5. Results are machine-specific and meaningful only when compared on the same host. Hot reads isolate lookup traversal, short transactions include snapshot coordination, and reopen reads include mapping and format inspection.

make benchmark-typed compares direct cursor decoding with the former typed scan path that repeated the main-tree lookup for every record.

Commands

make build
make run
make test
make benchmark
make benchmark-compare
make benchmark-memory
make benchmark-merge
make benchmark-typed
make operator-size
make verify