DedupBench is a benchmarking tool for content-defined chunking techniques used in data deduplication. It currently supports eleven unique CDC techniques and five different vector instruction sets.
-
Updated
Jun 22, 2026 - C++
#
content-defined-chunking
Here are 19 public repositories matching this topic...
Fast and efficient content-defined chunking for data deduplication. Java implementation of FastCDC as library.
-
Updated
Sep 21, 2023 - Java
Fast content-defined chunking in Go.
-
Updated
Dec 12, 2023 - Go
Steadlith reuses unchanged RAG chunks with content-defined identities, cache-aware planning, and transactional indexing.
python cli open-source sqlite embeddings python-cli merkle-tree rabin-fingerprint content-addressed-storage rag vector-database content-defined-chunking retrieval-augmented-generation document-chunking incremental-indexing rag-indexing
-
Updated
Aug 16, 2026 - Python
CDCZip is a deduplication filter meant for use as long range match finder preprocessor for use alongside other compression algorithms.
-
Updated
Sep 19, 2025 - C++
Find (partial content) duplicate files.
-
Updated
Dec 10, 2022 - Python
A Rust library for Content-Defined Chunking (CDC).
-
Updated
Jan 6, 2026 - Rust
Print FastCDC rolling hash chunks and checksums.
-
Updated
Nov 27, 2022 - Python
Synclet is a content-defined, chunk-based file synchronization and backup tool that detects and syncs only the actual changes—efficiently, intelligently, and reliably.
-
Updated
Jan 2, 2026 - C++
Ship a search index to a device as signed, content-addressed bundles: fail-closed Ed25519 verification, FastCDC chunking so the next release downloads only the chunks that changed, then FAISS + BM25 rank fusion locally. Python. 309 tests, 98% coverage.
python offline-first ed25519 content-addressed edge-computing faiss delta-sync vector-search on-device-ai local-first supply-chain-security fastcdc content-defined-chunking hybrid-search rank-fusion deterministic-routing signed-bundles
-
Updated
Aug 17, 2026 - Python
Content-defined chunk boundaries so a one paragraph edit re-embeds two chunks, not the document
-
Updated
Aug 15, 2026 - TypeScript
Clojure implementation of Rabin-Karp rolling hash and chunker
-
Updated
Jul 3, 2025 - Clojure
Open-source toolkit for lossless, reference-based reconstruction of large binary artifacts using deterministic CDC and compact SBD1 seeds.
-
Updated
Jul 27, 2026 - Python
ANLA — Agent-Native Lossless Archive. An AI may plan how to pack; a deterministic, model-independent decoder must restore every byte exactly. Two frozen MVP implementations in Python and JavaScript, plus an ANLA 1.0 draft with independent Python and Rust implementations that produce byte-identical archives.
python rust file-format cbor lossless deduplication digital-preservation zstandard differential-testing archive-format blake3 content-defined-chunking
-
Updated
Aug 15, 2026 - Python
Eliminating storage redundancy at the sub-file level. A zero-copy deduplication engine in Rust, powered by Content-Defined Chunking (FastCDC) and BLAKE3, with instant browser WASM execution.
react rust storage-engine zero-copy webassembly wasm indexeddb client-side deduplication systems-engineering blake3 fastcdc content-defined-chunking data-redundancy
-
Updated
Aug 5, 2026 - JavaScript
A FUSE-based filesystem implementing content-defined chunking and deduplication in Rust.
rust fuse libfuse tags version-control filesystem zstd fuse-filesystem deduplication systems-programming snapshots sled copy-on-write fastcdc content-defined-chunking shadow-paging acid-compliance
-
Updated
Jun 6, 2026 - TeX
Coldkeep is a correctness-first, content-addressed cold-storage engine with deterministic restore, staged integrity verification, safe garbage collection, deduplication, transform-aware packed storage, stable storage semantics, and a reusable engine/catalog core. v1.x is complete; v2.x focuses on local-first productization.
go backup restore storage-engine verification research-project deterministic correctness deduplication data-integrity cold-storage content-addressed-storage content-defined-chunking
-
Updated
Aug 18, 2026 - Go
Open-source, self-hostable file sync for people and AI agents, with a Rust daemon and CLI, native macOS Finder integration, scoped sharing, and S3-compatible storage.
macos linux rust cli typescript file-sharing self-hosted file-sync agplv3 ai-agents delta-sync dropbox-alternative s3-compatible content-defined-chunking file-provider nextcloud-alternative
-
Updated
Jul 20, 2026 - Rust
The CAVS Hub command-line client — wire Git LFS to content-defined, deduplicated transfers so only changed bytes travel on push/pull.
git git-lfs rust cli command-line version-control developer-tools game-assets deduplication cavs content-defined-chunking self-update
-
Updated
Aug 10, 2026 - Rust
Improve this page
Add a description, image, and links to the content-defined-chunking topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the content-defined-chunking topic, visit your repo's landing page and select "manage topics."