rsync-archive

Stream-create random-access archives with rsync-grade path selection

Select like rsync · Compress like 7-Zip · Access like a seekable stream

rsync-archive create -o game.7z --method zstd --exclude '*.tmp' --level 5 /data/game/
rsync-archive create -o logs.tar.zst --max-total-size 500M --newer-than 7d /var/log/
rsync-archive embed  -o master.7z nest1.7z nest2.7z

Install · Quick start · Features · CLI reference · Formats · Selection · Docs

Why rsync-archive?

Packing large trees for backup, distribution, or incident response usually forces a trade-off:

Traditional approach	Pain
Solid 7z / tar\|zstd	Great ratio, no per-file random access
Full 7-Zip GUI / `7zz a`	Strong archives, weak rsync-style selection
`rsync` + archive later	Great filters, two tools, two passes, temp disk
Solid streams on live trees	Mid-run vanish → whole job aborts

rsync-archive unifies that workflow:

Rsync-style include/exclude (and budgets) decide what goes in
Streaming compress writes without copying the tree to temp
Non-solid / seekable layouts keep file-level random access
Embed nests finished archives into a master store 7z (Copy, no recompress)

Built in Rust, tested across Ubuntu and Rocky Linux, released as distro-matched x86_64 binaries.

At a glance


Version	v0.6.1
Commands	`create` · `embed`
Default format	Non-solid 7z (per-file packs)
Also	Seekable-zstd · tar.zst · tar.lz4
Codecs (7z)	LZMA2 · Zstd · LZ4
Selection	rsync filters + dir/global size & count budgets
Safety	Unique `OUT.partial.<pid>` → fsync → atomic rename · soft-skip live-tree races · verify-before-commit
License	MIT

Features

Create — select, stream, archive

Capability	Details
Rsync-grade filters	`--include` / `--exclude` / `--filter` / `--filter-from` / `-from` files; first-match-wins; basename rules (`.tmp`); `` / `*` / `?`; anchored `/pat`; dir-only `pat/`
Master lists	`SRC...` walks, `--files-from` explicit lists, optional `--include-cwd`
Directory budgets	`--dir-max-size PATH=SIZE`, `--dir-max-files PATH=N` — recursive, newest-mtime-first, longest prefix wins
Global caps	`--max-total-size`, `--max-files`, `--max-size`, `--min-size`, `--newer-than`
Restriction lists	`--file-size-from`, `--dir-max-size-from`, `--dir-max-files-from` (rsync-like line syntax)
Dry-run	`-n` lists selected archive paths without writing
Parallel encode (7z)	Worker pool, ordered write, `--threads`, `--encode-concurrency`, `--encode-size-budget` (default 500M)
Live-tree resilience	Soft-skip vanished/unreadable members; 7z pack rollback; tar/seekable zero-pad + warn; `--strict` exit 1 on any degraded member
Verify	`--verify` checks the finished archive before commit — a failed verify keeps the previous `-o`
Safe output	Refuse overwrite without `--force`; unique `.partial.<pid>.<n>` temp + fsync + atomic rename; concurrent runs at the same `-o` never interleave (last commit wins)
Self-exclusion	The `-o` output and its partial temps are never archived — even when they sit inside a SRC tree or CWD

Embed — master store without recompression

Capability	Details
Store / Copy 7z	Method `0x00` — wrap finished files as members
Naming	Basename by default; `--keep-path` + optional `--prefix`
Magic policy	Warn on non-7z · `--require-7z` hard-fail · `--allow-any` silent
Dry-run / verify / force	Same safety model as create

Formats — choose access pattern

Format	Random access	Metadata	Symlinks / hard links	Infer from `-o`
7z (default)	Per-file packs	mtime, empty flags	File bodies only	`.7z` / default
seekable-zstd	Byte-range + name index	Member names	File bodies only	`.zst`
tar-zstd	Seekable Zstd + `RATAIDX1`	mode/uid/gid, uname/gname, dirs	Full	`.tar.zst` / `.tzst`
tar-lz4	Multi-frame LZ4 + `RATLFRM1` + `RATAIDX1`	same as tar-zstd	Full	`.tar.lz4` / `.tlz4`

Create compresses. Embed only stores.
Embed does not convert solid → non-solid (use archiveconverter for that).

Design principles

Non-solid only for create — every member independently accessible
Disk-friendly — stream read / stream compress / append packs; no full-tree temp copy
Selection first — --dry-run before you spend CPU on compression
Fail soft on live trees — one rotated log should not kill a 100k-file pack
Docs + tests required on every behavior change (AGENTS.md)

Installation

Prebuilt binaries (recommended)

Release assets for Ubuntu 22.04 / 24.04 and Rocky Linux 8 / 9 / 10 (x86_64, glibc-linked):

# Example: Ubuntu 24.04
curl -fsSL -o rsync-archive.tar.gz \
  https://github.com/hilather/rsync-archive/releases/download/v0.6.1/rsync-archive-ubuntu24.04-x86_64.tar.gz
tar -xzf rsync-archive.tar.gz
./rsync-archive --version
# optional: install system-wide
sudo install -m755 rsync-archive /usr/local/bin/

Each tarball contains a binary named rsync-archive (no OS suffix inside the archive).
SHA-256 sidecars are published next to every asset.

Rocky Linux RPMs

Release tags v* attach Rocky Linux 8 / 9 / 10 (x86_64) .rpm packages built on that distro (that glibc). Pick the major that matches the host — the filename carries rocky8 / rocky9 / rocky10; the NVR inside is rsync-archive-<version>-1.x86_64 on all three. The package installs /usr/bin/rsync-archive.

# Example: Rocky Linux 9
curl -fsSL -O https://github.com/hilather/rsync-archive/releases/download/v0.6.1/rsync-archive-0.6.1-1.rocky9.x86_64.rpm
sudo dnf install ./rsync-archive-*.rocky9.x86_64.rpm
# or: sudo rpm -i ./rsync-archive-0.6.1-1.rocky9.x86_64.rpm
rsync-archive --version

On a Rocky/RHEL host with gcc + rustc:

PACKAGE_FAMILY=rpm DISTRO_LABEL=rocky9 SKIP_TARBALL=1 ./packaging/build-native-packages.sh
sudo dnf install ./dist/rsync-archive-*.rpm

Build from source

Requirements: Rust 1.85+ (locked deps include edition-2024 crates)

git clone https://github.com/hilather/rsync-archive.git
cd rsync-archive
cargo build --release
# binary: target/release/rsync-archive

Optional native codecs

# Denser LZMA2 (liblzma) + LZ4HC for --method lz4 at levels ≥3
cargo build --release --features native-codecs

Feature	Effect
`liblzma`	C-backed LZMA2 via system or bundled xz
`liblzma-static`	Force static/bundled liblzma
`lz4-hc`	True LZ4HC for levels ≥3 (`lz4` crate / vendored sources)
`native-codecs`	Convenience: `liblzma` + `lz4-hc`

Default build uses pure-Rust lzma-rust2 + lz4_flex + libzstd bindings — no system liblzma required.

Quick start

# Help
rsync-archive --help
rsync-archive create --help
rsync-archive embed --help

# Non-solid 7z (LZMA2 default)
rsync-archive create -o out.7z --exclude '*.tmp' --level 5 --verify ./src/

# Fast path — Zstd packs inside 7z
rsync-archive create -o out.7z --method zstd --level 5 ./data/

# Dry-run selection
rsync-archive create -o out.7z -n --exclude '*.tmp' ./src/

# Explicit file list
rsync-archive create -o out.7z --files-from list.txt

# Seekable Zstd stream (byte-range access)
rsync-archive create -o out.zst --format seekable-zstd --level 5 ./src/
rsync-archive create -o pack.zst --level 3 ./data/          # .zst → seekable-zstd

# RA-friendly tar containers
rsync-archive create -o out.tar.zst --format tar-zstd ./src/
rsync-archive create -o out.tar.lz4 --format tar-lz4 ./src/

# Log collection with budgets
rsync-archive create -o logs.7z \
  --max-total-size 500M \
  --max-files 1000 \
  --max-size 50M \
  --newer-than 7d \
  /var/log/

# Embed finished archives under a master store 7z
rsync-archive embed -o master.7z nest1.7z nest2.7z
rsync-archive embed -o master.7z --force --verify nest1.7z nest2.7z
rsync-archive embed -o master.7z --allow-any a.bin b.bin

Realistic create example

rsync-archive create -o game.7z \
  --exclude '*.tmp' \
  --exclude-from excludes.txt \
  --filter '- cache/**' \
  --method zstd --level 5 --verify \
  /data/game/

Directory & list restrictions

rsync-archive create -o o.7z --dir-max-size logs/=100M --dir-max-size cache=50M tree/
rsync-archive create -o o.7z --dir-max-files logs/=10 --dir-max-files-from limits.txt tree/
rsync-archive create -o o.7z \
  --files-from master.txt \
  --file-size-from sizes.txt \
  --dir-max-size-from dirs.txt

Output formats

Non-solid 7z (default)

Per-file compressed packs → file-level random access.

`--method`	Codec ID	Notes
`lzma2` (default)	LZMA2 `0x21`	Best ratio among built-in options; slower
`zstd`	Zstd `04 F7 11 01`	Best speed × ratio for many small files
`lz4`	LZ4 `04 F7 11 04`	Fastest; levels 1–2 flex, ≥3 HC with `lz4-hc`

Empty files use empty flags; mtimes from source
Parallel encode: fixed worker pool + 500M in-flight uncompressed budget
Symlinks / hard-link members are skipped (first regular-file body for a hard-linked inode is kept)

Seekable-zstd

Single seekable Zstd stream (zeekstd) with length-prefixed members and a trailer index (name → uncompressed offset).

Infer from bare -o *.zst when --format omitted
--method is 7z-only (omit for this format)
Layout: docs/FORMAT_SEEKABLE_ZSTD.md

tar-zstd

Valid ustar/pax tar inside seekable Zstd + RATAIDX1 member index (path, size, mtime, mode, uid, gid; uname/gname in tar headers).

Includes:

Parent directory members (typeflag='5', trailing /)
Symbolic links (typeflag='2')
Hard links (typeflag='1', linkname = first archive path for the inode)

Infer from -o *.tar.zst or *.tzst.
Layout: docs/FORMAT_TAR_ZSTD.md

tar-lz4

Same tar + index idea with independent LZ4 frames + cleartext RATLFRM1 frame table (not a standard “seekable LZ4” format).

Infer from -o *.tar.lz4 or *.tlz4.
Layout: docs/FORMAT_TAR_LZ4.md

Format inference

Output path ends with	Resolved format
`.tar.zst` / `.tzst`	tar-zstd
`.tar.lz4` / `.tlz4`	tar-lz4
`.zst`	seekable-zstd
`.7z` / other	7z

Selection & filters

Full semantics: docs/SELECTION.md · rsync parity: docs/RSYNC_PARITY.md

Core rules

Rule	Behavior
First match wins	Ordered rule list
No match	Include (default)
No `/` and no `**` in pattern	Basename match (`*.tmp` matches `dir/a.tmp`)
Leading `/`	Anchored from archive root
Trailing `/`	Directory-only (exclude prunes the tree)
Unanchored multi-segment	End-anchored suffix (`foo/bar` matches `a/foo/bar`)
`*` / `?`	One path segment
`**`	Across segments; forces full-path mode
`[...]` classes	Not in the dialect — brackets match literally; a parse-time warning is printed
`\` in patterns	Treated as `/` (Windows-style), not as a wildcard escape — parse-time warning
Filter file caps	10 MiB or 1,000,000 lines per file

Rule build order (`create`)

Clap stores each flag type in its own list — heterogeneous flags are not interleaved by CLI position:

1. --include-from FILE…
2. --exclude-from FILE…
3. --filter-from FILE…     ← preferred for full ordered lists
4. --filter RULE…
5. --include PATTERN…      (all includes as a batch)
6. --exclude PATTERN…      (all excludes as a batch)

Prefer --filter-from or repeated --filter for rsync-like ordered mixes.

Prune note (differs from rsync): rsync never scans an excluded directory, so an include for a path under it (+ skipme/keep.txt before - skipme/) finds nothing. rsync-archive conservatively keeps walking when an include could match below, so that include does take effect. The selected set diverges from rsync only when an include rule precedes the excluding rule and targets a path under the excluded tree; when the exclude comes first, first-match still drops those paths (identical to rsync). See docs/RSYNC_PARITY.md (P2-5).

# Include-only idiom (batch order works)
rsync-archive create -o o.7z --include '*.c' --exclude '*' ./src/

# True ordered mix
rsync-archive create -o o.7z --filter '- *' --filter '+ *.c' ./src/
rsync-archive create -n -o o.7z --filter-from rules.txt ./src/

Filter line syntax:

+ pattern
- pattern
include pattern
exclude pattern
# comments and blank lines ignored

Source mapping

Source	Behavior
`SRC` without trailing `/`	Archive paths include the directory name (`photos` → `photos/a.jpg`)
`SRC/` with trailing `/`	Strips the directory name (`photos/` → `a.jpg`)
`SRC` as a symlink	Followed (rsync arg behavior); members keep the link's name
`--files-from`	Exclusive of `SRC...`; relative lines keep path; absolute lines use basename
`--files-from-skip-missing`	Soft-skip missing list lines (default: hard-fail)
`--include-cwd`	Pack CWD at archive root; ignores rsync filters
Output self-exclusion	The `-o` file and its `.partial` temps are excluded from every source (SRC walks, `--files-from`, `--include-cwd`)
Multi-SRC missing roots	≥2 roots (or SRC + `--include-cwd`): soft-skip missing root; single missing SRC hard-fails

Selection pipeline

1. Master collect   SRC… / --files-from / --include-cwd
2. Rsync filters    include · exclude · filter (-from)
3. Per-file global  --max-size · --min-size · --newer-than
4. --file-size-from pattern max=SIZE (first match wins)
5. Dir budgets      --dir-max-size[-from] then --dir-max-files[-from]
6. Global fill      --max-total-size then --max-files  (newest mtime first)

Restriction lists only affect matching paths; unlisted paths ignore that list.

Restriction list formats

--file-size-from

**/*.log          max=100M
var/log/app.log   max=10M

--dir-max-size-from

logs/             max=500M
cache/            max=1G files=50
logs/=100M        # legacy PATH=SIZE

--dir-max-files-from

logs/=10
cache/ files=50

Members & live trees

Kind	tar-zstd / tar-lz4	7z / seekable-zstd
Regular file	Full body	Full body
Symlink	Stored as link	Skipped
Hard link (Unix)	Stored as hard link (size 0)	First body only
Special (fifo/device/…)	Skipped	Skipped

Encode soft-skip: members that vanish between selection and open are omitted (counted as vanished); a source-side EIO mid-read drops just that member instead of aborting the run. 7z rolls back partial packs; tar/seekable re-stat after open and zero-pad short mid-reads with a warning (counted as padded). A file that grows past 0 after being selected as empty is archived with its real current content; a file that grows during a tar/seekable read is truncated at the committed size and noted as padded. If all members soft-skip, create errors unless --allow-empty.

--strict: the archive is still written, but create exits 1 when any selected member vanished or was padded at encode (rsync exit-23 style) — automation can distinguish a complete archive from a degraded one.

CLI reference

Global

Flag	Meaning
`-v` / `-vv`	Debug / trace logging on stderr (`info` → `debug` → `trace`)
`RUST_LOG`	Overrides verbosity filter

Exit codes: 0 success · 1 operational error · 2 usage error

`create`

rsync-archive create -o OUT [OPTIONS] [SRC...]

Output & control

Flag	Default	Meaning
`-o`, `--output`	required	Output path
`--format` / `--output-format`	infer / `7z`	`7z` · `seekable-zstd` · `tar-zstd` · `tar-lz4`
`-n`, `--dry-run`	off	List selection only; no write
`--force`	off	Overwrite existing `-o`
`--verify`	off	Verify finished archive before commit (failed verify keeps previous `-o`)
`--allow-empty`	off	Empty / all-vanished → exit 0, no `-o`
`--strict`	off	Archive still written, but exit 1 if any member vanished or was padded at encode
`--level`	`5`	Compression level 0–9
`--method`	`lzma2`	7z only: `lzma2` · `zstd` · `lz4`
`--threads`	auto	7z only: encode workers (many tiny files → 1, else CPUs)
`--encode-concurrency`	`0`	7z only: max concurrent encodes (`0` = auto)
`--encode-size-budget`	`500M`	7z only: max in-flight uncompressed size (`0` = unlimited)

Selection inputs

Flag	Default	Meaning
`SRC...`	—	Source paths (required unless `--files-from` or `--include-cwd`)
`--files-from`	—	Master collect list (exclusive of `SRC...`)
`--files-from-skip-missing`	off	Soft-skip missing/unreadable list lines
`--include-cwd`	off	Pack CWD at archive root; ignores rsync filters

Filters

Flag	Meaning
`--exclude` / `--include`	Rsync-style patterns (repeatable; batched)
`--exclude-from` / `--include-from`	Pattern files (repeatable)
`--filter-from`	Ordered `+/-` filter file (repeatable; preferred)
`--filter`	`+ pattern` / `- pattern` (repeatable; CLI order among filters)

Size, count & age limits

Flag	Meaning
`--dir-max-size PATH=SIZE`	Cap selected bytes under dir tree (repeatable)
`--dir-max-size-from FILE`	Dir size/count list file
`--dir-max-files PATH=N`	Cap file count under dir tree (repeatable)
`--dir-max-files-from FILE`	Dir file-count list file
`--file-size-from FILE`	Per-path max size (`PATTERN max=SIZE`)
`--max-total-size SIZE`	Global selected-byte cap (newest-first)
`--max-files N`	Global max selected file count (newest-first)
`--max-size SIZE`	Skip any single file larger than SIZE
`--min-size SIZE`	Skip files smaller than SIZE (`0` = off)
`--newer-than DURATION`	Keep mtime within window (`7d`, `24h`, `30m`, `90s`)

SIZE syntax: raw bytes or K / M / G suffixes (e.g. 100M, 1G).
DURATION syntax: integer + optional d / h / m / s.

`embed`

rsync-archive embed -o OUT [OPTIONS] FILE...

Flag	Default	Meaning
`-o`, `--output`	required	Master `.7z` path
`FILE...`	required	Inputs to embed (typically finished `.7z`)
`-n`, `--dry-run`	off	List planned member names only
`--force`	off	Overwrite existing `-o`
`--prefix PREFIX`	—	Prefix all member names
`--keep-path`	off	Keep path as name (default: basename)
`--require-7z`	off	Fail if missing 7z magic
`--allow-any`	off	Allow non-7z store blobs (silent)
`--verify`	off	Post-write test of master archive

rsync-archive embed -o master.7z nest1.7z nest2.7z
rsync-archive embed -o master.7z --keep-path --prefix packs/ ./build/a.7z
rsync-archive embed -o master.7z --require-7z --verify a.7z b.7z
rsync-archive embed -o master.7z --allow-any --dry-run blob.bin

Performance notes

Fair non-solid benchmarks vs stock 7zz / 7-Zip-zstd (host: 12-core, 2026-07-28). Full tables: docs/bench/RESULTS.md.

Goal	Winner
Wall time (zstd / lz4, many small files)	rsync-archive — often 2–10× faster than 7zz-zstd
Wall time (lzma2 single-thread)	stock 7zz still ~1.2–1.4× ahead
Wall time (lzma2 multi-thread, small L5)	rsync-archive can beat 7zz
Archive size / ratio	Native tools denser (~1.3–1.9× smaller)

Product picks:

Priority	Recommendation
Speed	`--method zstd`
Best ratio we offer	`--method lzma2` (+ optional `--features native-codecs`)
Max throughput	`--method lz4`
Metadata + links	`--format tar-zstd` or `tar-lz4`
Simple byte-range stream	`--format seekable-zstd`

cargo build --release --bin rsync-archive --bin bench_compress
./target/release/bench_compress run --scale small --threads 1,4 --level 1,5 --methods all

Architecture

src/
  main.rs, cli.rs, lib.rs, error.rs
  archive/
    sevenz/            # non-solid header, store writer, LZMA2/Zstd/LZ4 packs
    seekable_zstd/     # seekable-zstd create + list/extract helpers
    tar_zstd/          # RA tar.zst (seekable Zstd + RATAIDX1)
    tar_lz4/           # RA tar.lz4 (multi-frame LZ4 + RATLFRM1 + RATAIDX1)
    tar_common.rs      # shared ustar/pax + index helpers
  select/              # pathnorm, rules, matcher, walk, dir budgets, global restrict
  pipeline/
    create.rs          # selection + multi-format write
    embed.rs           # store outer embed
    output.rs          # unique partial temps, --force, fsync + atomic rename, verify-before-commit
  util/                # size parse, soft log, auto threads
docs/                  # design, selection, format layouts, benches
tests/                 # CLI smoke, e2e create/embed/formats, filter parity, soft-fail
packaging/             # nfpm Rocky 8/9/10 RPMs (packages.yml + release.yml)

Implementation status

Stage	Status
Bootstrap, foundations, store writer	Done
Embed pipeline	Done
Rsync filter engine + walk + dry-run	Done
Non-solid LZMA2 create + verify	Done
Large-file streaming e2e	Done
Parallel encode (threads / concurrency / budget)	Done
Zstd + LZ4 in 7z	Done
Seekable-zstd / tar-zstd / tar-lz4	Done
Dir size & file-count budgets	Done
Global log-collection limits	Done
Encode performance (OPT-01..14)	Done

See docs/DESIGN.md and docs/BACKLOG.md for roadmap items (e.g. --older-than, head/tail sampling, free-disk guards).

CI & releases

Workflow	Trigger	What
CI	push / PR to `main`	Build on Ubuntu 22.04 / 24.04 and Rocky 8 / 9 / 10; full `cargo test` on Ubuntu 24.04
Packages	PR touching `packaging/` / release+packages workflows / `Cargo.*`	Build + `dnf install` Rocky 8 / 9 / 10 `.rpm` artifacts (no GitHub Release)
Release	tag `v*`	Per-distro `.tar.gz` + Rocky 8 / 9 / 10 `.rpm` + `.sha256` assets

# Maintainers: cut a release (Cargo.toml version must match the tag)
git tag -a v0.6.1 -m "v0.6.1"
git push origin v0.6.1

Documentation

Document	Contents
`docs/DESIGN.md`	Full design, stages, decisions
`docs/SELECTION.md`	Filter & restriction source of truth (v1)
`docs/RSYNC_PARITY.md`	Parity matrix vs rsync
`docs/FORMAT_SEEKABLE_ZSTD.md`	Seekable-zstd on-disk layout
`docs/FORMAT_TAR_ZSTD.md`	tar.zst layout
`docs/FORMAT_TAR_LZ4.md`	tar.lz4 layout
`docs/BENCH.md`	Benchmark harness & fairness rules
`docs/bench/RESULTS.md`	Published numbers
`docs/BACKLOG.md`	Future work
`AGENTS.md`	Contributor / agent policy

Contributing

Every behavior-changing change must update docs and include regression tests in the same PR.

cargo test
cargo run -- --help
./scripts/v1_acceptance.sh   # when present / for release checks

Live rsync parity tests soft-skip if rsync is not installed (CI stays green).

Related projects

Project	Relationship
archiveconverter	Solid → non-solid conversion; outer store append sibling to embed
rsync	Selection semantics inspiration
7-Zip / 7-Zip-zstd	Native non-solid peers for benchmarks

License

Select with confidence. Archive without thrashing the disk. Extract one file without unpacking the world.

↑ Back to top

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
.grok/skills		.grok/skills
docs		docs
packaging		packaging
scripts		scripts
src		src
tests		tests
.gitignore		.gitignore
AGENTS.md		AGENTS.md
Cargo.lock		Cargo.lock
Cargo.toml		Cargo.toml
LICENSE		LICENSE
README.md		README.md

Latest commit

History

Folders and files

Repository files navigation