Skip to content

SC-9: compression support (--compress / -z using zlib) #13

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
iampryce merged 1 commit into from
Aug 2, 2026
Merged

SC-9: compression support (--compress / -z using zlib) #13

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
161 changes: 157 additions & 4 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,16 +11,19 @@ really walks, filters, diffs, transfers, and reconstructs files, applying
requested attributes along the way. See
[End-to-End Sync Pipeline](#end-to-end-sync-pipeline) below for exactly
how the pieces connect and, just as importantly, what's still explicitly
out of scope (compression, partial/append transfers, batch mode, full
`--delete`, and device/special files) - this is real, working sync, not
yet full feature parity. Hard links *are* now preserved, opt-in via
`-H`/`--hard-links` exactly like real rsync's own flag (see
out of scope (partial/append transfers, batch mode, full `--delete`, and
device/special files) - this is real, working sync, not yet full feature
parity. Hard links *are* now preserved, opt-in via `-H`/`--hard-links`
exactly like real rsync's own flag (see
[File Attribute Preservation](#file-attribute-preservation) below), and
`--dry-run`/`-n` is a genuine trial run - full planning, zero filesystem
changes - with real `--itemize-changes`/`-i` output matching rsync's own
format (see [Dry-Run Mode](#dry-run-mode) below). `--progress` and
`--stats` are now implemented too, both matching real rsync's own output
formats (see [Progress and Stats](#progress-and-stats) below).
`--compress`/`-z` now genuinely compresses a file's literal delta data
with zlib, including real rsync's own `--compress-level` and
`--skip-compress` (see [Compression](#compression) below).

`grsync --daemon` also now speaks a real subset of the rsync daemon
protocol - `rsyncd.conf` parsing, the `@RSYNCD` greeting/handshake, module
Expand Down Expand Up @@ -671,6 +674,156 @@ higher-priority branch. Locked in by
inert for this direction (`Sender` never even looks at
`ReceiverOptions`), and the upload itself still completes correctly.

## Compression

`--compress`/`-z` compresses a file's literal delta data with zlib
before it crosses the wire; `--compress-level` controls how hard, and
`--skip-compress` excludes already-compressed file types. All three
match real rsync's own semantics, verified against upstream's actual
source (`token.c`, `RsyncProject/rsync`) and `rsync.1`'s own documented
wording rather than assumed.

### What gets compressed, and what doesn't

Only a delta's literal data - the bytes a `DataOp` carries because they
didn't match anything in the receiver's signature - is ever compressed.
`CopyOp`s (plain block-index references, not data) and every
`FrameSignature` (checksums) are never touched, matching the ticket's
own scope exactly.

Rather than compressing each `DataOp` independently, `Sender` (via
`toWireDeltaOps`, `internal/pipeline/messages.go`) concatenates *all* of
one file's literal data into a single buffer and zlib-compresses that as
one unit per `FrameDelta` message, with a single `Compressed bool`
marker on the message itself; each `DataOp`'s own wire form then carries
only how many of the decompressed stream's bytes are its
(`wireDeltaOp.Length`), so the receiver can re-slice it back apart after
one decompression. This amortizes zlib's fixed ~8-byte header/trailer
overhead across a whole file instead of paying it again for every
separate literal run a scattered-changes file can produce - closer to
real rsync's own `zlibx` compression choice (one persistent per-file
deflate stream with matched data excluded from it) than compressing
op-by-op would have been. If the compressed result isn't actually
smaller than the raw literal data - realistic for a file whose total
changed content is tiny, exactly the case delta transfer exists for, or
for already-incompressible data that slipped past `--skip-compress` -
the file is sent uncompressed instead; `Compressed` is a real, checked
outcome, not just a request.

Compression is entirely a **sending-side** decision. `Receiver` takes no
compression-related options of its own at all: it simply decompresses
whatever each `deltaMessage.Compressed` marker says, on every transport,
which is possible because grsync only ever pushes (see
[Status](#status)) - `Sender` always runs on the local, requesting
process, never remotely, for every path currently wired into the CLI.

### `--compress-level`

Verified against real rsync's own source (`token.c`'s
`init_compression_level`) and `rsync.1`'s own documented wording: for
zlib compression, valid levels are **1 (fastest) to 9 (smallest), with 6
as the default**. `--compress-level=0` explicitly turns compression off
- overriding a `-z` given alongside it - and `--compress-level=-1` means
"use the default." Giving `--compress-level` alone, without `-z`,
implies compression (unless the resulting level is 0), matching real
rsync's own documented "the `--compress` option is implied" rule. An
out-of-range value is silently clamped into `[1, 9]`, matching
`rsync.1`'s own "too-large or too-small value" wording. All of this is
`ClampCompressLevel` (`internal/pipeline/compress.go`) and
`effectiveCompressOptions` (`internal/cli/sync.go`).

### `--skip-compress`

Overrides the built-in list of already-compressed file suffixes
(`gz`, `zip`, `jpg`, `mp3`, `mp4`, and 91 others) that are sent
uncompressed even with `--compress` on, since running zlib over an
already-compressed format wastes CPU for no size benefit. The full
default list is real rsync's own, copied verbatim from `rsync.1`'s own
documented default (`DefaultSkipCompressSuffixes`,
`internal/pipeline/compress.go`) rather than invented. An explicit
`--skip-compress=""` is a meaningful override in its own right ("skip
nothing"), matching real rsync's own documented meaning for it - grsync
tells that apart from "the flag was never given at all" via
`cmd.Flags().Changed`, not `opts.skipCompress`'s zero value, since an
empty string is both.

Matching is a plain, case-insensitive suffix list
(`--skip-compress=gz/jpg/mp3`); real rsync's own `--skip-compress`
grammar additionally supports bracketed character classes inside a
suffix (e.g. `mp[34]`), which grsync's version does not - a deliberate,
disclosed scope reduction, since plain suffixes cover the default list
and the overwhelming majority of real-world uses.

**Worth disclosing**: real rsync's own current documentation (as of this
writing) admits `--skip-compress` "has no effect" in its own latest
implementation, because none of its currently-supported compression
algorithms allow changing level mid-stream - its per-file persistent
deflate context, once opened, keeps compressing everything at the same
level regardless of what the suffix list says. grsync's frame-per-file
design has no such persistent stream to be stuck with: `toWireDeltaOps`
makes a fresh, genuine "compress this file's literal data or don't" call
for every file, so `--skip-compress` actually works here - a real
improvement made possible by the architectural difference, not a silent
divergence from upstream's documented behavior.

### Interaction with `--stats` and `--dry-run`

`--stats`' "Total bytes sent"/"Total bytes received" (see
[Progress and Stats](#progress-and-stats)) already measure genuine wire
traffic via `countingReadWriter`, which wraps the connection itself - so
compressed bytes are reflected automatically, no changes needed for this
ticket. Since `Stats` is computed on the *receiving* side, it's
specifically **"Total bytes received"** that shrinks with compression
for an upload (`Sender` on the far end sends compressed data, this side
receives it) - "Total bytes sent" reflects this side's own small
signature/ack traffic back to the sender, which compression doesn't
touch. `Total file size` is unaffected either way, since it describes
the files themselves, not what crossed the wire.

`--dry-run` and `--compress` compose cleanly: the full signature/delta
exchange - including compressing the delta's literal data - still runs
during a dry run exactly as it would for a real sync (see
[Dry-Run Mode](#dry-run-mode)'s own explanation of why), so itemize
output stays accurate; only the final disk write is skipped, and
compression has nothing to do with that.

### Across transports

- **Local**: `Sender` runs in-process with the `CompressOptions`
`effectiveCompressOptions` computed from the CLI flags - no different
from any other in-process call.
- **SSH**: unlike `--dry-run`/`--itemize-changes`/`--verbose`/
`--progress`/`--stats` (see [Dry-Run Mode](#dry-run-mode)'s own
"Across transports" section), `--compress` needs **no remote argv
change at all**: `Sender` runs locally for this transport too, and the
remote `--server` process's `Receiver` just reacts to each
`deltaMessage`'s own `Compressed` marker, exactly like every other
transport. Verified over a real SSH connection to `127.0.0.1` by
`TestSSHLocalhost_CompressDoesNotBreakTheTransfer` (skipped gracefully
without a local `sshd`).
- **`rsync://` daemon, upload (`DirectionPut`)**: `Sender` runs on the
*client* side for this direction (see
[rsync Daemon Mode](#rsync-daemon-mode)), exactly where `--compress`'s
decision belongs - no daemon-protocol extension needed, the same way
SSH needs none. Verified over a real TCP connection by
`TestDaemon_RealTCP_PutWithCompressUploadsCorrectly`.
- **`rsync://` daemon, download (`DirectionGet`)**: the daemon's own
`Sender` call for a module download has no CLI wiring at all yet - see
[Status](#status)'s "push only" scope boundary, already established
before this ticket - so it always runs with compression disabled
(`pipeline.CompressOptions{}`), consistent with that existing
boundary, not a new gap introduced here.

### Hard links

A hard-link group's secondary members never go through the
signature/delta exchange at all - `Sender` and `Receiver` both skip them
outright, recreating the link directly instead (see
[File Attribute Preservation](#file-attribute-preservation)) - so
`--compress` is moot for them by construction, with nothing to compress
in the first place. `TestSenderReceiver_CompressWorksWithHardLinks`
confirms that skip still holds correctly with compression enabled.

## rsync Daemon Mode

`internal/daemon` implements grsync's `--daemon` server mode: a second way
Expand Down
63 changes: 38 additions & 25 deletions internal/cli/root.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,29 +41,31 @@ type FilterRule struct {
// in one struct (rather than loose variables) makes it straightforward to
// pass a single value into internal/sync once that package exists.
type options struct {
archive bool
verbose bool
compress bool
recursive bool
dirs bool
dryRun bool
delete bool
progress bool
perms bool
times bool
owner bool
group bool
links bool
hardLinks bool
itemize bool
stats bool
filterRules []FilterRule
rsh string
server bool
daemon bool
config string
port int
passwordFile string
archive bool
verbose bool
compress bool
compressLevel int
skipCompress string
recursive bool
dirs bool
dryRun bool
delete bool
progress bool
perms bool
times bool
owner bool
group bool
links bool
hardLinks bool
itemize bool
stats bool
filterRules []FilterRule
rsh string
server bool
daemon bool
config string
port int
passwordFile string
}

// filterRuleFlag implements pflag.Value. Each of --exclude/--include/
Expand Down Expand Up @@ -106,7 +108,7 @@ func NewRootCmd() *cobra.Command {
Short: "grsync synchronizes files between one or more sources and a destination",
Long: "grsync is an rsync-inspired file synchronization tool.\n" +
"Local-to-local and local-to-remote (SSH) syncs are supported, including --dry-run, " +
"--itemize-changes, --progress, and --stats; compression and full --delete are not yet.",
"--itemize-changes, --progress, --stats, and --compress; full --delete is not yet.",
// --server takes exactly one positional arg (the destination path)
// rather than the normal <source>...<destination> shape: it is how
// a remote-invoked grsync (e.g. `ssh host grsync --server /dest`)
Expand Down Expand Up @@ -141,7 +143,18 @@ func NewRootCmd() *cobra.Command {
flags.BoolVarP(&opts.verbose, "verbose", "v", false,
"mention each updated item's path (superseded by --itemize-changes when both are given, "+
"matching real rsync's own -v/-i relationship)")
flags.BoolVarP(&opts.compress, "compress", "z", false, "compress file data during transfer")
flags.BoolVarP(&opts.compress, "compress", "z", false,
"compress a file's literal delta data with zlib before sending it (checksums/signatures are never "+
"compressed); see the README's Compression section")
flags.IntVar(&opts.compressLevel, "compress-level", 0,
"explicitly set the zlib compression level (1-9, default 6) instead of --compress's implicit "+
"default; implies --compress even without -z, unless set to 0 (\"off\", which disables "+
"compression even if -z was also given) - matches real rsync's own --compress-level/--zl "+
"range, default, and off-semantics, verified against upstream source")
flags.StringVar(&opts.skipCompress, "skip-compress", "",
"override the default list of already-compressed file suffixes (slash-separated, e.g. gz/jpg/mp3) "+
"that --compress/-z sends uncompressed; an empty string means \"skip nothing\" - matches real "+
"rsync's own --skip-compress (see the README's Compression section for the full built-in default list)")
flags.BoolVarP(&opts.recursive, "recursive", "r", false, "recurse into directories")
flags.BoolVarP(&opts.dirs, "dirs", "d", false, "include directories themselves without recursing into their contents (implied by --recursive)")
flags.BoolVarP(&opts.dryRun, "dry-run", "n", false, "perform a trial run: full planning (file list, filters, deltas) with no filesystem changes")
Expand Down
11 changes: 9 additions & 2 deletions internal/cli/rsync_url.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,14 @@ const dialDaemonTimeout = 10 * time.Second
// Itemize/Verbose/Progress/Stats are all deliberately left unset - see
// runSync's own one-time note about why none of daemon-PUT's reporting
// output is available, printed before this function is ever called.
func syncToRsyncDaemon(src string, u daemon.URL, password daemon.PasswordFunc, walkOpts sync.WalkOptions, rules []sync.Rule, hardLinks bool, dryRun bool) error {
//
// copts, in contrast, is fully honored here: DirectionPut runs Sender on
// this (the client) side (see DialModule's own doc comment), exactly
// where --compress/-z's decision actually belongs - no wire extension
// like dryRunToken is needed for it at all, since the server's Receiver
// only ever reacts to what each deltaMessage's own Compressed marker
// says.
func syncToRsyncDaemon(src string, u daemon.URL, password daemon.PasswordFunc, walkOpts sync.WalkOptions, rules []sync.Rule, hardLinks bool, dryRun bool, copts pipeline.CompressOptions) error {
port := u.Port
if port == 0 {
port = daemon.DefaultPort
Expand All @@ -59,5 +66,5 @@ func syncToRsyncDaemon(src string, u daemon.URL, password daemon.PasswordFunc, w

user := resolveUser(u.User)
ropts := pipeline.ReceiverOptions{DryRun: dryRun}
return daemon.DialClient(nc, u.Module, user, password, daemon.DirectionPut, src, rules, walkOpts, sync.AttrOptions{HardLinks: hardLinks}, ropts)
return daemon.DialClient(nc, u.Module, user, password, daemon.DirectionPut, src, rules, walkOpts, sync.AttrOptions{HardLinks: hardLinks}, ropts, copts)
}
Loading