Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
a5720fe
docs(spec): rsync-over-ssh NAS transport design + implementation plan
Xander-git Jun 10, 2026
d36f7ac
feat(constants): add SyncTransport enum for NAS transport selection
Xander-git Jun 10, 2026
88f3722
feat(config): nas.transport selector + ssh fields on NasConfig
Xander-git Jun 10, 2026
c7497e4
feat(sync): parse rsync --list-only output into RemoteManifest
Xander-git Jun 10, 2026
596dde0
fix(lint): sort SyncTransport in __all__ + raw-string match pattern i…
Xander-git Jun 10, 2026
dfc4634
feat(sync): NasTransportDriver protocol + RcloneDriver.lsjson_manifest
Xander-git Jun 10, 2026
775148a
feat(sync): RsyncSshDriver skeleton — classifier, ssh args, push
Xander-git Jun 10, 2026
d7b0acd
feat(sync): RsyncSshDriver.check via --checksum dry-run itemize
Xander-git Jun 10, 2026
70ba629
feat(sync): RsyncSshDriver listing manifest + reachability probe
Xander-git Jun 10, 2026
22e440c
feat(sync): build_nas_driver transport factory
Xander-git Jun 10, 2026
dd68909
feat(sync): route NASSyncClient through build_nas_driver with stage-m…
Xander-git Jun 10, 2026
9b39e2b
refactor(sync): Verifier takes NasTransportDriver, drop vestigial no-…
Xander-git Jun 10, 2026
4897c8d
feat(tray): transport-aware NAS setup gate + Test-connection probe
Xander-git Jun 10, 2026
4a17bc9
feat(paths): rsync-mode missing-field reason for the NAS setup gate
Xander-git Jun 10, 2026
c17cba8
fix(security): end-of-options guard + leading-dash remote rejection i…
Xander-git Jun 10, 2026
19a6cac
feat(ui): transport-aware NAS-remote settings section + banner copy
Xander-git Jun 10, 2026
8b63c90
test(sync): stub rsync binary + driver round-trip coverage
Xander-git Jun 10, 2026
869c9ec
test(integration): rsync-over-ssh docker leg + characterization suite
Xander-git Jun 10, 2026
079dac5
docs: rsync-over-ssh NAS transport setup walkthrough + spec sections
Xander-git Jun 10, 2026
4b8d583
fix(sync): review fixes — narrow ssh auth marker, loud parser warning…
Xander-git Jun 10, 2026
e1f8e5b
style: ruff-format branch-touched files + spec status -> implemented
Xander-git Jun 10, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,9 @@ state/
# NAS emulator (tests/docker/) — keep the layout, ignore generated state
tests/docker/nas-data/*
!tests/docker/nas-data/.gitkeep
# rsync-over-ssh keypair generated by entrypoint on first boot (never commit)
tests/docker/keys/*
!tests/docker/keys/.gitkeep

# Local secrets / env (never commit)
.env
Expand Down
9 changes: 8 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -124,7 +124,10 @@ container weekly and on every merge to `main`.
## NAS sync setup

ExLab-Wizard syncs runs to your NAS via [rclone](https://rclone.org/) named
remotes. The app never stores NAS passwords — you configure the remote once
remotes (default) or via `rsync` over ssh for cluster nodes where IT blocks
SMB/SFTP. The app never stores NAS passwords or ssh key passphrases.

**rclone transport (default — lab acquisition PCs):** configure the remote once
with `rclone config` and wire the remote name into `config.yaml`:

```yaml
Expand All @@ -136,6 +139,10 @@ nas:
Step-by-step instructions (SFTP and SMB walkthroughs, performance tuning,
tray-service caveats): **[`docs/setup/rclone-remote-setup.md`](docs/setup/rclone-remote-setup.md)**.

**rsync-over-ssh transport (cluster nodes where IT blocks rclone):** set
`nas.transport: rsync_ssh` and `nas.remote: user@host`. Keypair provisioning
and `known_hosts` pre-loading walkthrough: **[`docs/setup/rsync-ssh-setup.md`](docs/setup/rsync-ssh-setup.md)**.

Settings → **NAS Remote** → **Test connection** verifies the remote is
reachable before you start syncing.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,10 @@ exlab_wizard/
generator.py # merge field layers, render YAML+Markdown, write README.md + readme_fields.json
sync/
nas_client.py # NASSync interface (see §7.1)
transports/
__init__.py # NasTransportDriver Protocol (push/check/lsjson_manifest/about) + build_nas_driver factory
rclone.py # RcloneDriver — default transport; thin rclone subprocess wrapper
rsync_ssh.py # RsyncSshDriver — rsync-over-ssh transport for cluster nodes (2026-06-10)
lims/
client.py # LIMSClient (read-only in v1; see §7.2). Uses httpx for REST + cookie session.
schemas.py # LIMSProject, LIMSUser as msgspec.Struct types
Expand Down Expand Up @@ -178,6 +182,8 @@ tests/

The `ui/` package depends on `controller/` and the API schema modules but never the reverse: no backend module imports from `ui/`. This is the testability boundary — backend can be exercised without a browser.

**NAS transport abstraction (2026-06-10).** `sync/transports/__init__.py` defines `NasTransportDriver`, a structural Protocol with four async methods (`push`, `check`, `lsjson_manifest`, `about`) consumed by `NASSyncClient` and the tray probe. `RcloneDriver` and `RsyncSshDriver` both implement the protocol. Every construction site routes through the `build_nas_driver(nas: NasConfig, perf: RclonePerf) -> NasTransportDriver` factory; the factory branches on `nas.transport`. Stage-mode equipment always gets a `RcloneDriver` regardless of `nas.transport` because its targets are rclone named-remote strings, not `user@host` ssh targets (see §7.1 and `CLAUDE.md`).

### 4.3.1 The `constants/` package

A small set of values appears in many specifications and must stay synchronized across the codebase: schema version numbers, file names of cache files, regex patterns, keyring service identifier, enum string values. The `constants/` package is the single source of truth for these. Rules:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -60,9 +60,30 @@ QUEUED → RUNNING → AWAITING_VERIFY → VERIFIED → CLEANUP_ELIGIBLE → CLE

Job rows persist across restarts. On startup, NASSync requeues any `RUNNING` or `AWAITING_VERIFY` jobs (treating them as `QUEUED` and `VERIFIED → AWAITING_VERIFY` respectively, since transport may have completed but verification didn't run).

### 7.1.3 Transport driver (`RcloneDriver`)

The sole transport for v1 is `RcloneDriver` (`exlab_wizard.sync.transports.rclone`),
### 7.1.3 Transport drivers

The transport used by each instance is selected by `nas.transport` in
`config.yaml` (default: `rclone`). The factory `build_nas_driver(nas, perf)`
in `exlab_wizard.sync.transports` returns the appropriate driver.

| Transport | Selected by | Binary | Auth | Verify authority |
|-----------|------------|--------|------|-----------------|
| `rclone` (default) | `nas.transport: rclone` (or absent) | `rclone` | Named remote in `rclone.conf` | `rclone check --download` — pulls file bytes to the client and hashes locally |
| `rsync_ssh` | `nas.transport: rsync_ssh` | `rsync` | ssh key (`BatchMode=yes`) | `rsync -rni --checksum` dry-run — computes checksums inside the rsync protocol on the remote side |

**Verify authority differs by transport.** `RcloneDriver.check` downloads the
remote bytes to the client to hash them locally, meaning the hash computation
runs on the client and the network carries the full file payload. `RsyncSshDriver.check`
invokes a dry-run with `--checksum` inside the rsync protocol, delegating the
hash computation to the remote rsync process; only the itemize-change output
(a handful of flag bytes per file) crosses the wire. Both signal
"content differs" correctly, but in rsync mode the trust is placed on the
remote-side rsync binary rather than a local hash. This is an accepted
trust-posture change (2026-06-10 spec-review).

#### `RcloneDriver`

`RcloneDriver` (`exlab_wizard.sync.transports.rclone`) is the default transport,
a thin wrapper around the `rclone` binary. The driver exposes four operations:

| Method | Command | Purpose |
Expand Down
17 changes: 13 additions & 4 deletions docs/design_specs/design_spec_sections/09_Configuration_File.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,13 +75,22 @@ equipment:
sync_mode: "nas"

# Top-level NAS sync configuration. A single named rclone remote covers all
# nas-mode equipment. Credentials live entirely in rclone.conf.
# nas-mode equipment. Credentials live entirely in rclone.conf (rclone mode)
# or in the operator's ~/.ssh/ (rsync_ssh mode).
nas:
remote: "lab-nas" # remote name from rclone.conf (set up with rclone config)
remote: "lab-nas" # rclone mode: remote name from rclone.conf (set up with rclone config)
# rsync_ssh mode: "user@host" target prefix
base_root: "lab" # path on the remote under which equipment folders live
# run target: lab-nas:/lab/<EQUIPMENT_ID>/<run-leaf>
rclone_config_path: "" # optional: pin --config <path> (blank = rclone default discovery)
mtime_tolerance_s: 2 # reconcile tolerance (s); absorbs SFTP/SMB modtime rounding
rclone_config_path: "" # rclone mode only: optional pin --config <path> (blank = default discovery)
mtime_tolerance_s: 2 # reconcile tolerance (s); absorbs SFTP/SMB/rsync modtime rounding
# rsync-over-ssh transport (2026-06-10). Cluster nodes where IT blocks
# rclone/SMB/SFTP can use rsync --server over ssh instead. Lab PCs keep
# the default rclone transport. See docs/setup/rsync-ssh-setup.md.
transport: "rclone" # "rclone" (default) | "rsync_ssh"
ssh_port: 22 # rsync_ssh mode: ssh port on the NAS (default 22)
ssh_identity_file: "" # rsync_ssh mode: path to the ed25519 private key
# (blank = ssh default key discovery; BatchMode=yes always set)
perf:
transfers: 4 # rclone --transfers; also the RAM dial on constrained machines
checkers: 8 # rclone --checkers
Expand Down
12 changes: 8 additions & 4 deletions docs/setup/rclone-remote-setup.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,13 @@
# Setting up your rclone remote(s)

ExLab-Wizard uses [rclone](https://rclone.org/) as its NAS transport. Rather than
storing NAS credentials inside the app, you configure a named remote once with
`rclone config` and then tell the app which remote name to use. The app never
sees a password — all connection details live in `rclone.conf`.
ExLab-Wizard uses [rclone](https://rclone.org/) as its **default** NAS transport.
Rather than storing NAS credentials inside the app, you configure a named remote
once with `rclone config` and then tell the app which remote name to use. The app
never sees a password — all connection details live in `rclone.conf`.

> **Cluster nodes where SMB/SFTP is blocked by IT:** use the `rsync_ssh`
> transport instead. See [`rsync-ssh-setup.md`](rsync-ssh-setup.md) for the
> key provisioning and `config.yaml` walkthrough.

---

Expand Down
173 changes: 173 additions & 0 deletions docs/setup/rsync-ssh-setup.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,173 @@
# Setting up rsync-over-ssh NAS transport

ExLab-Wizard's `rsync_ssh` transport is for cluster nodes where IT blocks
rclone (SMB/SFTP is disabled on the Synology) but allows rsync-over-ssh. Lab
acquisition PCs keep the default `rclone` transport unchanged; only cluster
instances need this walkthrough.

> **Also see:** [`rclone-remote-setup.md`](rclone-remote-setup.md) for the
> rclone/SMB setup used on lab acquisition PCs.

---

## How it works

Instead of a named rclone remote, the `rsync_ssh` transport drives `rsync -e
ssh` directly. `nas.remote` becomes `user@host` (the Synology service account),
and the app composes the full target as `user@host:/<base_root>/<equipment_id>/<run>`.
All four sync operations (push, manifest listing, content verify, connection
probe) use the rsync protocol channel only — no interactive ssh login, no
SFTP subsystem, no remote command execution. IT allowlists exactly
`rsync --server` over ssh; that is all this transport needs.

Credentials are ssh keys only (`BatchMode=yes`). No password is ever stored or
prompted. Key exchange happens before authentication, so the host-key can be
pre-provisioned without any login permission.

---

## Prerequisites

- **rsync** on the cluster node (`rsync --version` should report 3.x).
- **rsync** on the Synology NAS (installed via DSM Package Center or the
Synology rsync service; verify with `rsync --version` output recorded in
your runbook — see the "Recording rsync version" step below).
- **OpenSSH client** (`ssh`, `ssh-keygen`, `ssh-keyscan`) on the cluster node.

---

## Step 1 — Generate a dedicated keypair

Generate a key specifically for ExLab-Wizard NAS sync. Using a dedicated key
lets you revoke or rotate it without affecting your personal ssh access.

```bash
ssh-keygen -t ed25519 -f ~/.ssh/id_exlab -N ""
```

- `-N ""` sets an empty passphrase so the sync worker can use the key
non-interactively (`BatchMode=yes` requires no passphrase prompt).
- The private key is `~/.ssh/id_exlab`; the public key is `~/.ssh/id_exlab.pub`.

Restrict permissions:

```bash
chmod 600 ~/.ssh/id_exlab
```

---

## Step 2 — Install the public key on the Synology service account

Copy the public key to the person who manages your Synology NAS (lab IT or
the NAS admin). They need to append the contents of `~/.ssh/id_exlab.pub` to
the `~/.ssh/authorized_keys` file of the NAS service account
(`svc-sync@nas01.lab.example` in the example below).

If you have temporary password ssh access to the NAS service account:

```bash
ssh-copy-id -i ~/.ssh/id_exlab.pub -p <port> svc-sync@nas01.lab.example
```

Or share the one-line `~/.ssh/id_exlab.pub` content with your NAS admin for
manual installation.

---

## Step 3 — Pre-provision the NAS host key

`BatchMode=yes` means ssh will fail immediately if the host key is unknown
rather than prompting you interactively. Pre-provision the key before the
first sync run.

> **No login permission required.** The host-key exchange (Step 3) happens
> at the TCP/cryptographic layer, before authentication. You do not need
> shell access to the NAS to run `ssh-keyscan`.

```bash
ssh-keyscan -p <port> nas01.lab.example >> ~/.ssh/known_hosts
```

Verify the key fingerprint out-of-band against DSM or your NAS admin to
guard against a man-in-the-middle substitution:

```bash
# Compare this fingerprint against what DSM shows under
# Control Panel → Terminal & SNMP → SSH key fingerprints.
ssh-keygen -lf <(ssh-keyscan -p <port> nas01.lab.example 2>/dev/null)
```

---

## Step 4 — Record the NAS rsync version

Record the Synology's rsync version in your runbook. Differences in rsync
protocol versions between client and server can occasionally cause format
quirks in listing output. Run from the cluster node after keys are installed:

```bash
ssh -p <port> -i ~/.ssh/id_exlab svc-sync@nas01.lab.example rsync --version
# Record the first line (e.g. "rsync version 3.2.3 ...") in your lab runbook.
```

---

## Step 5 — Add the `rsync_ssh` block to `config.yaml`

```yaml
nas:
transport: "rsync_ssh"
remote: "svc-sync@nas01.lab.example" # user@host — doubles as the target prefix
base_root: "/volume1/lab" # absolute path on the NAS
ssh_port: 22 # optional, default 22
ssh_identity_file: "~/.ssh/id_exlab" # optional; blank = ssh default key discovery
mtime_tolerance_s: 2 # modtime tolerance for reconcile (seconds)
bandwidth:
upload_mbps: null # null = unlimited
```

The full target path for a run is:

```
svc-sync@nas01.lab.example:/volume1/lab/<EQUIPMENT_ID>/<run-directory-name>
```

---

## Step 6 — Test the connection

Open **Settings → NAS Remote → Test connection** in ExLab-Wizard.

> **Degraded probe by design:** the rsync transport cannot query free-space
> without remote command execution (which IT blocks). Test-connection reports
> reachable + auth ok or a classified failure — no free-space info is shown.
> This is expected and not a configuration error.

---

## Failure modes (BatchMode=yes)

When ssh runs in `BatchMode=yes` it fails immediately instead of prompting.
The error is classified by the driver and surfaced in the Settings panel and
the log.

| Symptom | Likely cause | Fix |
|---------|-------------|-----|
| `Permission denied (publickey)` | Key not installed on the NAS service account, or wrong identity file | Verify `ssh_identity_file` path; re-run `ssh-copy-id` |
| `Host key verification failed` | Host key not in `~/.ssh/known_hosts` | Re-run `ssh-keyscan` (Step 3); verify fingerprint |
| `Connection refused` / timeout | Network or IT policy blocking the port | Check firewall rules; confirm with IT that the NAS ssh port is reachable from this cluster node |
| `Too many authentication failures` | SSH agent offering too many keys before the correct one | Set `ssh_identity_file` explicitly in config to skip key negotiation |

---

## Notes on `perf` and rclone-only fields

`nas.perf.transfers` and `nas.perf.checkers` are rclone parallelism dials.
They are **ignored** (not rejected) by the `rsync_ssh` transport — rsync is
single-stream per invocation. Set them to rclone defaults for forward
compatibility; they will be used again if you ever switch the instance back
to rclone.

`nas.rclone_config_path` is also ignored by `rsync_ssh` — there is no
rclone involved.
Loading