Skip to content

send-file: stream the body instead of buffering the whole file on both sides - #97

Merged
myobie merged 1 commit into
mainfrom
fix/send-file-streams-instead-of-buffering
Aug 30, 2026
Merged

send-file: stream the body instead of buffering the whole file on both sides#97
myobie merged 1 commit into
mainfrom
fix/send-file-streams-instead-of-buffering

Conversation

@myobie

@myobie myobie commented Aug 29, 2026

Copy link
Copy Markdown
Collaborator

Finding 7 of the 2026-08-29 review (reproduced: receiver peaked +1.0 GB, sender +1.0 GB on a 1 GiB transfer).

The hole

The sender read the file whole (std::fs::read in the daemon) and passed the slice to send; the receiver allocated vec![0u8; header.len] and read_exact into it. So a 1.5 GiB transfer held about 1.5 GiB of resident memory on each daemon at once. On a host with a MemoryMax, that is enough to kill the daemon mid-transfer, taking every shell, exec and sync session with it.

The fix

  • Sender: send_file(stream, name, path) opens the file, reads its size from metadata, and send_from_reader streams exactly len bytes with tokio::io::copy over a bounded buffer. The daemon passes the path instead of std::fs::read-ing it.
  • Receiver: streams the body straight to the temp file with the same bounded copy, reading exactly header.len bytes via take(len) so it never waits for an EOF the sender does not send before its ack.

Neither side allocates against the file size. The wire format is unchanged (header + len body bytes + ack), so a streaming build and an old build interoperate in either direction.

Tests

  • a_large_file_streams_through_in_chunks: round-trips 5 MiB over a 64 KiB duplex (many chunks), bytes intact.
  • a_short_stream_is_refused_and_leaves_no_file: a sender that promises 1000 bytes and sends 10 is refused, and no file (partial or final) reaches the inbox. This is a strict improvement — a short transfer previously errored on read_exact, now it is refused explicitly and cleans up.

The memory improvement is structural (no vec![0; len] on either side); the file that lands is byte-identical, so the unit tests prove correctness and the reviewer's two-daemon RSS measurement is the memory proof. cargo test green.

@myobie
myobie force-pushed the fix/send-file-streams-instead-of-buffering branch from 65b2db1 to 1e89b0a Compare August 29, 2026 13:24
…h sides

Finding 7 of the 2026-08-29 review. The sender read the file whole with
std::fs::read and the receiver allocated header.len bytes up front, so a
1.5 GiB transfer cost about 1.5 GiB of resident memory on each daemon at
once. On a host with a memory ceiling that is enough to kill the daemon
mid-transfer.

send_file opens the file and send_from_reader streams exactly len bytes with
tokio::io::copy over a bounded buffer; the daemon passes the path rather than
reading it. The receiver streams the body straight to its temp file with the
same bounded copy, reading exactly header.len bytes via take() so it never
waits for an EOF the sender does not send before its ack. Neither side
allocates against the file size. The wire format is unchanged, so a streaming
build and an old build interoperate.

A transfer that ends short of its declared length is now refused and leaves
no file, where before a short read would error on read_exact. Tests:
a_large_file_streams_through_in_chunks round-trips 5 MiB over a 64 KiB duplex;
a_short_stream_is_refused_and_leaves_no_file pins the truncation guard.
@myobie
myobie force-pushed the fix/send-file-streams-instead-of-buffering branch from 1e89b0a to c42615b Compare August 30, 2026 12:02
@myobie
myobie merged commit 470a131 into main Aug 30, 2026
2 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant