Skip to content

perf: Reduce memory overhead for multipart uploads - #771

Merged
kylebarron merged 11 commits into
mainfrom
kyle/multipart-upload-memory
Aug 20, 2026
Merged

perf: Reduce memory overhead for multipart uploads#771
kylebarron merged 11 commits into
mainfrom
kyle/multipart-upload-memory

Conversation

@kylebarron

@kylebarron kylebarron commented Aug 20, 2026

Copy link
Copy Markdown
Member

As described well in #767, we currently always copy input into a scratch buffer for any pull source. But for buffer sources this is unnecessary as we can upload the input buffer directly.

Change list

  • Make Buffer a first-class variant of PutInput, rather than shoe-horning it into a PullSource. Also we no longer need to use a Cursor wrapping the Buffer.
  • Don't use a scratch buffer copy when uploading a buffer
  • For a pull source, create a single scratch buffer outside the loop, which we reuse on each iteration, rather than creating a new buffer allocation in every loop iteration.

This PR was created in discussion with Claude but I wrote the source code changes myself. Claude wrote the new tests.

Closes #767

@ds-release-bot ds-release-bot Bot added the perf label Aug 20, 2026
@kylebarron
kylebarron enabled auto-merge (squash) August 20, 2026 23:25
@kylebarron
kylebarron merged commit 29d9dd1 into main Aug 20, 2026
9 checks passed
@kylebarron
kylebarron deleted the kyle/multipart-upload-memory branch August 20, 2026 23:26
@kylebarron kylebarron self-assigned this Aug 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Avoid multipart buffer copies for in-memory buffers

1 participant