Skip to content

Decompress: reuse the caller-supplied buffer for unknown-size frames#168

Merged
KrisKennawayDD merged 1 commit into
1.xfrom
kris.kennaway/decompress-reuse-caller-buffer
Jul 24, 2026
Merged

Decompress: reuse the caller-supplied buffer for unknown-size frames#168
KrisKennawayDD merged 1 commit into
1.xfrom
kris.kennaway/decompress-reuse-caller-buffer

Conversation

@KrisKennawayDD

@KrisKennawayDD KrisKennawayDD commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Problem

Decompress sizes its destination buffer from decompressSizeHint(src), which reads the frame's content-size field. Frames that don't carry that field make the hint fall back to a pessimistic upper bound (max(50×len(src), decompressSizeBufferLimit) — at least 1 MB). This happens for:

  • legacy zstd v0.5 frames (ZSTD_getFrameContentSize reports unknown), and
  • streaming frames compressed without a pledged source size.

When a caller passes a smaller-but-adequate buffer, Decompress discards it (cap(dst) < bound) and allocates the full bound. So every decode of such a frame allocates at least 1 MB regardless of the real payload size — a significant per-call heap/GC cost for callers decoding many small unknown-size payloads with a pooled/hinted buffer.

Fix

decompressSizeHint now also returns foundHint — whether the decompressed size was actually read from the frame header.

Decompress and ctx.Decompress select the destination buffer with a single switch, make one DecompressInto, and fall back to the streaming reader only if that buffer is too small:

  • cap(dst) >= hint — reuse the caller buffer (unchanged from before).
  • !foundHint && cap(dst) > 0 — size unknown, so reuse the caller buffer as-is rather than allocating the pessimistic bound.
  • otherwise — allocate hint (the exact size when known; the bound when unknown and no caller buffer was supplied).

Consequences:

  • Known-size frames (the common case where the same zstd version compressed and decompressed): behavior is unchanged, and a caller passing a too-small buffer does not pay for a failed DecompressInto.
  • Unknown-size frames with an adequate caller buffer: no more ≥1 MB allocation — the caller buffer is used.
  • Unknown-size frames with a too-small (or no) caller buffer: fall back to the streaming reader instead of allocating the ≥1 MB bound.

The same change is applied to ctx.Decompress; BulkProcessor.Decompress is adapted to the new signature with its behavior preserved.

Regression tests cover caller-buffer reuse (unknown- and known-size), the too-small fallback, nil dst, and the foundHint signal — run as subtests against both Decompress and ctx.Decompress.

🤖 Generated with Claude Code

@KrisKennawayDD
KrisKennawayDD force-pushed the kris.kennaway/decompress-reuse-caller-buffer branch from c4263b9 to 06c5aaa Compare July 24, 2026 16:46
@KrisKennawayDD KrisKennawayDD changed the title Decompress: reuse the caller-supplied buffer before sizing from the frame Decompress: reuse the caller-supplied buffer for unknown-size frames Jul 24, 2026
@KrisKennawayDD
KrisKennawayDD force-pushed the kris.kennaway/decompress-reuse-caller-buffer branch 3 times, most recently from df06769 to 7406064 Compare July 24, 2026 17:13
@KrisKennawayDD
KrisKennawayDD marked this pull request as ready for review July 24, 2026 17:17
Comment thread zstd.go Outdated
Comment thread zstd.go Outdated
Comment thread zstd_bulk.go
}

contentSize := decompressSizeHint(src)
contentSize, _ := decompressSizeHint(src)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should make it the same logic here to be consistent ?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a comment about why this diverges

Comment thread zstd_reuse_buffer_test.go Outdated
Comment thread zstd_reuse_buffer_test.go Outdated
Comment thread zstd_reuse_buffer_test.go Outdated
Comment thread zstd_reuse_buffer_test.go Outdated
Comment thread zstd_reuse_buffer_test.go Outdated
Comment thread zstd_reuse_buffer_test.go Outdated
Decompress sized its destination from decompressSizeHint(src), which reads the
frame's content-size field. Frames that do not carry that field make the hint
fall back to a pessimistic upper bound (>= decompressSizeBufferLimit, i.e. 1 MB):
this is the case for legacy zstd v0.5 frames and for streaming frames compressed
without a pledged source size. When the caller passed a smaller-but-adequate
buffer, Decompress discarded it and allocated that bound, so every such decode
allocated at least 1 MB regardless of the real payload size.

decompressSizeHint now also reports whether the size was read from the frame
(foundHint). When it was not, Decompress and ctx.Decompress try the
caller-supplied buffer first via DecompressInto -- which reports a too-small
buffer without writing past it -- before falling back to the hint-sized
allocation and then the stream API. When the size is known (the common case
where the same zstd version compressed and decompressed) the original path is
unchanged, so callers passing a too-small buffer do not pay for a failed attempt.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@KrisKennawayDD
KrisKennawayDD force-pushed the kris.kennaway/decompress-reuse-caller-buffer branch from 539664d to ddc2ccc Compare July 24, 2026 17:44
@KrisKennawayDD
KrisKennawayDD requested a review from Viq111 July 24, 2026 17:46

@Viq111 Viq111 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


(I've disabled circleCI for future runs / PR since we moved to github actions)

@pr-shepherd-6ad11f

Copy link
Copy Markdown

PR Shepherd is now watching this PR: fixing basic CI failures, rebasing when it falls behind, and re-queueing after transient merge-queue failures. To disable it, add the pr-shepherd:ignore label.

If you have any questions, reach the team in #ai-devx-flow.

@KrisKennawayDD
KrisKennawayDD merged commit 4d120d2 into 1.x Jul 24, 2026
11 of 13 checks passed
@KrisKennawayDD
KrisKennawayDD deleted the kris.kennaway/decompress-reuse-caller-buffer branch July 24, 2026 18:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants