feat(storage): implement 2 MiB coalescing buffer for appendable upload - #6571
feat(storage): implement 2 MiB coalescing buffer for appendable upload#6571vsharonlynn wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces a CoalescingBuffer to batch client append operations into standard 2 MiB chunks, matching the GCS protocol limit, and integrates it into the AppendableObjectWriterTransport along with updated unit tests. The review feedback suggests optimizing the buffer's ingestion logic by using direct slicing to avoid atomic reference counting overhead, and refactoring the flush method to remove an unnecessary else block in compliance with the repository style guide.
| let prefix = chunk.slice(..needed); | ||
| self.buffer.extend_from_slice(&prefix); |
There was a problem hiding this comment.
Slicing Bytes via chunk.slice(..needed) creates a new Bytes handle, which incurs atomic reference counting overhead. Since extend_from_slice only requires a byte slice (&[u8]), we can pass &chunk[..needed] directly to avoid this overhead. Please document this safety guarantee with a comment.
| let prefix = chunk.slice(..needed); | |
| self.buffer.extend_from_slice(&prefix); | |
| // SAFETY: `needed` is guaranteed to be within the bounds of `chunk`. | |
| self.buffer.extend_from_slice(&chunk[..needed]); |
References
- When slicing a collection directly in Rust, document the safety guarantee with a comment.
| if self.buffer.is_empty() { | ||
| None | ||
| } else { | ||
| let residual = self.buffer.split().freeze(); | ||
| self.buffer.reserve(COALESCING_CHUNK_SIZE); | ||
| Some(residual) | ||
| } |
There was a problem hiding this comment.
According to the repository style guide, we should avoid unnecessary else blocks to keep the main logic flow linear and reduce indentation.
if self.buffer.is_empty() {
return None;
}
let residual = self.buffer.split().freeze();
self.buffer.reserve(COALESCING_CHUNK_SIZE);
Some(residual)References
- Avoid unnecessary else blocks to reduce indentation and keep the main logic flow linear. (link)
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #6571 +/- ##
==========================================
- Coverage 96.52% 96.51% -0.01%
==========================================
Files 304 305 +1
Lines 87969 88123 +154
==========================================
+ Hits 84908 85054 +146
- Misses 3061 3069 +8 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
No description provided.