-
Notifications
You must be signed in to change notification settings - Fork 10
Add Blob openStream() so missing-blob-file errors are catchable before the response commits #2138
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -1199,6 +1199,57 @@ describe('Blob test', () => { | |
| env.setProperty(CONFIG_PARAMS.STORAGE_BLOBREADTIMEOUT, undefined); | ||
| } | ||
| }); | ||
| it('openStream() resolves for a healthy disk-backed blob and streams the full content', async () => { | ||
| const payload = randomBytes(20000); | ||
| const store = BlobTest.primaryStore.rootStore; | ||
| const blob = await createBlob(Readable.from(payload), { size: payload.length }); | ||
| await decodeFromDatabase(() => saveBlob(blob).saving, store); | ||
| const streamed = await streamToBytes(await blob.openStream()); | ||
| assert(streamed.equals(payload)); | ||
| // a slice shares the same backing file and must stream just the slice content | ||
| const slicedStream = await blob.slice(300, 400).openStream(); | ||
| assert((await streamToBytes(slicedStream)).equals(payload.subarray(300, 400))); | ||
| }); | ||
| it('openStream() resolves for a small in-memory blob', async () => { | ||
| const payload = randomBytes(100); | ||
| const blob = await createBlob(payload); | ||
| const streamed = await streamToBytes(await blob.openStream()); | ||
| assert(streamed.equals(payload)); | ||
| }); | ||
| it('openStream() rejects with a 404 for a cleanly-missing blob file, before any read', async () => { | ||
| const { blob, filePath } = await makeDiskBackedBlob(); | ||
| unlinkSync(filePath); | ||
| await assert.rejects(blob.openStream(), (error) => { | ||
| assert.strictEqual(error.statusCode, 404); | ||
| assert.strictEqual(error.code, 'ENOENT'); | ||
| return true; | ||
| }); | ||
|
Comment on lines
+1222
to
+1226
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Use await assert.rejects(blob.openStream(), (error) => {
assert.strictEqual(error.statusCode, 404);
assert.strictEqual(error.code, 'ENOENT');
return true;
});References
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Applied in 0e993c6 — all four assertions switched to |
||
| }); | ||
| it('openStream() rejects with a 503 while a write is in flight', async () => { | ||
| const { blob, filePath, store } = await makeDiskBackedBlob(); | ||
| const lockKey = getFileId(blob) + ':blob'; | ||
| assert(store.tryLock(lockKey), 'should be able to take the blob write lock for the test'); | ||
| try { | ||
| unlinkSync(filePath); // file gone while a "writer" still holds the lock | ||
| env.setProperty(CONFIG_PARAMS.STORAGE_BLOBREADTIMEOUT, '150'); | ||
| await assert.rejects(blob.openStream(), (error) => { | ||
| assert.strictEqual(error.statusCode, 503); | ||
| return true; | ||
| }); | ||
| } finally { | ||
| store.unlock(lockKey); | ||
| env.setProperty(CONFIG_PARAMS.STORAGE_BLOBREADTIMEOUT, undefined); | ||
| } | ||
| }); | ||
| it('openStream() rejects with a 500 for a blob truncated to a self-consistent smaller size', async () => { | ||
| const { blob, filePath } = await makeDiskBackedBlob(); | ||
| truncateBlobConsistently(filePath, 256); | ||
| await assert.rejects(blob.openStream(), (error) => { | ||
| assert.strictEqual(error.statusCode, 500); | ||
| assert.match(error.message, /size mismatch/); | ||
| return true; | ||
| }); | ||
| }); | ||
| afterEach(function () { | ||
| setAuditRetention(60000); | ||
| setDeletionDelay(50); // restore shorter, but need to have it happen for the last test | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The
openStreammethod and its innerpullhandler can be optimized to avoidasync/awaitoverhead. In high-performance hot paths like stream chunk pulling, avoidingasync/awaitstate machines and extra promise allocations by returning.then()chains directly improves performance and reduces garbage collection pressure, aligning with the repository's performance-first style guide.References
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Applied in 0e993c6 for the per-chunk
pull()handler, which is the hot part. KeptopenStream()itselfasync: it runs once per open (not per chunk), and theasyncwrapper guarantees a synchronous throw fromstream()/getReader()(e.g. no store for the file path) surfaces as a rejection rather than a sync exception, keepingblob.openStream().catch(...)uniform for callers. (reply generated by AI)