You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Track native service implementations of Writer::copy_from after the portable API, streaming fallback, and S3 UploadPartCopy implementation landed in #8140.
Writer::copy_from is available on every service with read and write support. A service sets Capability::write_can_copy_from only when its writer can insert every supported OpRead input into the existing destination transaction while preserving range, version, conditions, ordering, completion, and abort semantics. Otherwise OpenDAL keeps using the bounded streaming fallback.
Validate the S3-compatible provider presets that share services/s3; add provider-specific capability gating if an actual provider does not support the required UploadPartCopy contract.
High-confidence multipart candidates
These services already use oio::MultipartWriter, and their storage APIs can copy a bounded source range into an existing multipart or large-file transaction.
TOS — Reuse the existing TosUploadPartCopyRequest and TosCopier implementation from MultipartWrite::copy_part. Extend the request to preserve source version and all four advertised read conditions before enabling write_can_copy_from. TOS documents versioned UploadPartCopy and the copy-source condition headers.
OSS — Add UploadPartCopy to OssWriter. OSS supports bounded source ranges, versionId, and all four copy-source conditions. The existing multipart writer already owns initiation, ordered completion, and abort. UploadPartCopy API
COS — Add Upload Part Copy to CosWriter, including source version and all four advertised read conditions. The existing multipart writer can reuse the same upload ID and part sequence for local and copied inputs. Upload Part Copy API
OBS — Add Copy Part to ObsWriter, including all four advertised read conditions. OBS supports source ranges and conditional copy headers inside a multipart upload. Copying Parts API
Backblaze B2 — Resolve the source path to a file ID and call b2_copy_part with the writer's large-file ID, part number, and range. Update part checksum handling so uploaded and copied parts produce the SHA-1 list expected by b2_finish_large_file. B2 currently advertises no conditional or versioned reads, so no additional OpRead lowering is required. b2_copy_part
Swift — Server-side copy each source range into the SLO segment path with PUT + X-Copy-From + Range, then reuse the existing SLO manifest completion and cleanup. Verify the advertised source conditions against an actual Swift deployment before enabling the capability. Object copy API
Block-writer prerequisite
Extend oio::BlockWrite and oio::BlockWriter with ordered native copy blocks under the same concurrency, completion, and abort state used by local blocks.
Azure Blob — Implement PutBlockFromURL through that block-writer path. Reuse the existing block copier request, preserve blob version and all four source conditions, and commit copied and uploaded block IDs through the writer's existing Put Block List transaction. Put Block From URL API
Investigation
GCS — Test whether the actual XML API accepts x-goog-copy-source together with uploadId and partNumber. The current multipart documentation describes uploaded data parts but does not document UploadPartCopy, and it states that multipart preconditions are unsupported. Do not enable write_can_copy_from based only on S3 compatibility. XML API multipart uploads
Azure Files — Put Range From URL can copy a source range, but the current service uses OneShotWriter/AppendWriter, and file creation requires the destination size before range writes. Define a writer transaction that satisfies online scheduling and abort semantics before treating it as an implementation candidate. Put Range From URL API
Services that only expose whole-object copy, rename, rewrite, or compose remain on the portable streaming path unless their API can place arbitrary bounded ranges into the same open writer transaction. A whole-object copy() implementation alone is not sufficient.
Acceptance checklist for every service PR
Reproduce and verify the native operation against the actual service before changing service code or capability advertisement.
Preserve every OpRead option advertised by the service, including source version and conditions; otherwise leave the capability false.
Keep local uploads and native copied ranges in one ordered writer transaction and one chunk/concurrency budget.
Preserve arbitrary write/copy_from interleaving, multipart minimum sizes, and the 5 GiB public per-call limit.
Treat every native execution error as terminal for the writer while keeping abort available; do not retry through streaming fallback after mutation may have occurred.
Verify close, abort, range boundaries, condition failures, versions where supported, and a mixed local/native byte sequence.
Disable the capability for incompatible write configurations such as checksums or encryption modes that native copied parts cannot satisfy.
This issue update was drafted with assistance from OpenAI Codex. The candidate list is based on the merged RFC, current OpenDAL service implementations, and linked provider documentation. Every service task still requires actual-service verification under the repository's contribution policy.
Goal
Track native service implementations of
Writer::copy_fromafter the portable API, streaming fallback, and S3UploadPartCopyimplementation landed in #8140.Writer::copy_fromis available on every service with read and write support. A service setsCapability::write_can_copy_fromonly when its writer can insert every supportedOpReadinput into the existing destination transaction while preserving range, version, conditions, ordering, completion, and abort semantics. Otherwise OpenDAL keeps using the bounded streaming fallback.Completed
UploadPartCopyin the writer's multipart upload transaction in feat(services/s3): support Writer::copy_from #8140.services/s3; add provider-specific capability gating if an actual provider does not support the requiredUploadPartCopycontract.High-confidence multipart candidates
These services already use
oio::MultipartWriter, and their storage APIs can copy a bounded source range into an existing multipart or large-file transaction.TosUploadPartCopyRequestandTosCopierimplementation fromMultipartWrite::copy_part. Extend the request to preserve source version and all four advertised read conditions before enablingwrite_can_copy_from. TOS documents versionedUploadPartCopyand the copy-source condition headers.UploadPartCopytoOssWriter. OSS supports bounded source ranges,versionId, and all four copy-source conditions. The existing multipart writer already owns initiation, ordered completion, and abort. UploadPartCopy APICosWriter, including source version and all four advertised read conditions. The existing multipart writer can reuse the same upload ID and part sequence for local and copied inputs. Upload Part Copy APIObsWriter, including all four advertised read conditions. OBS supports source ranges and conditional copy headers inside a multipart upload. Copying Parts APIb2_copy_partwith the writer's large-file ID, part number, and range. Update part checksum handling so uploaded and copied parts produce the SHA-1 list expected byb2_finish_large_file. B2 currently advertises no conditional or versioned reads, so no additionalOpReadlowering is required.b2_copy_partPUT+X-Copy-From+Range, then reuse the existing SLO manifest completion and cleanup. Verify the advertised source conditions against an actual Swift deployment before enabling the capability. Object copy APIBlock-writer prerequisite
oio::BlockWriteandoio::BlockWriterwith ordered native copy blocks under the same concurrency, completion, and abort state used by local blocks.PutBlockFromURLthrough that block-writer path. Reuse the existing block copier request, preserve blob version and all four source conditions, and commit copied and uploaded block IDs through the writer's existingPut Block Listtransaction. Put Block From URL APIInvestigation
x-goog-copy-sourcetogether withuploadIdandpartNumber. The current multipart documentation describes uploaded data parts but does not document UploadPartCopy, and it states that multipart preconditions are unsupported. Do not enablewrite_can_copy_frombased only on S3 compatibility. XML API multipart uploadsPut Range From URLcan copy a source range, but the current service usesOneShotWriter/AppendWriter, and file creation requires the destination size before range writes. Define a writer transaction that satisfies online scheduling and abort semantics before treating it as an implementation candidate. Put Range From URL APIServices that only expose whole-object copy, rename, rewrite, or compose remain on the portable streaming path unless their API can place arbitrary bounded ranges into the same open writer transaction. A whole-object
copy()implementation alone is not sufficient.Acceptance checklist for every service PR
OpReadoption advertised by the service, including source version and conditions; otherwise leave the capability false.write/copy_frominterleaving, multipart minimum sizes, and the 5 GiB public per-call limit.abortavailable; do not retry through streaming fallback after mutation may have occurred.close,abort, range boundaries, condition failures, versions where supported, and a mixed local/native byte sequence.References
AI assistance
This issue update was drafted with assistance from OpenAI Codex. The candidate list is based on the merged RFC, current OpenDAL service implementations, and linked provider documentation. Every service task still requires actual-service verification under the repository's contribution policy.