Skip to content

tracking: implement native Writer::copy_from for additional services #8134

Description

@Xuanwo

Goal

Track native service implementations of Writer::copy_from after the portable API, streaming fallback, and S3 UploadPartCopy implementation landed in #8140.

Writer::copy_from is available on every service with read and write support. A service sets Capability::write_can_copy_from only when its writer can insert every supported OpRead input into the existing destination transaction while preserving range, version, conditions, ordering, completion, and abort semantics. Otherwise OpenDAL keeps using the bounded streaming fallback.

Completed

High-confidence multipart candidates

These services already use oio::MultipartWriter, and their storage APIs can copy a bounded source range into an existing multipart or large-file transaction.

  • TOS — Reuse the existing TosUploadPartCopyRequest and TosCopier implementation from MultipartWrite::copy_part. Extend the request to preserve source version and all four advertised read conditions before enabling write_can_copy_from. TOS documents versioned UploadPartCopy and the copy-source condition headers.
  • OSS — Add UploadPartCopy to OssWriter. OSS supports bounded source ranges, versionId, and all four copy-source conditions. The existing multipart writer already owns initiation, ordered completion, and abort. UploadPartCopy API
  • COS — Add Upload Part Copy to CosWriter, including source version and all four advertised read conditions. The existing multipart writer can reuse the same upload ID and part sequence for local and copied inputs. Upload Part Copy API
  • OBS — Add Copy Part to ObsWriter, including all four advertised read conditions. OBS supports source ranges and conditional copy headers inside a multipart upload. Copying Parts API
  • Backblaze B2 — Resolve the source path to a file ID and call b2_copy_part with the writer's large-file ID, part number, and range. Update part checksum handling so uploaded and copied parts produce the SHA-1 list expected by b2_finish_large_file. B2 currently advertises no conditional or versioned reads, so no additional OpRead lowering is required. b2_copy_part
  • Swift — Server-side copy each source range into the SLO segment path with PUT + X-Copy-From + Range, then reuse the existing SLO manifest completion and cleanup. Verify the advertised source conditions against an actual Swift deployment before enabling the capability. Object copy API

Block-writer prerequisite

  • Extend oio::BlockWrite and oio::BlockWriter with ordered native copy blocks under the same concurrency, completion, and abort state used by local blocks.
  • Azure Blob — Implement PutBlockFromURL through that block-writer path. Reuse the existing block copier request, preserve blob version and all four source conditions, and commit copied and uploaded block IDs through the writer's existing Put Block List transaction. Put Block From URL API

Investigation

  • GCS — Test whether the actual XML API accepts x-goog-copy-source together with uploadId and partNumber. The current multipart documentation describes uploaded data parts but does not document UploadPartCopy, and it states that multipart preconditions are unsupported. Do not enable write_can_copy_from based only on S3 compatibility. XML API multipart uploads
  • Azure FilesPut Range From URL can copy a source range, but the current service uses OneShotWriter/AppendWriter, and file creation requires the destination size before range writes. Define a writer transaction that satisfies online scheduling and abort semantics before treating it as an implementation candidate. Put Range From URL API

Services that only expose whole-object copy, rename, rewrite, or compose remain on the portable streaming path unless their API can place arbitrary bounded ranges into the same open writer transaction. A whole-object copy() implementation alone is not sufficient.

Acceptance checklist for every service PR

  • Reproduce and verify the native operation against the actual service before changing service code or capability advertisement.
  • Preserve every OpRead option advertised by the service, including source version and conditions; otherwise leave the capability false.
  • Keep local uploads and native copied ranges in one ordered writer transaction and one chunk/concurrency budget.
  • Preserve arbitrary write/copy_from interleaving, multipart minimum sizes, and the 5 GiB public per-call limit.
  • Treat every native execution error as terminal for the writer while keeping abort available; do not retry through streaming fallback after mutation may have occurred.
  • Verify close, abort, range boundaries, condition failures, versions where supported, and a mixed local/native byte sequence.
  • Disable the capability for incompatible write configurations such as checksums or encryption modes that native copied parts cannot satisfy.
  • Document service-specific permissions, region/account constraints, encryption limitations, and observable fallback cases.

References

AI assistance

This issue update was drafted with assistance from OpenAI Codex. The candidate list is based on the merged RFC, current OpenDAL service implementations, and linked provider documentation. Every service task still requires actual-service verification under the repository's contribution policy.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions