Register the S3 request signer on the session - #3783
Conversation
S3FileSystem creates the client that issues requests lazily inside the running event loop, so a handler attached to the eagerly constructed fs.s3 is never inherited by it. With signature_version set to UNSIGNED, requests then reach S3 with no Authorization header and are rejected. Registering on the session means every client it creates carries the signer. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Resolves a conflict in _s3(): main added s3_additional_kwargs handling for server-side encryption while this branch replaced the post-construction signer registration with registration on an AioSession. Both apply — the additional kwargs are added to s3_fs_kwargs before the session is attached.
|
This also fixes #3896, which is a second, independent reason the Your change closes it because it drops the
Both are in They are on #3896 if useful. Happy to raise them against this PR instead so the fix and its regression cover land together — your call, and no action needed from you either way. |
Closes #3625
Rationale for this change
FsspecFileIOremote request signing (s3.signer=S3V4RestSigner) silently stops signing oncurrent
aiobotocore/s3fs._s3()builds the filesystem first and then attaches the signer tofs.s3:Touching
fs.s3forces one eager client, but the client that actually issues requests is createdlazily inside the running event loop by
S3FileSystem.set_session, and it does not inherithandlers registered on that earlier client. Since
config_kwargs["signature_version"]is set toUNSIGNED, requests then go out with noAuthorizationheader and S3 rejects them withInvalidRequest: The authorization mechanism you have provided is not supported.This change registers the signer on an
aiobotocoresession and passes that session toS3FileSystem, so every client the session creates carries the handler.register_lastand theunique_idare kept, so ordering relative to the stockbefore-sign.s3handlers is unchanged.The
unregistercall is dropped because a freshly created session has nothing registered underthat id.
Credit for the diagnosis and the proposed approach goes to the issue reporter.
Are these changes tested?
Yes.
test_s3v4_rest_signer_registered_on_sessionis added totests/io/test_fsspec.py. Itbuilds the
FileIOwith a signer configured, takes the session handed toS3FileSystem, emitsbefore-sign.s3on it, and asserts the REST signer ran — the request URL is rewritten and anAuthorizationheader is set. Onmainit fails withKeyError: 'session', since no session ispassed today.
The mechanism was also confirmed directly against
s3fs2026.4.0 andaiobotocore3.8.0 bydriving
set_session()and inspecting the handlers on the client that ends up issuing requests:make lint— passes, including mypymake test— 3804 passed, 3 skippedNot verified end to end against a live REST catalog with remote signing and real S3, as that
needs credentials and an S3 endpoint that enforces SigV4;
motodoes not. The end-to-endconfirmation in the issue report covers that path.
Are there any user-facing changes?
No API change. Remote signing starts working again on current dependency versions, so
FsspecFileIOusers on a modernaiobotocorewill see requests signed rather than rejected.AI assistance
Claude Code was used to investigate the issue, write the change and the test, and run the
verification described above. The full diff was reviewed before submission.