[Bug] Load the proper AWS credential for glue/dynamodb catalog

[kevinjqliu]

Apache Iceberg version

None

Please describe the bug 🐞

s3.access-key-id and s3.secret-access-key are the pyiceberg way to pass AWS credentials as properties to the catalog.
However, in the glue and dynamodb catalog, aws_access_key_id and aws_secret_access_key is expected
https://github.com/search?q=repo%3Aapache%2Ficeberg-python+aws_secret_access_key+path%3A.py+-path%3Atests&type=code

Also see #515 (comment)

We should standardize on a using s3.access-key-id and s3.secret-access-key for glue/dynamodb catalog. And perhaps also fall back to aws_access_key_id/aws_secret_access_key

[jayceslesar]

seems like a misnomer for dynamo/glue to expect something called s3.some-aws-access-key?

[kevinjqliu]

Another consideration is that there can be 2 separate s3 credentials. One to pass to glue/dynamo. Another for the catalog itself.

[jayceslesar]

maybe its fine to just break compatibility leading up to 1.0.0? What does the java iceberg do when looking for s3 credentials?

[HonahX]

Hi @kevinjqliu @jayceslesar. Thanks for the issue and valuable discussion. I would like to give a try with my proposal in #570 (comment). Draft PR: #922. Please let me know what you think! Thanks in advance!

What does the java iceberg do when looking for s3 credentials?

Java implementation supports s3.access-key-id and other s3.* properties to config S3FileIO, but it does not support config glue/dynamodb by explicitly passing the credentials. Instead, it allows user to specify a custom credential provider implementation.