Pin AWS SDK to 2.44.4 to match the copy bundled in iceberg-aws-bundle

[velo]

What

Pins software.amazon.awssdk to 2.44.4 via the AWS SDK BOM, matching the SDK version bundled inside iceberg-aws-bundle:1.11.0.

Before this change the runner image shipped two unrelocated copies of AWS SDK v2 at different versions on the Flink system classpath:

jar in /opt/flink/lib	AWS SDK	brings
iceberg-aws-bundle-1.11.0.jar	2.44.4	auth, sts, aws-query-protocol, WebIdentityTokenFileCredentialsProvider, …
sql-runner.uber.jar	2.44.12	same classes + software.amazon.msk.auth.iam.* + kafka-clients

2.44.12 arrives transitively through aws-msk-iam-auth:2.3.7:

software.amazon.msk:aws-msk-iam-auth:jar:2.3.7:compile
 +- software.amazon.awssdk:auth:jar:2.44.12:runtime
 +- software.amazon.awssdk:sts:jar:2.44.12:runtime
 \- ...

Iceberg's 2.44.4 is not a Maven edge — it is baked into the bundle jar — so Maven never sees the conflict and dependency convergence rules can't catch it.

Classpath order on a running TaskManager (from /proc/1/cmdline):

/opt/flink/lib/iceberg-aws-bundle-1.11.0.jar     <- wins
...
/opt/flink/lib/sql-runner.uber.jar               <- its 2.44.12 is dead weight

Class loading is first-wins, so 2.44.4 is what actually loads, and the MSK IAM auth code — compiled against 2.44.12 — runs against 2.44.4 classes. Pinning makes the two copies semantically identical, so which one wins stops mattering. They are not byte-identical: the shade plugin renumbers the constant pool when it repackages, which flips some ldc to ldc_w and shifts every subsequent bytecode offset. Verified on the PR image by normalising pool indices out of javap -c output — same instructions, same operands, same targets.

Why now — the bug under investigation

On dev-signature, TaskManagers running 0.10.5-flink-2.2 produced this during task teardown:

WARN  software.amazon.msk.auth.iam.internals.MSKCredentialProvider
      Exception loading credentials. Retry Attempts: 1
SdkClientException: Unable to load credentials from any of the providers in the chain ...
  WebIdentityTokenFileCredentialsProvider(): Unable to unmarshall response
    (Trying to access closed classloader...). Response Code: 200

The STS call returned 200 — IRSA is fine, this is not a permissions problem. The SDK simply could not unmarshal the response, and the failing code path is exactly aws-query-protocol / AwsXmlErrorProtocolUnmarshaller, one of the classes duplicated across the two jars.

Immediately before it, the same TaskManager logged:

ERROR org.apache.kafka.clients.producer.internals.Sender
      [Producer clientId=producer-34] Error in kafka producer I/O thread while aborting transaction when during closing:
IllegalStateException: Transactional method invoked on a non-transactional producer.
  at TransactionManager.ensureTransactional(TransactionManager.java:1149)
  at TransactionManager.beginAbort(TransactionManager.java:362)
  at Sender.run(Sender.java:273)

That second one is a separate, pre-existing kafka-clients 4.2.0 issue: Sender.run()'s force-close path calls transactionManager.beginAbort() guarded only on transactionManager != null. An idempotent-but-non-transactional producer (enable.idempotence=true, no transactional.id) still has a non-null TransactionManager, so ensureTransactional() throws. It fires only on close(Duration.ZERO), which Flink issues when cancelling a task. This PR does not address that — tracked separately.

Both traces are teardown exhaust rather than the root cause, and the pod that produced them was later OOMKilled and evicted for exceeding its 20Gi flink-local-data EmptyDir limit. The duplicate-SDK skew is real regardless, and it is the cheapest of the three to remove — hence trying it first.

Alternatives considered

Relocating software.amazon.awssdk in the uber jar's shade config is the more correct fix and would make the two copies genuinely independent. It is also a much larger change with its own risk around the SDK's reflective/ServiceLoader lookups. Trying the version pin first; if the skew turns out not to be the problem, shading is the next step.

Note that org.apache.kafka.* and software.amazon.msk.* are also unrelocated in the uber jar and carry the same collision risk — out of scope here.

Verification

mvn dependency:tree -Dincludes='software.amazon.awssdk:*' now reports a single version:

software.amazon.awssdk:auth:jar:2.44.4:runtime
software.amazon.awssdk:sts:jar:2.44.4:runtime
software.amazon.awssdk:aws-query-protocol:jar:2.44.4:runtime
software.amazon.awssdk:apache-client:jar:2.44.4:runtime

mvn clean test passes across all 23 modules.

The pin carries a comment with the command to re-check the bundled version after any iceberg.version bump, since the two must move together.

Also included: a comment on the S3A core-site.xml workaround

Second commit, docs only — no behaviour change.

While investigating, the core-site.xml added in #290 came up as a suspect. It is not related to the MSK failure: it configures fs.s3a.aws.credentials.provider with com.amazonaws.auth.DefaultAWSCredentialsProviderChain, which is AWS SDK v1 and only affects the S3A filesystem. MSK IAM auth uses the separate SDK v2 (software.amazon.awssdk) chain.

What did turn up is that the workaround is load-bearing in a non-obvious way: flink-s3-fs-hadoop-2.2.1.jar ships com/amazonaws/** unrelocated, and that is the only reason the unshaded class name in core-site.xml resolves at all. If that jar is ever shaded, the setting silently stops applying and S3A IRSA breaks again with no error pointing here.

Added a comment above the RUN recording that, plus a line explaining that noop.jar exists only to satisfy flink run's required JAR positional argument (from #317). docker build --check passes.