v4: Deploys fail with "Rate exceeded" after `UPDATE_COMPLETE`: post-deploy DescribeStacks and prune calls lack the throttling retry that monitor-stack has

[timpitman-cb]

Summary

First up: we're really glad to see osls under active development with v4! We migrated a ~40-service monorepo from v3, and the upgrade was smooth overall; this is one of the few rough edges we've hit, so thanks for the work that's gone into the 4.x line 🙏

When many services deploy concurrently (our CI deploys ~40 stacks in parallel), CloudFormation and Lambda API throttling is sustained for the duration of the deployment. Under v4, stack monitoring survives this because monitor-stack.js wraps its polling in retryOnThrottlingError (4 retries, 5s apart). But the calls that run immediately after the stack reaches UPDATE_COMPLETE are not wrapped:

• lib/plugins/aws/info/get-stack-info.js — DescribeStacksCommand (and the API Gateway lookups around it)
• lib/plugins/aws/prune.js — ListVersionsByFunctionCommand, ListAliasesCommand, DeleteFunctionCommand, layer equivalents

Those get only the SDK's built-in standard retries (maxAttempts 5 by default, roughly 2–3 seconds of total backoff), which sustained throttling easily outlives. The result:

UPDATE_COMPLETE_CLEANUP_IN_PROGRESS - AWS::CloudFormation::Stack - my-service-staging
UPDATE_COMPLETE - AWS::CloudFormation::Stack - my-service-staging

× Stack my-service-staging failed to deploy (64s)
Environment: linux, node 24.18.0, osls 4.1.0 (local)

Error:
Rate exceeded

The stack has actually deployed successfully; the CLI exits 1 during the cosmetic/cleanup phase. CI reports a red deploy for a healthy stack, and retrying re-runs the whole deploy (which then passes with a no-op changeset), so it presents as flakiness.

Regression vs v3

v3 routed every AWS call through lib/aws/request.js, which retried any retryable error (throttling included) up to 4 times with 4–7s backoff on top of the SDK's own retries, and capped each process at 2 concurrent AWS requests (promiseLimit(2)). The same deploy concurrency produced the same throttling under v3 (we measured slightly more throttle events on v3 builds), but it was always absorbed silently. v4 restored that patient retry only at three call sites (monitor-stack.js, wait-for-change-set-creation.js, upload-s3-object.js); the post-deploy info and prune paths were missed.

The comment in lib/aws/retry.js already states the rationale that applies here:

On top of the SDK's own per-request retries: sustained throttling outlives the SDK retry budget during polling and bulk uploads...

Potential fix

Wrap the AWS calls in get-stack-info.js and prune.js in retryOnThrottlingError, as monitor-stack.js already does. Prune is the more important of the two: it issues several Lambda control-plane calls per function in a loop, right at the moment every sibling deploy is doing the same.

Reproduction

Deploy N services concurrently against one account (N large enough to sustain CloudFormation/Lambda throttling; ~40 for us). Some subset of runs fails with Rate exceeded immediately after UPDATE_COMPLETE, with provider.pruneFunctionVersions enabled making it more likely.

Workaround we're using

SLS_AWS_REQUEST_MAX_RETRIES=10 in the deploy environment, which raises maxAttempts on every SDK client and widens the backoff budget to ~40–85s.