Skip to content

fix(runners): support instance requirements in EC2 Fleet overrides - #5314

Merged
edersonbrilhante merged 1 commit into
github-aws-runners:mainfrom
slavab89:slava-b/fix-fleet-instance-requirements
Sep 8, 2026
Merged

fix(runners): support instance requirements in EC2 Fleet overrides#5314
edersonbrilhante merged 1 commit into
github-aws-runners:mainfrom
slavab89:slava-b/fix-fleet-instance-requirements

Conversation

@slavab89

@slavab89 slavab89 commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Description

Dynamic EC2 labels allow a workflow to request attribute-based instance selection using InstanceRequirements. This is useful when a workload needs a specific CPU and memory shape while allowing EC2 Fleet to choose among several eligible instance families for Spot capacity.

Today, generateFleetOverrides combines every configured static InstanceType with the dynamic InstanceRequirements object. AWS treats these fields as mutually exclusive within one Fleet override, so CreateFleet rejects the request. The dynamic labels parse successfully, but no runner can launch.

This change gives the two selection modes separate request paths:

  • Fixed selection continues to produce one override per subnet and instance type.
  • Attribute-based selection produces one InstanceRequirements-only override per subnet, allowing EC2 Fleet to choose from the eligible families.
  • Configurations containing both InstanceType and InstanceRequirements are rejected before an AWS request is sent.

Existing fixed-instance behavior is unchanged.

Test Plan

  • Added regression coverage proving requirements-only overrides omit InstanceType.
  • Added regression coverage proving conflicting fixed and attribute-based selection sends no CreateFleet request.
  • Ran the focused EC2 runner suite: 71 tests passed.
  • Ran the compute-provider suite: 10 files, 272 tests passed.
  • Ran the compute-provider Prettier and ESLint checks.
  • Ran git diff --check.

Validated the request against AWS using a temporary launch template without runner userdata or an IAM profile. A requirements-only Spot Fleet request for 4 vCPU / 8 GiB across several allowed compute families completed successfully and selected c7i-flex.xlarge with no Fleet errors. The instance and temporary launch template were deleted afterward.

The project typecheck currently reports existing Vitest 4 configuration errors in untouched configuration files. Those unrelated formatting and type changes are intentionally excluded from this PR.

Related Issues

Fixes #5315. Related to #1400 and follows the dynamic EC2 label support introduced in #5003.

@slavab89
slavab89 marked this pull request as ready for review August 28, 2026 09:20
@slavab89
slavab89 requested a review from a team as a code owner August 28, 2026 09:20
Comment thread lambdas/libs/compute-providers/aws/ec2/src/control-plane/runners.ts Outdated
Comment thread lambdas/libs/compute-providers/aws/ec2/src/control-plane/runners.ts Outdated
@slavab89
slavab89 force-pushed the slava-b/fix-fleet-instance-requirements branch from 3fa6f25 to 80fccdb Compare September 8, 2026 09:32
@edersonbrilhante

Copy link
Copy Markdown
Contributor

@slavab89 Once this error is raised, it will pass along to this call const runnerLabelResolution = await computeProvider.resolveLabelsForRunners(messageLabels); in lambdas/functions/control-plane/src/scale-runners/scale-up.ts:205, then it will pass to the parent caller and it will be retried.

Shouldn't the message be discard?

@slavab89
slavab89 force-pushed the slava-b/fix-fleet-instance-requirements branch from 80fccdb to 1c40276 Compare September 8, 2026 13:31
@slavab89

slavab89 commented Sep 8, 2026

Copy link
Copy Markdown
Contributor Author

@edersonbrilhante Good point. Addressed in 1c40276. Invalid runner-label configurations now raise a typed InvalidRunnerLabelsError. The control-plane catches only that error at the per-label-group boundary and skips the group without adding its message IDs to batchItemFailures, so SQS acknowledges and discards those messages. Other label-resolution errors still propagate and retry. I also added a mixed-batch regression test proving an invalid group is discarded while a valid group continues to launch.

@edersonbrilhante edersonbrilhante left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@slavab89 Good work.

@edersonbrilhante
edersonbrilhante merged commit 4f80654 into github-aws-runners:main Sep 8, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Dynamic instance requirements produce invalid EC2 Fleet overrides

2 participants