Skip to content

feat(aws): add Lightsail CloudFormation template#142

Draft
defangdevs wants to merge 2 commits into
masterfrom
feat/lightsail-template
Draft

feat(aws): add Lightsail CloudFormation template#142
defangdevs wants to merge 2 commits into
masterfrom
feat/lightsail-template

Conversation

@defangdevs

Copy link
Copy Markdown
Owner

What

Adds aws/lightsail-template.yaml — a separate template (not a toggle on the EC2 one) that runs the same services.agent-box on AWS Lightsail, priced as one flat monthly bundle. Motivated by a cost comparison: Lightsail folds compute + SSD + a static IPv4 + a multi-TB transfer allowance into a single price, with no separate EBS/public-IPv4 line items.

EC2 t4g.small Lightsail small_3_0
vCPU / RAM 2 / 2 GiB 2 / 2 GiB
Disk 30 GiB gp3 (billed separately) 60 GiB SSD (in bundle)
Public IPv4 ~$3.60/mo (separate) included
Transfer 100 GB free, then $0.09/GB multi-TB included
Price $13.4/mo on-demand-equiv ($12.6 measured on Spot) $12.0/mo flat

How it works (nixos-infect)

Lightsail has no NixOS blueprint and AWS::Lightsail::Instance can't boot a custom image, so the box launches the Ubuntu 24.04 blueprint and converts itself to NixOS in-place with nixos-infect (pinned by commit).

  • The UserData pre-writes /etc/nixos/configuration.nix before running nixos-infect; the script's makeConf() early-returns when that file exists, so it never clobbers our config.
  • PROVIDER=lightsail is first-class in nixos-infect: it relabels root to nixos and imports virtualisation/amazon-image.nixthe same module the EC2 template uses. So the platform layer is the proven one; only the delivery is new.
  • The baked config is the EC2 config minus Spot/NAT64, plus the Lightsail bits (amazon-image.nix import, GRUB on /dev/nvme0n1). selfUpdate, the web-password-hash activation script, the first-boot WaitCondition signal, disk-GC, and the amazon-init disable all carry over unchanged.

Differences from the EC2 template

  • No VPC/subnet/IGW/SG/EIP/IAM — Lightsail manages networking; firewall is the Networking.Ports block (443 always, 22 when DebugSsh=true).
  • IPv4-native → no PublicIpv4/Nat64; no Spot → no UseSpot; fixed bundle disk → no RootVolumeSize.
  • A static IP is always attached (free while attached, stable across stop/start). It attaches after instance creation, so to avoid a CFN dependency cycle the box discovers its settled public IPv4 at runtime for the sslip.io hostname; Outputs report the same address via GetAtt StaticIp.IpAddress.
  • Debug access is root SSH with the Lightsail default key (DebugSsh=true opens 22), not SSM. Infect logs at /var/log/agent-box-infect.log.

Validation

  • cfn-lint passes on both templates (aws-ci.yml now lints both).
  • ✅ The generated configuration.nix parses as valid Nix (rendered the Fn::Sub, ${!x}${x}, substituted params + sed placeholders, nix-instantiate --parse).
  • ⚠️ The end-to-end nixos-infect bootstrap has NOT been exercised by a live deploy yet — hence draft. Treat a first real launch as the acceptance test.

Known risk to validate on first launch

The runtime IP-settle poll (sleep 150 + stability check) assumes the static IP attaches within the sleep window. The margin is large (attach ≈ 1–2 min; the NixOS closure build + reboot takes 10+ min), but a live run should confirm the baked sslip.io host matches the attached static IP.

Follow-ups (deferred to keep this diff focused)

  • A Lightsail leg in deploy-test.yml (create → assert WebURL reachable over IPv4 — runners are IPv4-only and Lightsail is IPv4-native, so this can smoke the live URL — → tear down).
  • S3 publish + a Launch button. publish-template.yml currently publishes only template.yaml and scopes the bucket policy to that one object; a second template needs both objects covered by one policy (they must not fight over PutBucketPolicy).

Deploy instructions (CLI) and the full rationale are in aws/README.md → "Lightsail variant".

🤖 Generated with Claude Code

Add `aws/lightsail-template.yaml`, a separate (not a toggle) template that
runs the same services.agent-box on AWS Lightsail for one flat monthly
bundle price — folding compute, SSD, static IPv4, and a multi-TB transfer
allowance into a single line item (no separate EBS/IPv4 charges).

Lightsail has no NixOS blueprint and CloudFormation can't boot a custom
image, so the box launches the Ubuntu 24.04 blueprint and converts itself to
NixOS in-place with nixos-infect (pinned by commit). PROVIDER=lightsail is
first-class in nixos-infect and imports the same virtualisation/amazon-image.nix
the EC2 template uses, so the platform layer is the proven one; the baked
configuration.nix is the EC2 config minus Spot/NAT64 plus the Lightsail bits
(amazon-image import + GRUB on nvme). selfUpdate, the web-password-hash
activation script, the first-boot WaitCondition signal, and the amazon-init
disable all carry over.

Design notes:
- No VPC/subnet/IGW/SG/EIP/IAM — Lightsail manages networking; firewall is
  the Networking.Ports block (443 always, 22 when DebugSsh=true).
- A static IP is always attached (free while attached, stable across
  stop/start). It attaches after instance creation, so to avoid a dependency
  cycle the box discovers its settled public IPv4 at runtime for the
  sslip.io hostname; Outputs report the same address via GetAtt.
- Debug access is root SSH with the Lightsail default key (no SSM); infect
  logs at /var/log/agent-box-infect.log.

Validation: cfn-lint passes and the generated configuration.nix parses as
valid Nix; aws-ci.yml now lints both templates. The end-to-end nixos-infect
bootstrap still needs a live deploy-test (documented as a follow-up, along
with S3 publish + a Launch button), so this ships as a draft.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Sw7j9FYnbChZJbYArxbhuo

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add at least one E2E test

Addresses the review ask to include at least one end-to-end test for the
Lightsail template rather than deferring it. aws-ci.yml only lints; the
existing deploy-test.yml deploys the EC2 template only.

Adds a standalone deploy-test-lightsail.yml (create -> wait -> assert ->
smoke -> destroy). Lightsail is IPv4-native and GH runners are IPv4-only,
so this single leg runs the full browser-terminal smoke suite (auth 200,
unauth 401, tabbed workspace, ttyd HTML, WebSocket upgrade, live-tmux via
scripts/ws_smoke.py). Reuses the OIDC + module-pin conventions; drops the
EC2-only bits (ipv4/ipv6 matrix, Spot, SSM, serial console) since the
FirstBootDone WaitCondition already gates CREATE_COMPLETE on the first
nixos-rebuild switch. DebugSsh=false keeps 22 closed on the public box.

BundleId/WebPassword are passed via the step env so the shell never
expands the "$12" price annotation as a positional parameter (which would
fail AllowedValues validation). Also adds the new workflow to aws-ci.yml
trigger paths for symmetry with deploy-test.yml.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wx9PVLmDbzS35JCN5N85i6
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants