Skip to content

Fix heavy-build rootfs provisioning - #22

Open
H3xept wants to merge 1 commit into
mv37-org:mainfrom
H3xept:agent/fix-heavy-build-rootfs
Open

Fix heavy-build rootfs provisioning#22
H3xept wants to merge 1 commit into
mv37-org:mainfrom
H3xept:agent/fix-heavy-build-rootfs

Conversation

@H3xept

@H3xept H3xept commented Jul 16, 2026

Copy link
Copy Markdown

Fixes #21.

What

  • add a 64 GiB heavy-build curated image with Docker Engine, Buildx, and Compose
  • make placement image-aware across local and remote nodes and return 503 no_capacity with reason image_unavailable before creating a sandbox record
  • wait for docker info during guest boot and include the daemon log when startup fails
  • make node provisioning boot the image through the real API/Firecracker path and verify Compose, a nested container, DNS, and HTTPS egress
  • document how the curated image is staged

Why

The catalog advertised heavy-build, but node provisioning did not build its rootfs. The scheduler assumed curated images were present, so requests reached Firecracker, failed while staging a missing artifact, and left a failed sandbox record behind a generic HTTP 500.

The control plane now queries each worker's local runtime through the authenticated internal node API. Inventory checks run concurrently and each remote call has a two-second total timeout. A timeout or error makes only that worker ineligible, while nodes that report the requested artifact remain eligible for base, browser, and heavy-build placement.

Failure mode

Provisioning now stops if the Docker-capable image cannot boot or if Docker, Compose, nested containers, DNS, or bridge egress do not work. At runtime, a missing image produces a stable capacity response instead of a partial sandbox lifecycle.

Checks

  • cargo fmt --all -- --check
  • cargo clippy --workspace --all-targets -- -D warnings
  • cargo test --workspace (59 sandboxd unit tests and 22 integration tests passed)
  • cargo audit (passed; one repository-allowed anyhow warning reported)
  • cd sdk/typescript && npm ci && npm run build && npm pack --dry-run
  • cd sdk/python && python -m build
  • shellcheck deploy/provision-node.sh deploy/images/heavy-build/sandbox-init
  • built the image recipe locally and verified Docker, Compose, containerd/runc, legacy iptables, jq, and git
  • verified the Docker-enabled Firecracker path with an equivalent Docker 27 DinD custom image: cgroup v2, overlay2, Compose, a nested Alpine container, DNS, and HTTPS egress
  • the provisioner now enforces the same runtime checks against the curated heavy-build image before a node is accepted; this exact target-image smoke runs during Linux/KVM provisioning

@H3xept
H3xept force-pushed the agent/fix-heavy-build-rootfs branch from 15b90b2 to ebff2d4 Compare July 16, 2026 06:20
@H3xept
H3xept force-pushed the agent/fix-heavy-build-rootfs branch from ebff2d4 to 3f08403 Compare July 16, 2026 06:25
@H3xept
H3xept marked this pull request as ready for review July 16, 2026 09:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

heavy-build sandboxes fail because the curated rootfs is not provisioned

1 participant