Every deployment builds on the control plane, and there is no way to build anywhere else

[jlucaso1]

Problem description

temps-deployer registers exactly one Arc<dyn ImageBuilder>, the local Docker runtime (crates/temps-deployer/src/plugin.rs:268), and WorkflowExecutionService hands that same instance to every BuildImageJob it constructs (crates/temps-deployments/src/services/workflow_execution_service.rs:1301). There is no configuration, per project or per environment, that changes where a build runs. Every build runs on the control plane.

On the 4 GB reference box that build contends with the Pingora proxy, Postgres/TimescaleDB, analytics ingest, the log aggregator and every running container. A Next.js or Rust build peaks at several GB and saturates every core for minutes, and the proxy's tail latency is collateral damage. Preview environments make it worse: they produce the most builds, they are worth the least per build, and they land on the same box as production traffic.

Two smaller costs come from the same place. Cross-architecture builds are QEMU-emulated on the control plane, which multiplies them by roughly an order of magnitude. And there is no layer cache that survives anything: DockerRuntime::build_image sets BuildKit's version/session (crates/temps-deployer/src/docker.rs:1292) but exports no cache, so a rebuild after a restart, a reinstall, or on a second control plane starts from zero.

Building is the most resource-hungry, burstiest and least latency-sensitive thing the box does. It is the one workload that should be movable, and it is the one workload that cannot move.

Proposed solution

What exists today points the other way

Temps has a documented CI story (docs/howto/set-up-ci-cd-pipeline), and all of it is the inverse integration: CI builds, then tells Temps. POST /projects/{id}/images/push registers an externally built image, POST …/deploy/image-upload takes a docker save tarball, POST …/deploy/static takes a bundle, POST /projects/{id}/trigger-pipeline starts a source build on the control plane.

That works, and it should stay. What it costs is a workflow file per repository, written and maintained by the user, kept in sync by hand with whatever Temps expects. It solves the problem one repo at a time.

Nothing in the tree drives a builder. I grepped crates/, web/src and apps/ for workflow_dispatch, repository_dispatch, actions/runs and token.actions.githubusercontent — no hits. No dispatch, no run polling, no OIDC verification.

One near-miss worth naming, because it reads like the feature already exists: RemoteNodeDeployer implements ImageBuilder and talks to a node agent over HTTP, but both of its build methods return Err("Remote image building not supported — images are transferred via tar") (crates/temps-deployer/src/remote.rs:455), and the agent exposes only /agent/images/import and /agent/images/{name}/exists (crates/temps-agent/src/server.rs:100). Building on a worker node needs a real endpoint on the agent. What that type does prove is the seam: an ImageBuilder that is not the local daemon, speaking HTTP, already works inside the job DAG unmodified.

The part that makes this harder than it looks

Moving the build is not enough, because five things downstream of the build read the image off the local daemon. A remote build that leaves the image somewhere else breaks all of them, and pulling the image back to satisfy them spends most of what the move saved.

consumer	what it needs	where
DeployImageJob	inspect_image for the EXPOSE port and for platform validation, three call sites	jobs/deploy_image.rs:475, :564, :615
ScanVulnerabilitiesJob	Trivy in a container with the Docker socket; it explicitly asserts the image is local first	jobs/scan_vulnerabilities.rs:224 → temps-vulnerability-scanner/src/trivy.rs:396
CaptureSourceMapsJob	inspect_image for WORKDIR, then extract_from_image	jobs/capture_source_maps.rs:190, :244
PersistStaticAssetsJob	same pair	jobs/persist_static_assets.rs:231, :323
DeployStaticJob	extract_from_image — this is the static deployment	jobs/deploy_static.rs:207

The scan is not a corner case. The planner adds it whenever has_git_info, which is every git deployment, and trivy.rs:396 fails with Target image '{}' not found locally rather than falling back to a registry pull.

extract_from_image is docker create plus a copy out of the container filesystem (temps-deployer/src/docker.rs:1727), so it needs a daemon holding that image, not just the bytes.

So the design cannot be "build elsewhere, carry on". Either the image comes back — and on the reference box, pulling a 1.2 GB Node image back to run Trivy against it is most of the cost we were trying to avoid — or the work that needs the image happens where the image already is. That is the second one, and it turns out to be strictly better: it moves the Trivy scan off the control plane too, which is the other multi-minute CPU burst in the pipeline.

The consequence for the protocol: a build does not return an image, it returns a result envelope — digest, platforms, the image config, and whatever side artifacts the plan asked for.

pub struct BuildResultEnvelope {
    pub digest: String,
    pub platforms: Vec<String>,
    /// Authoritative image config, so `inspect_image` needs no local image:
    /// WORKDIR, ExposedPorts, User, Env, Entrypoint, architecture, size.
    pub config: ImageConfig,
    /// Extractions the plan requested, uploaded by the runner: static bundle,
    /// source maps, source files.
    pub artifacts: Vec<ArtifactRef>,
    /// Trivy findings, when the plan asked for a scan.
    pub scan: Option<ScanReport>,
}

ImageInfo (temps-deployer/src/lib.rs:123) can then be constructed from config without a daemon, which covers all four inspect_image callers. The static path gets better than it is today: the runner uploads a bundle through the existing deploy/static route and the control plane never touches the image at all.

Why "add a GitHub Actions integration" is the wrong shape

It serves the users who need this least.

The operator whose 4 GB box is being eaten by builds is disproportionately the one self-hosting everything, on Gitea or on a plain git URL. GitProviderService has the repository-writing capability any dispatch-based design needs — create_repository, push_files_to_repository, create_pull_request, mint_scoped_repo_token (crates/temps-git/src/services/git_provider.rs:471) — and Gitea, GitLab and Bitbucket implement it for real. The Generic provider returns NotImplemented for every one of them (crates/temps-git/src/services/generic_provider.rs:400).

That is not a gap to fill later. It is proof that a dispatch-only design cannot be universal, because there is no API on the other end to dispatch to. A design that starts from a CI vendor has to grow the portable path afterwards; a design that starts from the portable path gets the CI vendors as adapters.

There is also a class of operator for whom sending source to a third party is not a configuration choice. A design whose only remote builder is somebody else's cloud has nothing to offer them.

Pieces that already exist

Most of this is assembly rather than new infrastructure.

piece	where
ImageBuilder as a dispatchable seam, already used by a non-local implementation	crates/temps-deployer/src/remote.rs
Outbound long-poll control channel with a generation cursor and a 25 s hold	crates/temps-routes/src/route_sync.rs, crates/temps-agent/src/route_sync_client.rs
Bounded-use, label-pinned, hash-only enrollment tokens	crates/temps-entities/src/node_enrollment_tokens.rs
Short-lived, permission-scoped, deployment-bound credentials	deployment_tokens, already carries deployment_id, permissions, expires_at
Per-environment config overriding per-project config, already used for build decisions	crates/temps-entities/src/deployment_config.rs:246, resolved at workflow_execution_service.rs:1229-1238
Provider-agnostic repository creation and file push, incl. self-hosted via base_url/api_url	git_provider.rs:471, crates/temps-entities/src/git_providers.rs:14
A deployment parked on an external condition, re-checked from the job queue	DeploymentGateRecheck, services/job_processor.rs:717
An upload route for a built image and for a static bundle	deploy/image-upload, deploy/static
Isolated compute on nodes, Docker and Firecracker backends	crates/temps-sandbox/src/services/registry.rs
mTLS to a node over the cluster CA	RemoteNodeDeployer::new_mtls
Log ingest by log_id, with redaction and failure-report extraction already wired	temps-logs, services/log_redaction.rs

Proposal

One protocol, one runner, several ways to start it.

The runner

A single artifact, temps-build-runner, shipped as a static binary and an OCI image:

authenticate → GET plan → GET context → build → extract/scan → stream logs → POST result

What differs between environments is only how a runner comes to exist:

• Pull. A long-lived runner long-polls Temps for work. Needs outbound HTTPS and nothing else.
• Push. Temps asks a provider to start an ephemeral runner.

Both end in the same binary speaking the same protocol, which is the whole portability claim. Pull mode is the floor: it is the only mode that serves a Generic git provider, it works behind NAT, and setup is one command on any machine.

docker run -d --restart=always ghcr.io/gotempsh/build-runner:1 \
  --url https://temps.example.com --token trn_… --labels arch=amd64,size=large

A persistent runner also keeps its BuildKit cache and base layers on local disk, so the second build of a project is fast with no registry round-trip. Push mode has the mirror advantage: zero idle cost and unbounded elasticity, with a cold cache and a provider dependency.

The runner does not need a Docker socket, which matters because a socket mount is root-equivalent on the host and pull-mode runners will sit on machines that run other things. Every step has a daemonless form: BuildKit builds (rootless where the kernel allows it, otherwise a daemonless container); Trivy scans a docker save tarball or an OCI directory with --input and works with no Docker client present; extraction reads the OCI layout directly. The runner declares its build backend and isolation as capabilities, and socket mode stays available as an opt-in for environments where rootless is not viable.

The seam stays ImageBuilder

pub struct RoutedImageBuilder {
    executor: Arc<dyn BuildExecutor>,
    local:    Arc<dyn ImageBuilder>,   // fallback
    policy:   ResolvedBuildPolicy,
}

WorkflowPlanner does not change, BuildImageJob does not change, and the DAG, the pipeline view and the log viewer keep working, because the contract they depend on is unchanged. The single edit is at workflow_execution_service.rs:1301, where self.image_builder becomes a policy-resolved builder. The downstream jobs change only in where they get their inputs: from the result envelope when the build was remote, from the daemon when it was local.

A dedicated HostedBuildJob job type would fork the DAG and duplicate every consumer of build output, which turns "a remote build behaves like a local one" into a promise maintained by hand rather than a property of the type system.

The executor ladder

Ranked by how much of a third party each one needs. An operator picks a rung; the rungs below stay available as fallbacks.

	executor	needs	serves
L0	A Temps node or sandbox	nothing external	air-gapped, regulated, or anyone with a second node
L1	Pull pool, any machine	outbound HTTPS	everyone, including Generic providers and NAT
L2	Dispatch to CI (Actions, GitLab CI, Gitea Actions ≥ 1.24, k8s Job)	a CI API	elasticity, zero idle cost

Zero YAML, where there is CI at all

For dispatch, something has to exist on the provider side, because none of GitHub, GitLab or Gitea will execute a pipeline definition handed to them over the API.

Temps creates it once, through create_repository + push_files_to_repository: one builder repository per installation, not one workflow per project. Onboarding project #500 is then a database row, with no commit and nothing added to anyone's product repo. The shim is 5–15 lines and receives an opaque build_id; Dockerfile, build args, tags, platforms, cache and sink all come from the API at runtime. That is what makes zero-YAML durable — shipping a Temps feature never means re-committing YAML to N repositories.

The only provider-specific parts are the file path and the shim body: .github/workflows/temps-build.yml for GitHub and Gitea Actions, .gitlab-ci.yml for GitLab.

Per-environment policy, on the existing override chain

DeploymentConfig is a typed struct in an existing JSONB column, already resolved environment → project with three-state Option semantics, and #489 established the precedent by putting cross_architecture_builds there. One new section, no migration:

pub struct BuildPolicy {
    pub executor: Option<ExecutorKind>,             // Local | Node | Pool | Dispatch
    pub executor_ref: Option<String>,
    pub selector: Option<BTreeMap<String, String>>, // portable placement, like node labels
    pub runner_spec: Option<String>,                // provider-native: "ubuntu-24.04-arm"
    pub platforms: Option<Vec<String>>,
    pub allow_secret_build_args: Option<bool>,      // default false
    pub fallback_to_local: Option<bool>,            // default true
    pub max_queue_wait_seconds: Option<u32>,        // default 300
    pub timeout_minutes: Option<u32>,
    pub cache: Option<BuildCacheMode>,
}

selector is the portable knob and runner_spec the escape hatch, the same split NodeScheduler already makes between labels and node ids. Preview environments inherit the project policy with no new concept, which is the right default since they are the best offload candidates. An operator wanting the opposite sets Dispatch on the project and Node on production.

Identity is configuration, not code

Trust anchors are rows, so adding a CI system is an admin form rather than a release:

pub struct TrustedIdentityIssuer {
    pub issuer_url: String,                  // https://token.actions.githubusercontent.com
    pub jwks_uri: Option<String>,            // else OIDC discovery
    pub audience: String,
    pub required_claims: serde_json::Value,  // {"repository": "acme/temps-builders"}
    pub run_id_claim: String,                // "run_id" | "pipeline_id"
}

This is not speculative shape-fitting: GitHub Actions issues repository and run_id, GitLab issues project_path, namespace_path, ref, pipeline_id and job_id and publishes JWKS at /oauth/discovery/keys. The same four fields describe both.

Below OIDC, an enrollment token modelled on node_enrollment_tokens for pull-mode pools and CI without OIDC; below that, mTLS over the cluster CA for L0. Every anchor produces the same thing: a deployment_tokens row valid for one build, four endpoints and a deadline, revoked on first result.

permission	endpoints
builds:read	GET /api/builds/{id}/plan, GET …/context.tar.zst
builds:logs	POST /api/builds/{id}/logs
builds:artifacts	POST /api/builds/{id}/artifacts
builds:result	POST /api/builds/{id}/result, once

A compromised runner leaks one build's context, which it already had in its working directory, and nothing else.

Context is served by Temps, not cloned by the runner

DownloadRepoJob stays on the control plane and the prepared context is served as tar.zst. Two reasons that survive every executor choice.

Dockerfile generation is source-tree dependent — PresetProvider::dockerfile takes a local_path and inspects it, and the Nixpacks preset runs autopack over the checked-out directory (crates/temps-presets/src/nixpacks_preset.rs:417). Moving generation to the runner means shipping the preset engine to the runner.

And the runner then receives exactly what a local build would, including the generated Dockerfile, the .dockerignore and the confinement checks BuildImageJob already performs, so "same semantics as local" is a property of the design rather than a test we hope covers it. It also removes a class of coupling: the runner needs no git access, so a GitHub-hosted runner can build a GitLab-hosted repo.

What leaves the control plane is a full build and a Trivy scan. What stays is a shallow clone and a few MB of HTTP.

Artifact return: tar by default, registry when configured

sink	when	cost
Tar upload, existing deploy/image-upload path	default	no credentials, no new infrastructure; whole image through the control plane, 1 GB cap
OCI registry	operator supplies credentials	layer dedup, real multi-arch manifests, and nodes can pull directly — once the agent learns an authenticated pull
Local	L0 node executor building on the node that will run the container	nothing moves at all

Tar is the default because the alternative cannot be made zero-configuration on any provider, including GitHub. GHCR does not accept GitHub App installation tokens: docker login succeeds and docker pull returns denied, confirmed by GitHub staff in August 2025 and still unresolved as of July 2026 (discussion #171423). Inside a run the ephemeral GITHUB_TOKEN with packages: read works, and outside it the only supported credential is a classic PAT with read:packages — fine-grained tokens are not supported either. So the registry sink needs an operator-supplied credential on an image_registries row, encrypted at rest via EncryptionService, for every registry including GHCR. Better to treat all registries the same than to build a GitHub special case that does not work.

Two corrections to claims worth making explicitly, because I made them earlier in the design and they were wrong:

• The registry sink does not retire the per-node tar transfer for free. Today PullExternalImageJob pulls on the control plane and ships a tar to nodes, and the agent's own pull (service_handlers.rs:221) passes no credentials, so it can only fetch public images. Nodes pulling private images directly is new work in the agent plus a credential-delivery path. It is worth doing, but it belongs in a later phase and not in the justification for this one.
• L0 and the tar sink are the paths with no registry story at all, and they are the ones that work everywhere on day one.

On every sink the runner reports a digest, never an image reference. Temps builds the reference server-side from the project's pinned namespace, resolves it, and checks digest, platforms and run id against what it dispatched. A runner cannot redirect a deployment to an image of its choosing.

Leases, because pull mode can lose a runner

Work is leased rather than assigned: a lease has a deadline extended by heartbeats, an expired lease returns the build to the queue, and after max_attempts the build falls back to local instead of looping.

Runners declare capabilities — build backend, platforms, cache backends, disk, whether they can push to a registry — at enrollment, and the router refuses to schedule a two-platform build onto a runner that cannot do it, rather than failing eight minutes in.

Pools are instance-scoped and selected by label, exactly like nodes. nodes has no project_id; projects reach specific nodes through configured_target_nodes and target_labels in deployment_config. Copying that avoids inventing a second tenancy model, and it gives cross-project cache reuse for free. A pool row carries an optional allowed_project_ids allowlist for operators who need hard isolation rather than a scheduling preference.

Degradation

With fallback_to_local on, which is the default, every executor failure — nothing configured, no eligible runner, queue timeout, capability mismatch, expired lease, missing shim, failed dispatch — writes a warning into the build log and builds locally. The deploy still ships.

The two that must never fall back are a digest mismatch and a rejected identity. A failed integrity check has to fail the deployment, not quietly build something else.

Restart safety is worth stating because remote builds can have it and local ones cannot: the run handle is persisted into deployment_jobs.outputs on acquire, before the await, and a reconciler re-attaches to in-flight builds on startup through a queue job shaped like the existing DeploymentGateRecheck.

Impact

The control plane stops competing with itself. A deploy's peak cost on the box drops from a full build plus a Trivy scan to a shallow clone plus a few MB of HTTP, and preview-heavy projects stop degrading production latency.

Cross-architecture builds stop being emulated when a native runner exists for the target, and persistent runners give a layer cache that survives restarts, which nothing does today.

Static deployments improve outright: the runner uploads a bundle and the control plane never handles the image, where today it creates a container and copies files out of it.

And an operator who cannot send source anywhere still gets all of it, on their own hardware, through L0 and L1.

What this is not

Not a CI product. Tests, lint and e2e stay where they are, and users who want them before a deploy keep their own workflow and call trigger-pipeline. Not a change of default either: local stays the default, and every rung is opt-in per environment. external_images and deploy/image-upload keep working unchanged.

Scope

Three phases, ordered so the first ships value with no CI vendor involved.

1. The protocol and the portable floor: the runner binary and image, the versioned build protocol including the result envelope and side artifacts, BuildExecutor and BuildArtifactSink, RoutedImageBuilder at the one construction site, pull-mode pools with enrollment, leases, heartbeats and selectors, the tar sink, runner-side extraction and scanning, deployment_config.build, the capabilities endpoint and settings card, fallback-to-local.

2. Dispatch adapters: GitHub Actions and GitLab CI via the builder-repo bootstrap, generic OIDC verification through TrustedIdentityIssuer, Gitea Actions gated on ≥ 1.24 with pull-mode degradation below it, cancel propagation, restart reconciliation.

3. Registry and own compute: the registry sink with encrypted credentials, an authenticated pull on the agent so nodes fetch directly, a build endpoint on temps-agent plus a build-capable node selector, the sandbox executor, and checkout: runner for presets with a static Dockerfile.

Happy to split this into per-phase issues, and to start with phase 1 only.

Open questions

The five I opened this with are answered above — the image-local consumers, GHCR and App tokens, Gitea's dispatch API, rootless runners, and pool scoping. What is left is genuinely undecided.

1. Layer extraction on the runner is the fiddliest piece. Reading a subtree out of an OCI layout means applying layers in order and honouring whiteouts and opaque directories. Hand-rolling that risks subtle wrong answers on exactly the Next.js layouts we care about. Shelling out to skopeo + umoci is correct but adds two binaries to the runner image. I lean toward the dependency, but it deserves an argument.
2. Multi-arch representation diverges by sink. feat(deployments): support worker nodes of a different architecture #489 produces one image per platform with suffixed tags and no manifest list, deliberately, because there was no registry. A registry sink makes real manifest lists possible. Do the two paths converge on manifest lists when a registry is present, or does the remote path keep suffixed tags for uniformity?
3. Should Temps ship its own registry? GHCR's App-token limitation makes "bring a credential" the price of the registry sink on every provider. A minimal distribution-v2 endpoint on temps-blob would remove that price and make nodes-pull-directly work with the credentials Temps already issues. It is a real chunk of work and its own ADR, but the case for it is stronger than it was before this research.
4. Does the vulnerability scan stay optional on the remote path? Running it on the runner is free CPU we do not pay for, so the temptation is to always run it. But it lengthens the critical path of a deploy, and today it is explicitly non-blocking. Keeping the current semantics means the result envelope has to tolerate a missing scan.

Related: #489 added the per-environment build configuration this builds on, and its required_build_platforms is the selector logic a build router would reuse for placement.

Alternatives considered

No response

Additional context

No response