Summary
The scaffolded build job in .github/workflows/build-test-deploy.yml provisions the site and runs tests inside the drevops/ci-runner container on ubuntu-latest, using Docker-in-Docker. On projects whose provisioning is disk-heavy - a large database dump, a second (migration) database, and/or migrations that generate or download files - the job can exhaust the runner's disk during the Provision site step.
There is currently no disk-headroom management or disk visibility in the build job.
Why this is worth addressing: the failure is opaque and hard to diagnose
When the runner disk fills up:
- The job dies mid-Provision site with no step-level error - the step stays
in_progress and the job is killed externally, so it looks like a hang or a timeout rather than a disk problem.
- The GitHub Actions logs are unretrievable -
gh run view --log / --log-failed and the job-logs API all return 404. The runner ran out of disk and could not write its own diagnostic log, so there is nothing left to read. The only surviving clue is the runner-level error:
System.IO.IOException: No space left on device : '/home/runner/actions-runner/.../_diag/Worker_*.log'
- With no logs and no step error, "the runner ran out of disk" is effectively invisible; it presents as a generic, un-debuggable stall.
ubuntu-latest has roughly ~14 GB free on /, and the pulled Docker images, database volumes, and any files created during provisioning all accumulate there. A build that comfortably fits today can tip over after a dependency bump grows vendor/, or as the database/migration data grows over time.
Suggested improvements (in the build job, before "Provision site")
- Print
df -h before provisioning. This is the cheapest, highest-value change: when a job does fail this way, the logs would at least show the disk was full, turning an un-debuggable hang into an obvious one.
- Reclaim disk space - e.g. prune Docker build cache and unused images. (Freeing the runner's large preinstalled toolchains or moving Docker's data-root to the larger
/mnt volume reclaims far more, but is trickier from inside a container job and may be better as documentation than automation.)
- Document the runner disk constraint for projects with large databases or migration-based provisioning, and how to reclaim space.
Even item (1) alone would save significant diagnosis time, since the current failure gives no indication that disk is the cause.
Environment
- Vortex
build job (.github/workflows/build-test-deploy.yml), container drevops/ci-runner, runs-on: ubuntu-latest, Docker-in-Docker provisioning.
- Observed when a dependency update grew
vendor/ enough to push an already-tight disk over the limit during a migration-based provision.
Summary
The scaffolded
buildjob in.github/workflows/build-test-deploy.ymlprovisions the site and runs tests inside thedrevops/ci-runnercontainer onubuntu-latest, using Docker-in-Docker. On projects whose provisioning is disk-heavy - a large database dump, a second (migration) database, and/or migrations that generate or download files - the job can exhaust the runner's disk during the Provision site step.There is currently no disk-headroom management or disk visibility in the build job.
Why this is worth addressing: the failure is opaque and hard to diagnose
When the runner disk fills up:
in_progressand the job is killed externally, so it looks like a hang or a timeout rather than a disk problem.gh run view --log/--log-failedand the job-logs API all return404. The runner ran out of disk and could not write its own diagnostic log, so there is nothing left to read. The only surviving clue is the runner-level error:ubuntu-latesthas roughly ~14 GB free on/, and the pulled Docker images, database volumes, and any files created during provisioning all accumulate there. A build that comfortably fits today can tip over after a dependency bump growsvendor/, or as the database/migration data grows over time.Suggested improvements (in the
buildjob, before "Provision site")df -hbefore provisioning. This is the cheapest, highest-value change: when a job does fail this way, the logs would at least show the disk was full, turning an un-debuggable hang into an obvious one./mntvolume reclaims far more, but is trickier from inside a container job and may be better as documentation than automation.)Even item (1) alone would save significant diagnosis time, since the current failure gives no indication that disk is the cause.
Environment
buildjob (.github/workflows/build-test-deploy.yml), containerdrevops/ci-runner,runs-on: ubuntu-latest, Docker-in-Docker provisioning.vendor/enough to push an already-tight disk over the limit during a migration-based provision.