Skip to content

Disk-full ntfy alert never fires: check_disk_and_alert is unreachable from the deployed entrypoint #102

Description

@ClaydeCode

Summary

The host disk-usage alert added in #92 has never fired. check_disk_and_alert() is only reached from the freeshard exec loop, but the deployed container runs the orchestrator entrypoint, which never calls it. The disk hit 92% on clayde.net on 2026-08-07 (3.8 GB free) and no ntfy was sent.

Evidence

check_disk_and_alert() has exactly one caller on main:

$ git grep -n "check_disk_and_alert(" origin/main -- src/ | grep -v "def "
src/clayde/freeshard/loop.py:176:        check_disk_and_alert(settings)

There are two console entrypoints:

[project.scripts]
clayde = "clayde.orchestrator:run_loop"
clayde-freeshard = "clayde.freeshard.entry:run_loop"

The running container uses the first one, and orchestrator.py never references disk at all:

$ docker inspect clayde-clayde-1 --format '{{.Config.Entrypoint}}'
[/opt/clayde/.venv/bin/clayde]

$ git grep -n "disk" origin/main -- src/clayde/orchestrator.py
(no matches)

Two independent signs the code path is never entered, as opposed to entered and suppressed by the cooldown:

  • ~/clayde/data/disk_alert_state.json does not exist. It is written on every send.
  • docker logs clayde-clayde-1 contains zero lines matching disk — not the log.warning send branch, and not the log.info("disk %d%% over threshold but within alert cooldown") branch either.

Not a deployment-lag problem: #92 merged 2026-06-22, the running image was built 2026-06-30, and the container has been up 3 weeks. The code is in the image; it is simply unreachable in the mode that runs.

What is not wrong

disk_alert_path defaults to /data, which is the bind-mount living on the host root partition, so the measurement would have been correct. Config defaults are fine too: disk_alert_enabled=True, disk_alert_threshold_pct=85, disk_alert_cooldown_s=21600. Only the wiring is wrong.

Suggested fix

Move the call somewhere both entrypoints reach — the orchestrator tick, or a shared scheduler both loops drive. Whichever it is, the check should not live behind a single optional loop.

Acceptance criteria

Fix it so a passing test would fail against today's code. Specifically:

  • A test asserting that the orchestrator path invokes check_disk_and_alert. The existing tests/test_disk.py covers the function's own logic and passes today, which is exactly why this went unnoticed — the gap is in the wiring, not the function.
  • A manual confirmation that is falsifiable: point disk_alert_path at a nearly-full mount (or temporarily set disk_alert_threshold_pct below current usage), run the deployed entrypoint, and confirm an ntfy arrives and disk_alert_state.json appears. A check that has never been observed failing has not been verified.

Impact

Low severity, but it is a silent failure of a safety net that a runbook actively tells readers to rely on. The KB runbook misc/clayde-net/vm-disk-cleanup.md claimed protection at ≥85%; that claim is being corrected separately. A full root disk breaks clones, builds and the agent loop, which is the condition the alert exists to prevent.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions