fix(kimaki): stop a failed system prompt patch from blocking service start - #570
Merged
Conversation
…start post-upgrade.sh runs with set -e and is wired as ExecStartPre with no '-' prefix, so any non-zero exit inside it prevents kimaki.service from starting at all. Pass 2 invoked node unguarded, which meant the failure branch immediately below it was unreachable: the shell exited before $? could be read. The error handling was written but could never run. The reachable shape of that is routine. 'npm install -g kimaki' runs as root and rewrites dist/system-message.js root-owned, while the rendered unit runs as a non-root SERVICE_USER. The next start of the service then tries to patch a root-owned file as that user, gets EACCES, and systemd refuses to start. One install sat down for five days across roughly 29,900 restart attempts before a version bump happened to let root re-patch the file. The journals for those attempts show node's stack trace followed directly by systemd's failure line, with neither the warning nor the closing summary printed - the branch never ran. Passes 1 and 3 already settled the rule for this file: mutating the root-owned package directory is best-effort hygiene, and being unable to do it is worth a warning, never a failed start. try_remove_package_path says so explicitly, and the obsolete-plugin removal repeats it. Pass 2 was the only one not following it, so this makes it follow it rather than adding a new mechanism. An unwritable target is now reported separately from a genuinely broken patch, and names the remedy, since 'root installed it, a service user runs it' is the expected shape rather than a defect. Tests pin both: a patch that fails must leave the script exiting 0 and reaching its summary, and the exact unwritable-target case. Both fail against the unfixed script (exit 2 and exit 1); the unwritable case is skipped when the caller is root, which bypasses file permissions.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug is one missing
||post-upgrade.shsetsset -euo pipefail(line 50) and is wired asExecStartPrewith no-prefix, so any non-zero exit inside it stops the service from starting at all.Pass 2 invoked
nodeunguarded:The failure handling was written correctly and could never execute —
set -eterminated the script at thenodeline, before$?was ever read.Why that matters in practice
The reachable shape is routine, not exotic:
One install sat down for five days across ~29,900 restart attempts (2026-08-28 → 2026-09-02), recovering only when a kimaki version bump happened to let root re-patch the file out of band. Four user requests were silently dropped in that window.
The journal is the proof that the branch never ran — node's stack trace goes straight to systemd's failure line, with neither the warning nor the closing summary in between:
Why this fix and not a chown
This file had already settled the rule for exactly this hazard, twice.
try_remove_package_path(Pass 1):And the obsolete-plugin removal in Pass 3 repeats it: "Same reasoning as try_remove_package_path."
Pass 2 was the only one of the three not following the rule. So this makes it follow the rule rather than introducing a new mechanism. Chowning the npm tree at install time was the other candidate, and it is strictly more machinery guarding a narrower case — it fixes the installs we provision, and does nothing for a service that dies because some other root process ran
npm install -g. The invariant that actually holds is the one this file already states: mutating the package directory is best-effort.Changes
|| patch_exit=$?on thenodeinvocation, which activates the existing branch.kimaki-system-message-patch.sh(already registered inshell.yml).Verification
Both new cases fail against the unfixed script and pass against the fixed one — baselined, not assumed:
runner, so it executes there.bash -nclean on both files; the repo'ssyntaxjob covers this.Scope
Deliberately not addressed here: the affected install's nightly workflow re-armed this every night by running
npm install -g kimakias root without re-patching. That is fixed separately in that repo (chubes4/h44-lacrosse#3). This PR is the reason a mistake like that could take a host down instead of merely leaving a prompt unpatched.