manager: restart self-crashed processes (backport of commaai/openpilot#36755) - #311
manager: restart self-crashed processes (backport of commaai/openpilot#36755)#311PeterPhuTran wants to merge 3 commits into
Conversation
A managed process that dies on its own while its should_run predicate stays true is never reaped: stop() - the only place that logs the death and clears self.proc - is only reached when should_run flips false, and start() early-returns whenever self.proc is set, even for a dead child. The process stays down for the rest of the onroad session while managerState reports shouldBeRunning=True/running=False, so selfdrived raises processNotRunning (NO_ENTRY + SOFT_DISABLE) permanently. Observed on-device while investigating FrogAi#309: loggerd SIGABRTed 3s after starting (NVMe failure) and stayed dead for the whole session with the engagement-blocking alert on screen and no restart or death log. Upstream fixed this in commaai#36755 with an opt-in restart_if_crash flag and a reap in ensure_running; this backports that mechanism verbatim while preserving FrogPilot's watchdog machinery and should_run signature. Unlike upstream, the flag is also plumbed through NativeProcess (FrogPilot's crash-relevant targets are native). Flagged processes: ui (mirrors upstream's choice), plus loggerd and encoderd (per FrogAi#309 - a logging process crash must not permanently block engagement). Tradeoff accepted upstream: a flagged process that crash-loops is restarted once per manager loop pass. Adds test_restart_if_crash covering both the restart and the opt-in (unflagged stays dead) semantics. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Thanks for contributing! A maintainer will review your pull request soon. |
Field incident on 2026-07-10 (same device as FrogAi#309): updated logged a 130KB 'git diff output' record for a dirty working tree; logmessaged crashed publishing it to the 256KB logMessage msgq (msgq asserts 3*msg_size <= queue_size, so anything over ~85KB is fatal while logmessaged's own guard only rejects >2MB). Being unflagged, it stayed dead for the rest of the session - permanent processNotRunning alert and total swaglog loss (the crash of the logging daemon is the one crash it can never log). logmessaged is exactly the kind of always_run, non-safety process this flag exists for. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Pushed one more flag: |
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Description
A managed process that dies on its own while its
should_runpredicate stays trueis never restarted:
stop()— the only place that logs the death and clearsself.proc— is onlyreached when
should_runflips falsestart()early-returns wheneverself.procis set, even for a dead childcheck_watchdog()only covers processes withwatchdog_max_dtSo the process stays down for the rest of the onroad session while
managerStatereports
shouldBeRunning=True / running=False, and selfdrived raisesprocessNotRunning(NO_ENTRY + SOFT_DISABLE) permanently — engagement blockedand force-disengaged until the car is restarted.
Observed live during the #309 investigation: loggerd SIGABRTed 3 seconds after
starting (NVMe failure) and stayed dead the whole session — alert on screen, no
restart, no death log, RouteCount frozen.
Fix: backport of commaai#36755 — an opt-in
restart_if_crashflag onManagerProcessplus a reap inensure_running()(verbatim upstream logic),preserving FrogPilot's watchdog machinery and
should_runsignature. Unlikeupstream, the flag is also plumbed through
NativeProcess, since FrogPilot'scrash-relevant targets (loggerd/encoderd/TICI ui) are native.
Flagged processes:
ui(mirrors upstream's only flagged process), plusloggerdand
encoderd(per #309 — a logging-process crash must not permanently blockengagement).
Known tradeoff (accepted upstream in commaai#36755): a flagged process that crash-loops is
restarted once per manager loop pass. Opt-in, limited to these processes.
Verification
New
test_restart_if_crashinsystem/manager/test/test_manager.py:(comma three, including a
-W errorrun matching CI's-Werror)🤖 Generated with Claude Code