Teardown matches the socket spelling the engine carries, and verifies after the cascade (refs #344, refs #345) - #363
Merged
Merged
Conversation
… verifies after the cascade (refs #344) Two halves of one defect, and they only work together. THE MATCHER. reap.go identifies this run's engine by looking for a socket path in a host-visible command line. Engine.paths() returned e.sock, the HOST path /tmp/snug-<uid>-<pid>/sock/podman-<pid>.sock. Since Tier C the engine is exec'd with an argv written in its DERIVED view — engine.go's Spec builds "unix://" + guestSock — so its host-visible cmdline names /snug/engine/sock/podman-<pid>.sock. engine.go:547-550 says the two names must not be tidied into one; reap.go was never told. Measured by the owed redteam round, twice: a decoy naming the GUEST path — what a leaked engine looks like — SURVIVED teardown, while a decoy naming the HOST path was killed and burned the full 15s budget. So waitQuiet returned nil on its first poll on every run, signalOwned was never reached, the SIGKILL fallback never happened, and the "this sandbox's containers did not die with it" warning was unreachable. #167's lesson — host-side code reading engine-namespaced pids — unlearned for paths. paths() now returns the recorded GUEST spelling, and Spec is the one place that records it: a second derivation of the host->guest mapping in reap.go would agree with Spec until the day it did not. It returns nil rather than falling back to e.sock when Spec never ran, because that spelling appears in no command line on the machine and falling back to it is indistinguishable from not sweeping while looking like a sweep. Emptiness has its own failure mode, so stopLocked reconciles it: a dialled lifeline with no mark is named as a wiring bug rather than swept vacuously. THE POSITION, which is why the matcher could not simply be swapped. stopLocked ran from OnPayloadExit — inside runStaged, BEFORE its deferred st.Close(), so before the Pdeathsig cascade P1 -> engine has fired. The engine is ALIVE there by construction, so the only thing that could make a working sweep go quiet is the engine's own idle timeout: measured 15.3s of added teardown on every clean run. quietBudget was idleTimeout + 5s precisely because of that, i.e. snug's own exit waited out a timeout whose entire purpose is to cover snug NOT BEING THERE. So Engine.Stop is split. Detach (drop the keepalive, idempotent) is what OnPayloadExit gets; Stop — verify, reap, remove the run directory — stays wired to the caller's own cleanup, which main.go registers with `defer ctr.cleanup()` BEFORE it calls sandbox.Run and therefore runs strictly after runStaged's deferred st.Close(). st.Close() does not return until P1 has been reaped, and the kernel delivers a child's pdeathsig in forget_original_parent before do_notify_parent wakes that wait, so by the time Stop sweeps the engine's SIGKILL is already queued. That is the first position from which the question has an answer other than "yes, obviously": a process the cascade killed is gone or a zombie, and a zombie reads back an empty cmdline, which cmdlineNamesPath already answers "not ours" to. What survives is an engine outside the cascade's reach — the case this sweep was written for. It also fixes a latent bug: step 4 removed the run directory, holding the live engine's socket and generated conf, while the engine was still serving it. quietBudget is REWRITTEN, not renumbered: 2s, covering scheduling and a walk of /proc for a SIGKILL already delivered, with the relation to idleTimeout deleted rather than kept as a coincidence. killBudget (3s) names the second wait, which was a bare literal. waitQuiet's doc loses the "quietBudget is idleTimeout + 5s" clause, which is now false. THE ACCIDENT SENTENCE INVERTED, and the new one says what actually holds. The old text accepted one risk: the only way this reaps something else is a foreign command line containing the host socket path. With the mark fixed the string is /snug/engine/sock/podman-<pid>.sock — a string the PAYLOAD can author, since it can put candidate pids on its own argv and payload processes are visible in the host /proc. What stops that is not the matcher but WHEN the sweep runs: after the stage is reaped, the sandbox's pid namespace has collapsed and no payload process is alive to match. Stated in the file, because moving the sweep back re-opens it. describe() now escapes through policy.VisibleText. internal/cli sweeps its own screens for forging runes and that sweep does not reach internal/engine, which is why this could print a raw command line for as long as it did. The warning is a screen a human reads especially carefully — it is the one telling them what to kill -9 — and what it renders is a container's or a process's own argv, not snug's text. Escaped BEFORE the length clamp, so a cut cannot leave a raw 0x9b behind. TESTS TestTeardownMatchesTheArgvTheEngineIsStartedWith — the ratchet, pure and cheap. Build an Engine, call Spec, assert every element of paths() appears in the argv Spec produced, plus the negative (the HOST spelling must not be a mark when the derived view gives a different name) and an empty-paths() control, since an empty mark set satisfies every other assertion while meaning the sweep matches nothing. It fails on the code that shipped, printing both spellings. TestTheTeardownWarningEscapesTheCommandLinesItPrints — a real process with a poisoned argv, read back through /proc the way describe() reads it. The poison goes in argv[0]: `sleep 60 <poison>` exits before /proc can be read, and `sh -c 'sleep 60' <poison>` is exec-optimised by the shell and silently loses the marker (the harness caveat on #344). Control that an ordinary command line still renders as itself, because a describe() that quoted everything would pass the escape assertion and make the warning unreadable. TestOwnedPIDsMatchesOnlyThisEnginesPaths now runs Spec first and builds its positive marker from the guest spelling. Without Spec its paths() is empty and every negative in it would pass vacuously. make gate green. PARKED, NOT FINISHED — read this before picking it up. Session stopped at the maintainer's call with this branch incomplete. What is here is coherent and green, but it is NOT ready to land, and the missing half is named rather than left to be rediscovered. DONE: - the matcher: paths() returns the recorded GUEST spelling, nil before Spec - the position: Stop split into Detach (payload exit) and Stop (after the cascade), wired in internal/cli/container.go - quietBudget rewritten to 2s with killBudget named, and the stale prose in main.go, exec.go and reap.go corrected - describe() escapes through policy.VisibleText - the Spec-argv ratchet, which is the deliverable: it fails on the code that shipped and passes here - make gate green; go test ./... green NOT DONE, and each is a real gap: - THE MANDATORY redteam ROUND HAS NOT RUN. Definition-of-done rule 3 applies — this is engine/stage, a host-integration surface, and it changes what gets SIGKILLed and when. It was deliberately not started, not skipped by oversight. Nothing here may land before it does. - The regression tests specified by the host-bridge review are missing: paths() empty until Spec has run plus the reconciliation warning; a decoy naming the HOST spelling must NOT match (the negative that makes guest-only a fact rather than an implementation detail); the SIGKILL escalation, which no test has ever exercised because it was unreachable; a source-text test pinning the three-package ordering (container.go wires Detach not Stop, main.go defers cleanup before sandbox.Run, exec.go defers st.Close); and the integration test that would have caught the 15s a naive matcher fix introduces — wall clock from payload exit to snug exit under 3s, with a mandatory positive control that the engine actually served. - SNUG_REAP_SOCK is dead (reaperScript reads only $SNUG_REAP_DIR since #167). The review says to delete the parameter and the variable in a separate commit while KEEPING the sentence as a rule about the reaper's argv. Not done here. - Unverified against a real podman: this host's /usr/bin/podman is distrobox's host-exec shim, which preflight P1 refuses, so every measurement behind this change used the committed fakepodman stand-in via $SNUG_PODMAN. The claim that a zombie engine reads back an empty cmdline, and the end-to-end timing, both want a run against a real engine.
…er's argv (refs #344) reaperScript has read only $SNUG_REAP_DIR since #167 deleted the reaper's host-side `podman stop`. SNUG_REAP_SOCK survived that deletion unread: a path handed to a process that by design outlives snug, for no reason anyone can state. startReaper's sock parameter goes with it. WHAT SURVIVES IS THE RULE, not the variable. The old comment said "the path travels in the ENVIRONMENT, not in argv, so the reaper's own command line does not name the socket the sweep in reap.go matches on". That is one variable's fact. Restated forward: EVERY run-specific path this helper is given travels in the environment. Its argv is `/bin/sh -c <script>` and the script names only variables, never values. Two /proc sweeps match on path strings — reap.go's cmdlineNamesPath and internal/sandbox/teardown.go's confirmTeardown — so a path in argv makes snug's own cleanup match its own sweep: reported as a leaked engine by the first, killed as one by the second were it not exempted by pid (#113). THE TEST ASSERTS THE SET, NOT THE SITE. TestReaperFiresOnEOF...'s negative named `sock` alone, so deleting the parameter would have turned it green by deletion rather than by the property holding. It now checks both the run directory (which startReaper is handed) and the socket path (which it is not, any more), with a distinguishable message per path. MUTATION OBSERVED. Adding runDir to the argv -- exec.Command("/bin/sh", "-c", reaperScript, runDir) -- fails it on both arms: the reaper's command line names this run's directory ("/tmp/.../snug-1000-508950"); a /proc sweep matching on path strings will match snug's own cleanup process. The empty-cmdline control (#317's arg_start window) is unchanged and still asks its question separately: an empty read means the assertion never ran. Two mentions of $SNUG_REAP_SOCK stay in engine_test.go as history (`rm -f "$SNUG_REAP_SOCK"`, `rm -rf "$(dirname "$SNUG_REAP_SOCK")"`) because that history is why the target is STATED rather than DERIVED. Marked as history at the first, so a grep finding no live one is not read as the comment being stale. ArmReaper's doc said it "needs only this run's socket path" -- now false, it needs the run DIRECTORY. Engine unchanged: fakepodman via $SNUG_PODMAN throughout, never a real podman (this host's /usr/bin/podman is distrobox's shim, refused by preflight P1). make gate green.
…ring (refs #344) Two of the five regression tests the host-bridge review specified. Both guard code that has never been exercised, for the same reason: #344's matcher named a string no process on the machine carried, so everything downstream of it was dead code that looked green. THE ESCALATION (internal/engine/reapescalation_test.go). stopLocked step 3 is waitQuiet -> signalOwned(SIGKILL) -> waitQuiet -> WARNING. The OUTER condition could never be true before this branch, so signalOwned, the second waitQuiet and the warning had never run once. TestStopEscalatesToSIGKILLWhenTheEngineOutlivesTheCascade builds an Engine, runs Spec (paths() is empty until it has), starts a decoy through the existing marker() helper -- /bin/sh -c 'read x' <mark>, which blocks forever because nothing closes its stdin, so it survives quietBudget (2s) by construction rather than by timing -- and asserts the decoy died BY SIGKILL. Three positive controls before Stop runs, each removing a way the test could pass vacuously: e.paths() non-empty (an empty mark set sweeps for nothing), the decoy's cmdline really names the mark, the decoy really alive. The wait is BOUNDED (5s, channel + select) rather than a bare Process.Wait: a broken escalation leaves the decoy alive forever, so a bare Wait would HANG the suite instead of failing it. MUTATION OBSERVED, run twice independently: signalOwned(..., syscall.SIGKILL) -> syscall.SIGCONT --- FAIL: TestStopEscalatesToSIGKILLWhenTheEngineOutlivesTheCascade (10.06s) the decoy (pid 525598) was still alive 5s after Stop() returned: Stop's SIGKILL escalation did not reach it That mutation also printed "snug: WARNING - this sandbox's containers did not die with it" with the decoy's real pid and cmdline -- the first time that branch has ever rendered. The final warning branch is NOT covered, stated in the file rather than faked: no process survives SIGKILL, and the one thing that does (a zombie) reads back an empty cmdline, which cmdlineNamesPath correctly answers "not ours" to, so it would make waitQuiet report quiet rather than reach the warning. No honest fixture exists for "signalled, still there, still identifiable". THE ORDERING (test/guard/enginereapordering_test.go). The fix's other half is a position, spread over three files in three packages that no compiler, vet, or in-package test sees together: 1. internal/cli/container.go cleanup calls eng.Stop(), onPayloadExit is eng.Detach -- never the reverse 2. internal/cli/main.go `defer ctr.cleanup()` textually BEFORE the sandbox.Run( call (LIFO: registered first, fires last) 3. internal/sandbox/exec.go `defer st.Close()` textually BEFORE the opts.OnPayloadExit() call Any one reintroduced alone silently reopens #344, and internal/engine's own tests cannot notice: they call stopLocked directly and never touch the wiring that decides WHEN it runs relative to the stage's collapse. Every regex is proved to match today's source before it is trusted to prove absence. mustFindOne fails on zero matches AND on more than one, with a message distinct from the ordering failure, so a rename fails loudly instead of the sweep finding nothing and reporting a clean pass. Test 1 additionally runs its negative pattern against a synthetic pre-#344 fixture first. MUTATIONS OBSERVED (one verified independently, all reverted): container.go onPayloadExit: eng.Detach -> eng.Stop "onPayloadExit set to eng.Detach (pattern onPayloadExit:\s*eng\.Detach) was not found at all" main.go `defer ctr.cleanup()` moved after sandbox.Run( "(byte offset 22182) does not appear before the sandbox.Run( call (byte offset 22125)" exec.go `defer st.Close()` moved after opts.OnPayloadExit() "(byte offset 26807) does not appear before the opts.OnPayloadExit() call (byte offset 26785)" TEST 3 IS THE WEAKEST OF THE THREE AND SAYS SO FIRST. opts.OnPayloadExit() is called synchronously inside runStaged's guard.wait callback, so it always runs before that function's own deferred st.Close() fires whichever line comes first textually. Reversing the two changes nothing the runtime does. What it pins is which window the call is WRITTEN to belong to, so a later edit making OnPayloadExit order-sensitive cannot silently inherit the wrong position. A reader who assumes it carries the same weight as tests 1 and 2 would trust it for something it does not do. Two stale references corrected before commit: the ratchet lives in reapmark_test.go, not reapscreen_test.go, and the escalation test is in internal/engine, not test/guard. No non-test source file touched. Engine throughout is fakepodman via $SNUG_PODMAN, never a real podman. make gate green.
…measurement (refs #344) The review's fifth slot wanted an integration test catching "the 15s a naive matcher fix adds": wall clock payload-exit to snug-exit, under 3s. Written against a real podman 5.8.4 bundle. THE MANDATORY MUTATION DOES NOT FAIL IT, and that is the deliverable rather than a shortfall. MEASURED, mutation = internal/cli/container.go's onPayloadExit: eng.Detach rewired to eng.Stop, the exact shape #344 shipped with. Real engine, real /v1.41/version 200 positive control in the same run being timed: fixed wiring 15ms mutated wiring 15ms, 16ms, 15ms (three runs) Threshold 1s, quietBudget 2s. No difference. Run twice independently, once with temporary stderr instrumentation in stopLocked (reverted): the first waitQuiet found ZERO processes owning the engine's socket on its FIRST poll, ~8-16ms after onPayloadExit ran, on BOTH wirings, across seven runs. WHY. internal/stage/serve.go's MainServe returns as soon as runOneSandbox has sent the "exited" event, and internal/cli/main.go's exitOnStageError calls os.Exit(0) the instant it returns. The engine is Pdeathsig'd to P1 (the stage), not to P0 -- internal/stage/enginefork.go:156. So P1 exits on its own, unconditionally and near-instantly, once the payload is reaped; it waits for nothing P0 does. By the time P0 wakes from its blocking read of that event and calls onPayloadExit, the cascade has already felled the engine. On a clean run the shipped bug costs nothing, so no black-box clean-run timing test can discriminate. THE 15.3s WAS MEASURED ON A DECOY, which is the whole discrepancy. A decoy started by a harness is not forked by the stage and carries no Pdeathsig, so it CANNOT die with P1: it survives, waitQuiet burns the whole budget, 15.3s is correct for that process. It models an engine OUTSIDE the cascade's reach -- the rare case the sweep exists for -- not an ordinary engine on a clean run. The review read a decoy's cost as every clean run's cost and the spec inherited it. THE POSITION FIX IS STILL RIGHT, on a narrower claim than the review made, and the file says so at length so nobody reverts it off the back of this measurement. Stage.Wait() (stage.go:484) returns on recvEvent -- when the event BYTES ARRIVE -- and does NOT wait for P1 to exit, so onPayloadExit sits in a RACE with a still-exiting P1 that P1 happens to win 7/7 on an idle laptop. Stage.Close() (stage.go:503) ends with cmd.Process.Wait(), which BLOCKS until P1 is reaped, and the kernel delivers a child's pdeathsig in forget_original_parent before do_notify_parent wakes that wait. Shipped wiring: a race it usually won. Fixed wiring: a barrier it cannot lose. Real property, not one a wall clock can see, because the losing side of the race is the rare side. WHAT THE TEST IS KEPT AS. A real-engine, positive-controlled ratchet that clean container teardown stays well under quietBudget -- worth catching for any reason -- with the package comment stating in its first section that it does NOT close #344's fifth slot. Threshold is quietBudget/2, computed by parsing internal/engine/engine.go's own `const quietBudget` at test time rather than copying the value, because that value already went stale once (15s to 2s, in this same commit series). The comment names the trap: raising this toward 3s is not a relaxation, it is a disablement. Three corrections made to the agent's draft before commit. It claimed the shipped bug's "cost floor IS quietBudget" while its own headline measurement refutes that -- the bound is about what a threshold can RESOLVE, never about what the bug costs. It framed P1's exit as making the split unnecessary, missing that Wait() returns on the event and Close() on the reap -- race versus guarantee. And it cited OWED-REDTEAM.md, which is gitignored: TestEveryDesignDocACommentCitesExists caught it ("a citation a reader cannot follow"), now cites the issue instead. No non-test source file touched. make gate green; the new test green under SNUG_REQUIRE_SANDBOX=1 with a real engine.
… expires loudly (refs #344) "Unwritable" is a fact about today's architecture, not a law. CLAUDE.md's own rule -- when a round clears a route, RECORD WHICH PROPERTY DID THE CLEARING -- exists because #313's round cleared a route on a property #276 then deleted, and the pass was silently inherited. The package comment said the fifth slot is not closed, which is the fact. It now says WHY, which is the part that expires. The slot is unfillable because, and only because, BOTH hold: 1. the engine is Pdeathsig'd to P1, the stage, not to P0 (internal/stage/enginefork.go:156), so P1's exit alone fells it; 2. MainServe waits on nothing P0 does -- it returns the moment runOneSandbox has sent the "exited" event, and exitOnStageError then calls os.Exit(0). Pdeathsig the engine to P0, or give MainServe anything to wait for, and the shipped bug acquires a real per-run cost -- at which point the wall-clock test becomes both writable and NECESSARY. The paragraph is addressed to whoever makes that change. make gate green.
…344, refs #345) Two conflicts, both prose, both in paragraphs #362 and this branch edited for opposite reasons. Neither side taken wholesale. reap.go, conflict 1 -- the accident sentence. #362 describes the matcher as UNFIXED and says so outright ("The matcher is not corrected here; #344 carries it"). This branch corrects it, so HEAD's substance wins: the mark is /snug/engine/sock/podman-<pid>.sock, a string the PAYLOAD can author, and what stops that is WHEN the sweep runs, not the matcher. Kept from #362: the host spelling is /tmp/snug-<uid>-<pid>/sock/podman-<pid>.sock and NOT under $XDG_RUNTIME_DIR, where an earlier design put it before #63 Tier B moved the run directory to /tmp. That fact now earns its place for a new reason -- it is the other half of paths() returning nil rather than falling back to e.sock: no process carries that string, so a fallback is indistinguishable from not sweeping. reap.go, conflict 2 -- describe()'s escaping. HEAD's substance again (the match is now a payload-authorable string, so escaping stopped being theoretical), plus #362's naming of WHICH test covers the sink, which HEAD omitted: TestNoSnugScreenEmitsARawControlCharacter drives the --dry-run screen and does not reach internal/engine; TestDescribeSanitisesACommandLine does. container.go -- not a real disagreement. main added `started = true` (disarming the early-cleanup defer), this branch added the Detach/Stop rationale. Both kept. The rationale is also CORRECTED by this branch's own measurement: it no longer says the shipped wiring makes snug wait out the engine's idle timeout, because that is false -- measured 15ms on both wirings against a real podman 5.8.4. It says what is true instead: Stage.Wait returns on recvEvent (the bytes ARRIVING), not on P1's exit, so payload-exit is a RACE that P1 merely happens to win; st.Close ends with cmd.Process.Wait, after which it cannot lose. A guarantee where there was a race. Five test call sites updated for main's two signature changes: Spec's fifth parameter (noSignaturePolicy(t) -- "this host configured none", never nil, which #307 makes an error naming the skipped ProjectHostSignaturePolicy) and New now taking a *policy.Policy (testPol(...)). #345's SECOND COPY, which is the reason this touches orphansweep.go at all. The doc comment there is correct since #362 -- the number IS recyclable while the fd is held, measured, with a closing warning that a future edit acting on the number is not made safe by the pin. Twelve lines below that warning, inside the very function it governs: // pid is pinned from here: the number cannot be recycled while pidfd is // held, so the checks below and the kill all name the same task. The original false claim, in its second location, and exactly the licence the corrected header withdraws. #270's shape: a rule fixed at the site the reporter quoted while a pre-existing copy survives. No behavioural consequence today -- the kill goes through the pin and fails closed with ESRCH. SWEPT THE SET, NOT THE SITE, before fixing. All three files the header claims "state the same guarantee the same way" were checked. NO THIRD COPY: internal/engine/reap.go:195 correct -- "pins the task the number named AT OPEN TIME, so pidfd_send_signal can never land on a later reuse" internal/sandbox/teardown.go:444 correct -- "pins whatever process holds pid RIGHT NOW"; names the recycled-occupant case explicitly and skips rather than kills So the header's claim about its two neighbours is TRUE, verified rather than inherited. Same guarantee, not the same words. The replacement says what the pin does guarantee, and adds the part the old text obscured: procStartTime and procNamespaceInodes below read /proc/<pid> BY NUMBER, so identity comes from what they COMPARE against the state file, never from the pin. make gate green.
…elling negative (refs #344) Completes the four that can be written. The fifth was the integration timing test; test/integration/enginereapteardown_test.go carries the measurement showing it cannot exist on this architecture. reapvacuity_test.go -- EMPTINESS, AND ITS RECONCILIATION. TestPathsIsEmptyUntilSpecHasRun: paths() is empty on a fresh Engine and non-empty after Spec. The second half is the positive control -- without it, "empty before Spec" passes on a paths() that returns nil always. TestStopLockedReconcilesADialledLifelineWithNoMark: both arms of `e.dialled && len(e.paths()) == 0`. A warning that always fires is as useless as one that never does, so the ordinary case (Spec ran, mark exists) is asserted silent. Stderr is captured by a local os.Pipe swap restored unconditionally; NO test-only writer field or hook was added to engine.go. The helper documents that it stops being safe if this package ever adds t.Parallel. reaphostspelling_test.go -- THE NEGATIVE THAT MAKES GUEST-ONLY A FACT. A live decoy naming the HOST spelling /tmp/snug-<uid>-<pid>/sock/podman-<pid>.sock must not be swept. That is exactly the string paths() returned before the fix, and exactly why the sweep matched nothing on every run. Asserted at BOTH levels: cmdlineNamesPath (the predicate) and ownedPIDs (what signalOwned and waitQuiet actually call). Two guards, because this test degrades into a vacuous one in two different ways: - MANDATORY GUEST-SPELLING POSITIVE CONTROL. A second decoy naming e.paths()[0] must be MATCHED. Without it, "the host decoy was not matched" is satisfied by a matcher that matches nothing at all -- #344's own defect, which would otherwise pass its own regression test. - host != guest asserted up front, as a Fatal. If those spellings ever became one string, both decoys become indistinguishable and every assertion holds for the wrong reason. MUTATIONS, both run independently rather than taken on report. 1. paths() -> []string{e.sock}, the pre-#344 spelling. Fails all three new tests, but for test B it fires the VACUITY GUARD rather than the negative: e.Socket() ("/tmp/snug-1000-619877/sock/podman-619877.sock") and e.paths()[0] (...same...) are identical -- ... makes the whole test vacuous Correct behaviour, and not sufficient proof: it shows the test refuses to run vacuously, not that the negative assertion works. 2. paths() -> []string{e.guestSock, e.sock}. Host and guest stay distinct, the guard passes, and the NEGATIVE fails for the right reason at both levels while the guest positive control still holds: cmdlineNamesPath matched decoy pid 620437, whose command line names the HOST socket spelling ... the engine is never exec'd with this string ownedPIDs(paths(), exclude) reports the host-spelling decoy pid 620437 as owned by this run That is the one that proves the assertion, and it is why mutation 1 alone would have been a weaker claim than it looks. Reconciliation-condition mutations (forced false, forced true) fail the respective arms of the stopLocked test. No production source file touched. make gate green; the three new tests green 10x with no flakiness from the arg_start-window polling.
This was referenced Aug 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
refs #344, refs #345
Teardown's verification leg matched a string no process on the machine carries,
so it could not fail. Fixed, plus the four regression tests that hold it, plus
the measurement showing the fifth cannot exist.
redteamround: OWED, deferred by maintainer ruling, NOT waived. See thelast section — it names the round's target so whoever runs it does not have to
re-derive it.
The defect
Engine.paths()returned the HOST socket/tmp/snug-<uid>-<pid>/sock/podman-<pid>.sock.Since Tier C the engine is exec'd with an argv written in its DERIVED view —
Specbuilds"unix://" + guestSock— so its host-visible cmdline names/snug/engine/sock/podman-<pid>.sock.engine.gosays the two names must not betidied into one;
reap.gowas never told.So
waitQuietreturned nil on its first poll of every run,signalOwnedwasnever reached, the SIGKILL escalation never happened, and the "this sandbox's
containers did not die with it" warning was unreachable. #167's lesson —
host-side code reading engine-namespaced pids — unlearned for paths.
paths()now returns the recorded GUEST spelling, andSpecis the one placethat records it: a second derivation of the host→guest mapping in
reap.gowould agree with
Specuntil the day it did not. It returns nil rather thanfalling back to
e.sock, because that spelling appears in no command line on themachine, so a fallback is indistinguishable from not sweeping while looking like
a sweep. Emptiness has its own failure mode, so
stopLockedreconciles itagainst the lifeline.
The position
Stopsplit intoDetach(drop the keepalive, at payload exit) andStop(verify, reap, remove the run directory, from the caller's own deferred
cleanup). Also fixes a latent bug: step 4 removed the run directory, holding the
live engine's socket and generated conf, while the engine was still serving it.
Be precise about what the split buys, because the loose version is false and
this PR measured it. Real podman 5.8.4 bundle, wall clock payload-exit to
snug-exit:
No difference. The engine is
Pdeathsig'd to P1, the stage, not to P0(
internal/stage/enginefork.go:156), andMainServereturns the momentrunOneSandboxhas sent the"exited"event, whereuponexitOnStageErrorcalls
os.Exit(0). P1 waits for nothing P0 does. On a clean run the shippedwiring costs nothing.
What the position actually buys is a GUARANTEE where there was a RACE:
Stage.Wait()returns onrecvEvent— when the event BYTES ARRIVE — and doesNOT wait for P1 to exit. At payload exit, P0 and a still-exiting P1 race on
separate cores. P1 wins 7/7 on an idle laptop and nothing MAKES it win.
Stage.Close()ends withcmd.Process.Wait(), which BLOCKS until P1 isreaped, and the kernel delivers a child's pdeathsig in
forget_original_parentbefore that wait wakes. After it, the engine'sSIGKILL is already queued.
A race it usually won, versus a barrier it cannot lose.
The 15.3s was measured on a decoy
The round that found #344 recorded "15.3s of added teardown with a decoy
standing in for the engine". A decoy started by a harness is not forked by the
stage and carries no
Pdeathsig, so it CANNOT die with P1: it survives,waitQuietburns the whole budget, and 15.3s is exactly right for thatprocess. What it models is an engine outside the cascade's reach — the rare
case the sweep exists for — never an ordinary engine on a clean run.
General form, which is the transferable part: a substitute in a round models
the subject only in the dimensions it shares with it, and the round must say
which dimensions those are. This decoy shared the cmdline (the dimension under
test) and not the parentage (the dimension that decides whether it dies).
quietBudgetis REWRITTEN, not renumbered: 2s, covering scheduling and a walk of/procfor a SIGKILL already delivered, with the relation toidleTimeoutdeleted rather than kept as a coincidence.
killBudget(3s) names the secondwait, which was a bare literal.
The accident sentence, inverted
The old text accepted one risk: the only way this reaps something else is a
foreign command line containing the host socket path. With the mark fixed the
string is
/snug/engine/sock/podman-<pid>.sock— a string the PAYLOAD canauthor, since it can put candidate pids on its own argv and payload processes
are visible in the host
/proc. What stops that is not the matcher but WHEN thesweep runs: after the stage is reaped, the sandbox's pid namespace has collapsed
and no payload process is alive to match. Stated in the file, because moving the
sweep back re-opens it.
Tests
Four of five specified, plus a documented impossibility for the fifth.
TestTeardownMatchesTheArgvTheEngineIsStartedWithpaths()appears in the argvSpecproduced; fails on the code that shippedTestStopEscalatesToSIGKILLWhenTheEngineOutlivesTheCascadestopLockedstep 3 for the first time in this repo's historyTestPathsIsEmptyUntilSpecHasRunpaths()empty beforeSpec, non-empty after (the positive control)TestStopLockedReconcilesADialledLifelineWithNoMarkTestSweepDoesNotMatchTheHostSocketSpellingcmdlineNamesPathandownedPIDstest/guard/enginereapordering_test.go(3 tests)container.gowiresDetachnotStop,main.godefers cleanup beforesandbox.Run,exec.godefersst.CloseTestDescribeSanitisesACommandLineA mutation that trips a test's own guard has graded the guard, not the assertion
Stated as a method, because it generalises to every negative test in this repo
and it is not obvious: a mutation is only evidence about the assertion it
actually reaches.
The host-spelling negative needed two. Reverting
paths()toe.sock— thepre-#344 spelling, the obvious mutation — does fail the test, but at the VACUITY
GUARD: host and guest collapse to the same string and the guard fires first.
That grades the guard. It says the test refuses to run vacuously, and nothing
whatsoever about whether the negative assertion works. Reported as sufficient, it
would have looked like proof.
paths()returning{e.guestSock, e.sock}keeps the two spellings distinct, sothe guard passes and the mutation reaches the assertion — which then fails at
both
cmdlineNamesPathandownedPIDs, while the guest-spelling positivecontrol still holds. That is the one that grades the test.
Before believing a mutation, check WHICH assertion it tripped.
Every one mutation-proved. The escalation test's decoy blocks on a shell
readnothing closes, so it outlives
quietBudgetby construction rather than bytiming;
SIGKILL→SIGCONTfails it and prints the never-before-rendered "didnot die with it" warning. The ordering guard proves each regex matches today's
source before trusting it to prove absence, and
mustFindOnefails on zero ANDon more than one, with distinct messages.
The ordering guard's third test is the weakest and says so in its first
sentence.
opts.OnPayloadExit()is called synchronously insiderunStaged'sguard.waitcallback, so it always precedes that function's own deferredst.Close()whichever line comes first textually. It pins a written convention,not a runtime ordering.
The fifth test cannot exist, and that is the deliverable
test/integration/enginereapteardown_test.gowas written as the fifth slot andthe mandatory mutation (
onPayloadExit: eng.Detach→eng.Stop) DOES NOT FAILIT — measurement above. No threshold was tuned to make it pass. On a clean run
no black-box wall-clock test can distinguish the fixed wiring from the shipped
bug's.
Kept as a narrower real-engine ratchet: clean teardown stays well under
quietBudget, positive-controlled by a real/v1.41/version200 in the same runbeing timed. Its threshold is
quietBudget/2, computed by parsinginternal/engine/engine.go's ownconst quietBudgetat test time rather thancopied, because that value already went stale once. The comment names the trap:
raising it toward 3s is not a relaxation, it is a disablement.
The property doing the clearing is named, so a later change re-opens the
question instead of inheriting a stale pass (CLAUDE.md's rule, and #313's round
cleared a route on a property #276 then deleted). The slot is unfillable because,
and only because, the engine is
Pdeathsig'd to P1 rather than P0 ANDMainServewaits on nothing P0 does. Change either and the wall-clock testbecomes both writable and necessary.
Also here
SNUG_REAP_SOCKdeleted.reaperScripthas read only$SNUG_REAP_DIRsince#167 deleted the reaper's host-side
podman stop; the variable survived thatdeletion unread — a path handed to a process designed to outlive snug, for no
reason anyone can state. The sentence survives as a RULE: every run-specific path
travels in the environment, never argv, because two
/procsweeps match on pathstrings and a path in argv makes snug's own cleanup match its own sweep. The
test's negative named
sockalone, so deleting the parameter would have turnedit green by deletion — it now asserts the SET (run directory and socket
path).
#345's surviving second copy (
refs #345).orphansweep.go's doc comment iscorrect since #362 — the number IS recyclable while the fd is held, measured —
and twelve lines below its closing warning, inside the function that warning
governs, sat the original false claim: "the number cannot be recycled while
pidfd is held, so the checks below and the kill all name the same task." #270's
shape: a rule fixed at the site the reporter quoted while a pre-existing copy
survives. No behavioural consequence today — the kill goes through the pin and
fails closed with ESRCH.
Swept the SET, not the site, before fixing. No third copy.
reap.go:195andteardown.go:444both state it correctly, so the header's claim about its twoneighbours is true — verified rather than inherited. Nuance worth carrying: they
state the same GUARANTEE, not the same WORDS, so a fixed-string sweep would
report two sites and miss the third, and a sweep that finds nothing looks
identical to one that proves absence.
Five stale inheritances in one branch, and what caught each
All the same shape — prose or a spec copying state that lived somewhere else,
each true where it was written and false where it was read. Recorded as a base
rate, not a confession, so the catches are counted alongside the defects:
excluding the ones caught in-lane would quietly imply this mechanism only
produces defects that ship.
/usr/bin/podman(the distrobox shim P1 refuses by name), false of the pinned 5.8.4 bundle the suite drivesunder 3s— derived fromquietBudget = 15s, which this same commit series rewrote to 2s, so the threshold passed on the bugorphansweep.go's inline pin claim — corrected at the header in #362, left standing twelve lines below, inside the function that header governsThe distribution is the interesting part. One was caught by a mechanical
guard (
TestEveryDesignDocACommentCitesExists, on a separate citation defect inthe same lane). One by checking the tree after a merge. Three only because
someone refused to repeat a number without checking it. That is an argument for
the habit, not an apology for the branch — and it is the same measured base rate
CLAUDE.md already records for
@git-robinding~/.gitconfigfor a fullmilestone before a mechanical sweep, not a
redteamround, caught it.Engines used
fakepodmanvia$SNUG_PODMANfor the matcher measurements behind80ca6b2.The pinned static podman 5.8.4 bundle for the integration timing test and its
positive control. Per-claim, not per-branch:
/usr/bin/podmanhere is distrobox'sdistrobox-host-execshim, which preflight P1 refuses by name.Gate
make gategreen on every commit. Integration green with a real engine.The owed
redteamround#344 owes a
redteamround. It was deferred by maintainer ruling, notwaived. Definition-of-done rule 3 applies in full: this is engine/stage and it
changes what gets SIGKILLed and when. The round is scheduled against MERGED code,
in its own lane.
The reasoning is worth stating rather than referencing: a round is worth running
against the code that SHIPS, and a lane in flight is not that code yet. Running
one mid-lane spends it on a diff that is still moving and produces the one thing
a round record must never be — a note saying a surface was attacked when what was
attacked was a draft.
The round's target, written here so it does not have to be re-derived:
/snug/engine/sock/podman-<pid>.sockagainst decoys naming the HOST spelling/tmp/snug-<uid>-<pid>/sock/podman-<pid>.sock. The guest string is one aPAYLOAD can author on its own argv, which the host string was not.
stopLockedstep 3 — unreachable before thisbranch, so it has one unit test and no adversarial exposure at all.
holds. The sweep runs only after the stage is reaped and the sandbox's pid
namespace has collapsed, so no payload process should be alive to match. That
is the claim; it has not been attacked.
Engines, per-claim and never per-branch. Everything behind
80ca6b2wasmeasured against
test/integration/testdata/fakepodmanvia$SNUG_PODMAN. Thenew integration work drives the pinned static podman 5.8.4 bundle at
~/.local/opt/podman-static./usr/bin/podmanon this host is distrobox'sdistrobox-host-execshim, which preflight P1 refuses by name.