gr2 propagation journal: a torn trailing line makes every later tick raise (undefended parse, now once per tick)

[laynepenney]

What

Journal._rows() in gr2/prototypes/propagation_state_machine.py reads journal.jsonl and calls json.loads on every non-blank line with no guard. The writer (_write) appends one JSON line and fsyncs, so a crash, a full disk, or a kill between the write and the fsync can leave a torn trailing line. The next read then raises json.JSONDecodeError (a ValueError) out of _rows() → rows_for() → find() → the machine → the daemon loop, which by design lets anything outside its named failure list propagate. Net effect for an operator: after one bad moment the daemon exits on its first tick and keeps exiting on every restart until someone hand-edits the journal, and the message is a JSON traceback, not a propagation diagnostic.

The cursor side already handles this correctly: cursor() catches JSONDecodeError/OSError and returns None, and advance_cursor() writes tmp → fsync → rename. The journal rows are the asymmetric half.

Why it matters more since Prototype 1

In Prototype 0 the journal was read once per operation (only when the source had moved). Prototype 1's daemon (branch feat/gr2-propagation-prototype-1) deliberately hands its observation to the machine on the not-new path too, so the machine's cursor check runs on every tick — which is right, because a corrupted sink behind an unchanged source must not read as healthy — but it means _rows() now parses the whole journal every 30 s. The frequency of the undefended parse moved from once-per-change to once-per-tick, and the cost is the whole file each time.

Expected

1. A torn trailing line (the only kind an append+fsync writer can produce) is tolerated: skipped, counted, surfaced once as a note or diagnostic, and never fatal. The intact prefix still answers the cursor check. This is safe only because of a write-ordering invariant that today lives in a code comment: _acknowledge appends the ACKNOWLEDGED transition row FIRST and advances the cursor AFTER ("the cursor is derived from it and repaired from it on replay, so the row is written first"). So a tear during acknowledgement leaves the cursor un-advanced and the operation simply replays; a torn trailing row can never be the only evidence the cursor depends on. Name that ordering as an invariant in the code (an assertion or a named helper, not only the comment), because a later reorder of those two writes would invalidate the tolerant read with nothing going red.
2. A malformed line that is not the last line is real corruption and should raise a named exception the loop lets propagate by name, not a bare ValueError. Today's JournalInconsistent docstring describes a different condition (a cursor naming a revision the journal cannot account for); either widen that docstring to "the journal cannot be read as a consistent record" or add a sibling, and say which in the fix.
3. Idle ticks should not re-parse the entire journal: cache parsed rows keyed on the file's size/mtime, or keep a read offset, so per-tick cost is proportional to what changed. (A size/mtime key is sound here because every write is an append and appends always move the size; the journal never rewrites in place.)

Witnesses to add

• Truncate journal.jsonl mid-last-line after an acknowledged operation; the next run_loop(once=True) must not raise, must report current, and must leave a diagnostic naming the torn line.
• Corrupt a middle line; run_loop(once=True) must raise the named exception chosen in expectation 2 (and the daemon's propagate-by-name witness covers the loop side).
• A witness that pins the write ordering: inject a fault between the ACKNOWLEDGED append and the cursor advance; the cursor must remain un-advanced and the next tick must replay and complete the operation. Reordering the two writes must redden it.
• A mutation that removes the trailing-line guard must redden exactly the first witness.

Provenance

Found in review of the Prototype 1 daemon (the reviewer disclosed it as pre-existing Prototype 0 substrate, not gating that PR, and noted the per-tick frequency change); the 2026-08-19 dogfood run of the daemon rode through a real network outage and a full-disk window on the host, which is exactly the class of moment that produces a torn line. Related: the Prototype 0 PR #889 and the Prototype 1 PR (dev-targeted, Ref — closes at promotion).

[laynepenney]

The discriminator in the issue body above is withdrawn as falsified, and the reason is worth recording because it changes what the fix has to be.

Measurement 1 — a tear costs two rows, not one

The issue treats a torn line as losing the interrupted write. It also destroys the next one. _write appends with no newline repair, so the following append glues onto the remnant:

{"n":1}
{"n":2}
{"n": 3, "no{"n":4}

Row 4 was serialized correctly, written, and fsynced. It is unreadable because row 3 died mid-line. This is the more serious half and it is not in the issue.

Measurement 2 — position cannot discriminate cause

The body says a torn trailing line is "the only kind an append+fsync writer can produce," and makes a non-trailing malformed line mean real corruption. That holds only until the daemon restarts and appends twice:

0: {"n":1}              parses
1: {"n":2}              parses
2: {"n": 3, "no{"n":4}  MALFORMED
3: {"n":5}              parses

The malformed line is now mid-file, produced by the ordinary writer, with no corruption anywhere. The rule would raise on the daemon's own crash-then-restart signature — refusing to run and demanding a human for a condition it caused itself and can tolerate. A check that fires on the common benign case is one people learn to route around.

What replaces it

The discriminator is cursor reconciliation, not line position. That check already exists and already raises JournalInconsistent by name; it asks the question actually worth asking — whether the sink is coherent — rather than a syntactic question about where a byte landed. Dropping an unparseable line lets that check be the one that decides, instead of being pre-empted by a parse error.

So expected behavior 2 above is restated: a malformed line anywhere means the daemon continues, skipping and counting it. JournalInconsistent fires only from cursor reconciliation.

The safety of skipping rests on the write ordering, which was checked rather than assumed: the acknowledgement row is written before the cursor advances. A torn acknowledgement therefore means the cursor never advanced, so the operation reads as unacknowledged and is redone idempotently. A destroyed acknowledged row is caught by reconciliation and is correctly loud. Both branches close.

Scope of the fix

1. _write terminates an unterminated remnant before appending, confining the damage to the line actually interrupted.
2. _rows drops unparseable lines and counts them, never raising; MalformedLine keeps the drop from being silent.
3. Rows are cached against (size, mtime_ns) for the per-tick cost, with the one state the cache cannot see named in the code rather than assumed away.