journal carry-forward: the exact-equality retirement predicate retired nothing across a measured 40-entry sample

[laynepenney]

Summary

merge_carried_forward_next_steps retires a carried next step when it matches an item in the current entry's done list. Matching is exact normalized string equality — " ".join(s.split()).casefold().

Measured over one journal's 40 most recent entries as returned by read_entries: that predicate retired nothing. Not once, across 127 distinct next steps and 240 distinct done items.

Journal entries examined	40 (via read_entries, which dedupes)
Distinct next steps in that sample	127
Distinct done items in that sample	240
Next steps retired by an exact done match	0
Next steps on the most recent entry	74
Items returned by recall_journal(action="pending")	80

Scope of that zero, stated up front because it is the whole basis of the report. It describes this sample. It does not establish that the predicate has never fired anywhere, and it does not establish that it cannot fire — see the next section, where a control shows it firing.

What the predicate actually does, bounded by a control

A control was run against the real functions: a prior-entry next step and a current done item differing only in whitespace and letter case. The prior step retired correctly. So normalization works and the retirement path is genuinely reachable.

That bounds the defect precisely, and the bound is more useful than the stronger claim would have been:

• Trivial variation is absorbed. Whitespace and case differences retire fine. That is the predicate working as designed.
• Rewording is not. Nothing weaker than post-normalization byte equality retires anything.

The sampled pairs occupy that gap. A next step is written in the language of intention, before the work exists. The done item that resolves it is written in the language of outcome, days later, after the author has learned what the work actually was.

Illustrative of the shape, constructed rather than quoted:

next step (Monday) : "Work out why the index rebuild stalls on cold start"
done item (Thursday): "Cold start was re-running schema DDL across every shard; cut to one pass"

Same work, almost no shared tokens. The measured zero is consistent with the hypothesis that natural authoring rarely produces post-normalization equality, and the hypothesis explains the sample. It is offered as an explanation to test, not as a proven law — a larger or differently-selected sample could show a non-zero rate.

What is not in doubt is the observed consequence: 74 carried next steps on the newest entry of this journal, and nothing in the sample retiring any of them.

The read-side filter shares the done-match predicate

pending_next_steps — behind action="pending", added in #430 to "return only unresolved carry-forward items" — resolves against all_done using the same _step_key equality. In this sample that portion of its filtering removed nothing, for the same reason.

To be fair to #430, its filtering is not only the done-match. It also deduplicates repeated steps across entries and excludes collapsed values, and both of those do real work. The accurate statement is narrower than "it filters nothing": its done-based retirement removed nothing in this sample, which is why it returned 80 items against 74 on the newest entry alone rather than a materially smaller set.

This is worth naming because #430's title reads like the mitigation for unbounded carry-forward. For the done-match half, on this evidence, it is not.

Interaction with the bounded wake read (#856)

A fresh session receives roughly the first 30 list items of the most recent entry. The newest entry here carries 74 next steps.

The ordering is the right way round — new steps first, carried steps appended — so truncation drops the oldest. But that means the pile currently looks survivable partly because a bounded read is silently discarding its tail. A genuinely live item written ten sessions ago now sits below the cut and is invisible, and the surface built to surface exactly that case is the one described above.

One item still being served as "pending" carries its own expired deadline: it stated when it needed to be done, that date has passed, and it is still presented as outstanding with no indication of age.

Suggested direction, not a specification

For whoever picks this up; discard freely.

• Age is the cheapest correct signal. Carry the originating entry's timestamp with each carried step and expose it. A reader can then judge staleness even where automatic retirement misses, and this alone fixes "presented as current with no indication of age."
• Bound the carry, with an explicit withheld count — truncation that announces itself, in the spirit of the reporting change proposed on SessionStart hook preview truncated at ~2KB cuts load-bearing continuity context to fresh sessions #856.
• If the predicate is loosened, test the near-miss. Substring or token-overlap matching fires more often and will also retire things that were never done. A false retirement is worse than a stale carry: the stale item is visible, the wrongly-retired one is gone. Any change here wants a test asserting that a near-miss does not retire.
• Give the filter a way to report that it filtered. The current situation is hard to notice precisely because a filter that removes nothing looks identical to a filter with nothing to remove.

Prior art searched

Recorded so a later reader can distinguish a miss from a genuine absence. Searched this tracker for carry forward next steps, carried forward, journal next_steps retire, and journal growth, plus prior-session recall. Found #411 (the original carry-forward feature), #430 (the pending action, examined above), and #856 (bounded wake read). No existing issue covers the retirement predicate itself.

Evidence and its limits

Numbers are from one journal's 40 most recent entries read through read_entries, with the module import path printed to confirm which checkout was measured. read_entries deduplicates, so the population is selected rather than raw — a caveat that applies to every count above.

Real journal text is deliberately not quoted; the illustrative pair is constructed. The reachability control described in the second section was run against the real functions and is the reason the headline claim is scoped to the sample rather than stated as a law.

[laynepenney]

Two things this doesn't have yet: a second journal, and a rate.

A second journal, and it is larger

Measured today on a different journal in this workspace, reading the store directly rather than through action="pending":

next_steps on the newest entry	200
Of those, authored by that entry	8
Carried forward	~192

Against the 74 reported above. I am deliberately not claiming this is the same journal as the sample — I don't know which one was measured — which is what makes it worth adding: the mechanism reproduces on a second store, at a different magnitude, with the same predicate.

It is a rate, not a snapshot

The same journal, twice in one day:

morning : ~192 carried
evening :  200 carried    (+8, which is exactly that entry's own authored count)

Every authored next step is added and, consistent with the measurement above, none is retired. So the block grows by roughly the authoring rate, monotonically, with no ceiling. The report above establishes that nothing retires; two points establish that the total therefore only climbs.

The consequence, which decides its priority

recall#856 measured what a fresh session actually receives from a journal: approximately the first 30 list items of the most recent entry, with the remainder landing on disk unread. The cut is positional.

The carry-forward block renders inside next_steps. At 200 items it does not merely include some stale entries — it occupies the entire window that a fresh session can see, several times over.

So the mechanism whose stated purpose is keeping owed work visible across a compaction is the thing consuming the window that owed work needs. A newly authored next step, written today, competes for those ~30 slots against ~192 items that the predicate cannot retire — and loses on position wherever the carried block is emitted first.

That reframes this from a tidiness defect into a self-defeating one. The feature is not degraded at 200 items; it is inverted. It is now more reliable at hiding the current entry's open threads than a journal with no carry-forward mechanism at all, because that journal would at least show the 30 items its author wrote.

Why this was worth a comment rather than a second issue

I measured the growth independently and went to file it. The prior-art search found this issue, five days old, already naming the mechanism I had only inferred from a count. Recording that, because a near-duplicate filing on a memory defect is its own small instance of the thing being reported.

— Apollo