trace: bound the temporal history and stop arming it on dead code (#827)

[InauguralPhysicist]

The bug

The per-name assignment history behind prev of x, <kw> is x at <line> and
state_at was append-only, uncapped, and held a reference to every value it
recorded. Any long-running program that so much as mentioned prev of grew
linearly until the machine died — and a whole-program compile scan armed it even
when the prev of sat in a function that was never called.

This froze a 4 GB box: ~20 minutes of sustained pressure with no OOM kill —
the kernel thrashed instead of killing, so the machine had to be power-cycled.
The failure mode is a machine freeze, not a crash.

LeakSanitizer never saw a byte of it. Every allocation was reachable from the
history table and correctly freed at exit. This was unbounded retention, not
a leak — which is why it survived a zero-leak-tally gate for its whole life.

Measured (peak RSS, /usr/bin/time -f %M, under ulimit -v)

Repro — the prev of is in a function nothing calls, and the loop uses no
temporal semantics at all:

define never_called(v) as:
    return prev of v
x is 0
i is 0
loop while i < N:
    x is i * 1.5
    i is i + 1

program	N=200k	N=800k	N=1.6M
no temporal query (floor)	2944	2944	2944
dead-code prev of — before	9088	28032	53120
dead-code prev of — after	2944	2944	2944
live prev of x — before	27904	103040	203008
live prev of x — after	2944	2944	2944
live what is x at L — before	9216	28032	53248
live what is x at L — after	2944	2944	2944

Every case now sits on the no-temporal-query floor and is flat in iteration
count. Also ~30% faster on the live prev of loop (0.45s → 0.31s, n=3) — it
stopped reallocating an ever-growing array.

The design, and why

Two independent defects. Neither fix changes any answer.

(B) Unbounded retention — the half that freezes machines

The key observation is that a backward query returns the latest assignment
whose line is <= L — a temporal walk. That makes most recorded entries
provably unreachable:

entry i is dead ⟺ some later entry j has line[j] <= line[i]

(any L that admits i also admits j, and j wins for being later). What
survives are the strict suffix minima of the line sequence — so the live
entries are sorted by line and can never outnumber the distinct source lines
that assign that name. Bounded by program TEXT, not by runtime. A loop
reassigning one name a billion times keeps one entry and pins one value.
Maintenance is one pop-while at append.

Two facts pruning would otherwise lose are carried explicitly, which is exactly
why the answers are identical:

• prev of x at L wants the value of the assignment immediately preceding
  (in execution order) the one that answers L — that predecessor is usually
  a pruned entry, so every live entry stores its own.
• when is x at L counts assignments with line <= L, pruned ones
  included. Counting is order-independent, so a per-name (line -> count)
  histogram carries it exactly — also bounded by distinct assigning lines.

The observer snapshot for where/why/how at L moved inside the live entry
(trace_record_obs always targets the newest entry, which is always live).
Backward queries became a binary search over the sorted live array, retiring
the periodic line-floor segment index — that index existed only to make scanning
an unbounded array survivable.

What I rejected:

• A cap / ring buffer. The smallest fix, but it is a semantic change: history
  older than N becomes unavailable. Unnecessary — the structural bound is exact.
• Keying the history by line. Wrong, and silently so. Assign at line 12, then
  at line 5, then ask at L=15: the answer is the line-5 value, because that
  assignment happened later. A line-keyed map answers line-12. This is pinned as
  an explicit test.
• Collapsing runs of consecutive same-line assignments. Correct but
  insufficient — it does not bound the alternating-lines case (12,13,12,13,…).
  The suffix-minima rule bounds both; both shapes are pinned as tests.
• Not pinning values (the issue's direction 3). Moot once the entry count is
  bounded — a handful of entries pin a handful of values.

(A) Whole-program arming — the dead-code half

g_trace_hist was set by a source scan, so one prev of v in unreachable code
armed recording for every name in the program. Both history-reading forms
(prev of x, <kw> is x at L) compile to a NAMED opcode carrying a
compile-time identifier, so the set of names a temporal query can reach is
exactly known. The compiler now arms only those names; an assignment to any
other name records nothing. Sound by construction — a name no query names cannot
be asked about.

Three things still force the wildcard, because they can reach names the compiler
cannot enumerate: state_at (queries every name), an open tape
(EIGS_TRACE / embed sink), and turning recording on without naming a name (the
REPL, record_history of 1). Arming only ever widens within a session. No
site outside trace.c writes g_trace_hist any more, so the flag can't be set
without arming something.

Semantics: unchanged. Tape: untouched.

• tests/test_temporal_pruning.eigs (22 checks) pins the four cases a naive
  prune silently breaks — the backward-line-jump counterexample, prev of x at L
  when the predecessor was pruned, when counting pruned assignments, and
  alternating lines. It passes on the pre-fix binary too, which is the point:
  it is a semantics pin, not a regression test.
• A 200-seed differential fuzz against the pre-fix binary — every query form
  (what/who/when/where/why/how/prev, with and without at, plus
  state_at), over programs with backward line jumps and loop re-assigns —
  reports zero divergences, JIT on and EIGS_JIT_OFF.
• Tape unchanged (Design: trace-tape format versioning — the tape is becoming a persisted artifact #411): no version bump. A records are written by
  trace_assign independently of the history table, one per assignment as
  before, and an open tape arms every name anyway. Verified: same program, old
  vs new binary → byte-identical tapes (668 records, A=207, L=358, S=101, N=1
  on both); record→replay→byte-diff clean under EIGS_REPLAY_STRICT=1 on both
  tiers; and a pre-fix tape replays byte-identically on the fixed binary.
  This was a retention bug, not a format one.
• eigsdap unaffected — it reads the tape, not the history table. make dap +
  DAP suite 30/30, including the trajectory checks.

The regression gate

tests/test_temporal_memory.sh (suite [70d]): peak RSS at two iteration
counts 8x apart for three programs (dead-code prev of, live prev of, live
at), asserting both a ceiling and flatness, under ulimit -v so a
regression on a small box fails the test instead of taking the box down.

Planted-fault validated, twice:

• Against the pre-fix binary: 0 passed, 6 failed (live 203136 kB,
  +175232 kB over 8x).
• Against an answer-preserving but unbounded prune (> line instead of
  >= line, keeping same-line duplicates): every temporal answer stays correct
  and the differential fuzz stays green, but live goes 13952 → 90496 kB and
  this gate goes red. That fault is exactly why the memory gate is a separate
  file from the semantic one.

The semantic tests were validated the same way — the fuzzer catches a naive
prev of x at L (reads the neighbouring live entry) and a when counted off
the pruned array, both in 3 of 40 seeds.

Gates

gate	result
release suite	3541/3541, 0 failed
ASan+UBSan, detect_leaks=1	3539/3539, 0 failed — leak tally still 0
make dap + DAP suite	30/30
tests/test_replay.sh	24/24
tests/test_trace_on_fail.sh	7/7
make freestanding-check	OK stage 1 + stage 2
make jit-smoke	all cases passed
tools/embed_stack_soak.sh	PASS
--lint	clean (only the pre-existing W021 hint the sibling test_temporal.eigs also carries)

ASan-stressed the prune path specifically with heap values (lists/dicts/strings
reassigned through same-line, alternating-line and backward-jump shapes) — clean,
no UAF, no leaks.

Consumer follow-up needed: dynamics

dynamics works around this bug today: orbit.eigs:707 calls
record_history of 0 for the lifetime of a lab window session, and its memory
gate tests/test_bif_mem.sh:205 carries a planted fault that removes that
call and must go red. Once this lands, record_history of 0 is no longer
load-bearing — so that planted fault will stop failing and the gate needs
rework alongside dropping the workaround.

That work belongs to the rung-1 UI ladder rooted at
InauguralSystems/dynamics#20, whose memory-gated slice is
InauguralSystems/dynamics#23 (the bifurcation sweep view). Filing the successor
issue there is the follow-up; it should not block this fix.

Closes #827

Closes #827

🤖 Generated with Claude Code

[InauguralPhysicist]

Consumer follow-up filed: InauguralSystems/dynamics#24 — drop the record_history of 0 opt-out and re-point planted fault 1 in tests/test_bif_mem.sh, which stops discriminating once this lands. Blocked on this PR + a release cut.

[Copilot]

Pull request overview

Fixes a major temporal-trace retention issue in the runtime by making per-name assignment history bounded (reachability-pruned) and by preventing history recording from being armed by unreachable/dead code. This keeps long-running programs using (or merely containing) temporal queries from growing RSS linearly.

Changes:

• Implement reachability-pruned per-name history in src/trace.c, plus per-name (vs whole-program) arming for temporal history recording.
• Add new regression gates: a semantics pin test (test_temporal_pruning.eigs) and a process-RSS boundedness test (test_temporal_memory.sh), and wire them into the full test runner.
• Update trace documentation and changelog to reflect the new behavior and complexity model.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/test_temporal_pruning.eigs	New line-number-sensitive semantics pin ensuring pruning preserves temporal answers (including backward-jump counterexample).
tests/test_temporal_memory.sh	New peak-RSS ceiling + flatness gate to prevent unbounded retention regressions.
tests/run_all_tests.sh	Hooks the new [70c]/[70d] gates into the main test runner.
src/trace.h	Documents new arming entry points and updated complexity/semantics notes for temporal queries.
src/trace.c	Core implementation: bounded history via pruning, per-name arming set, histogram for when, and binary search for backward queries.
src/repl.c	REPL now arms wildcard history recording via the new arming API.
src/compiler.c	Compiler arms history per targeted identifier for NAMED temporal queries; wildcard for cases that can’t be enumerated.
src/builtins.c	record_history now uses the new arming/disable entry points.
docs/TRACE.md	Updates documentation for per-name arming and bounded/pruned history behavior.
CHANGELOG.md	Adds an Unreleased entry describing the fix and new test gates.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[InauguralPhysicist]

Second commit (121ce4f) fixes two defects in the first, one found by review and one by CI.

1. Use-after-free under spawn. The per-name arming set added here is process-global and grown with realloc by the compiler. Single-threaded that is fine — compile, then run — but a worker calling eval/load_file compiles concurrently with other workers recording assignments, so the realloc lands under a reader walking the array. That is a UAF, not the benign torn-int race g_trace_hist already had, and nothing in the suite or CI's TSan leg exercises eval-on-a-worker with a temporal query, so it would not have been caught.

spawn now widens to the wildcard as its last single-threaded act, before the first pthread_create, so the value is published to every worker by the same happens-before #297 relies on; from then on the filter reads two ints and the name array is never touched again. It uses a separate entry point that does not set g_trace_hist — a program with no temporal query must not start recording just because it made a thread. Verified both directions: a spawning program with no temporal query stays at the 2944 kB floor and flat from 200k→1.6M iterations, and a spawning program that does use prev of/at/when still answers correctly.

Giving up the narrowing under MT costs nothing that matters — the history is bounded either way; the narrowing is a per-assign CPU optimization for the single-threaded long-running programs #827 was about.

2. macOS legs (the red ones above). The new RSS gate is Linux-only and must SKIP, not fail: BSD /usr/bin/time has no -f/%M and ulimit -v is a no-op there. It now probes the exact invocation rather than the platform name — a Linux box without GNU time skips for the same reason — and additionally requires /proc, since the thresholds are calibrated against Linux VmRSS accounting. Same platform reasoning as test_http_rss_growth.sh.

Validated in both directions, because a gate that skips when it shouldn't is a dead gate: against a stub mimicking BSD time it prints SKIP and exits 0; on Linux it still runs all 6 checks; and a genuinely missing binary still FAILS loudly rather than skipping.

All gates re-run on the final tree: release suite 3541/3541; ASan+UBSan detect_leaks=1 3539/3539 with the leak tally still 0 and [70d] correctly skipping; make dap + DAP 30/30; freestanding-check; jit-smoke; embed_stack_soak.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated no new comments.

Suppressed comments (3)

src/trace.c:293

• lc_bump returns early on realloc failure without recording anything, which makes when is x at L silently under-count and can stay wrong even if later allocations succeed. It would be safer to mark the histogram as disabled on OOM and have when ... at return a miss (null) rather than an incorrect count.

/* Bump the (line -> count) histogram for `when is x at L`. Sorted insert;
 * after the first few assigns this is a pure binary-search hit. */
static void lc_bump(PrevEntry *e, int line) {
    int lo = 0, hi = e->lc_count - 1;
    while (lo <= hi) {
        int mid = (int)(((unsigned)lo + (unsigned)hi) >> 1);
        if (e->lc[mid].line == line) { e->lc[mid].count++; return; }
        if (e->lc[mid].line < line) lo = mid + 1;
        else hi = mid - 1;
    }
    if (e->lc_count >= e->lc_cap) {
        int nc = e->lc_cap ? e->lc_cap * 2 : 8;
        LineCount *nl = realloc(e->lc, (size_t)nc * sizeof(LineCount));
        if (!nl) return;   /* `when at L` under-counts rather than aborting */
        e->lc = nl;
        e->lc_cap = nc;
    }

src/trace.c:446

• If the per-name (line -> count) histogram is unavailable (e.g. due to OOM while growing it), trace_query_at currently still returns a numeric count (possibly 0) which is indistinguishable from a real count and can be silently wrong. Consider returning 0 (miss) so the VM produces null instead of an incorrect number when exact counting cannot be maintained.

    if (kind == 2) {
        /* `when is x at L` — count of assignments with line ≤ L. Summed
         * from the histogram, which counts pruned assignments too (#827). */
        long long count = 0;
        for (int i = 0; i < e->lc_count && e->lc[i].line <= line; i++)
            count += e->lc[i].count;
        *out = slot_from_num((double)count);
        return 1;
    }

tests/run_all_tests.sh:2838

• If test_temporal_memory.sh exits non-zero before emitting a parsable TEMPORAL_MEM: summary line, this block will log a FAIL but may not increment FAIL, so the overall suite can still report success. Treat a non-zero TMEM_RC with no parsed failures as at least one failing check.

if [ "$TMEM_RC" -ne 0 ]; then
    echo "  FAIL: temporal history memory gate (rc=$TMEM_RC)"
fi