trace: bound the temporal history and stop arming it on dead code (#827)

CHANGELOG.md

@@ -2,6 +2,65 @@
 
 All notable changes to EigenScript are documented here.
 
+## [Unreleased]
+
+### Fixed
+
+- **The temporal assignment history is bounded, and it no longer arms on
+  dead code (#827).** The per-name history that backs `prev of x`,
+  `<kw> is x at <line>` and `state_at` was append-only, uncapped, and held
+  a *reference* to every value it recorded. Any long-running program that
+  so much as mentioned `prev of` grew linearly until the machine died —
+  `dynamics`' orbit lab retained 3.9 MB per rendered frame and hit 859 MB
+  by frame 300; the minimal repro froze a 4 GB box for ~20 minutes with no
+  OOM kill (power-cycle to recover). LeakSanitizer never saw a byte of it:
+  every allocation was reachable from the history table and freed at exit,
+  so this was unbounded *retention*, not a leak. Two independent defects,
+  both fixed, with **no change to any temporal answer**:
+  - **Unbounded retention.** The history is now pruned at append time,
+    because a backward query makes most entries provably unreachable:
+    entry `i` is dead exactly when some later entry `j` has
+    `line[j] <= line[i]` (any `L` that admits `i` admits `j` too, and `j`
+    wins for being later). What survives are the strict suffix minima of
+    the line sequence, so the live entries are line-sorted and can never
+    outnumber the distinct source lines that assign that name — bounded by
+    program *text*, not by runtime. A loop reassigning one name a billion
+    times now keeps one entry and pins one value. Two facts pruning would
+    otherwise lose are carried explicitly so the answers are identical:
+    each live entry stores its own execution-order predecessor (which is
+    what `prev of x at L` returns, and it is usually a pruned entry), and
+    a per-name `(line -> count)` histogram carries `when is x at L`, which
+    counts pruned assignments. Backward queries became a binary search
+    over the sorted live array, retiring the periodic line-floor segment
+    index that existed only to make scanning an unbounded array
+    survivable. Measured, 1.6M iterations: a live `prev of` went
+    203,008 kB -> 2,944 kB, a live `at` query 53,248 kB -> 2,944 kB, both
+    now flat in iteration count and equal to the no-temporal-query floor.
+  - **Whole-program arming.** `g_trace_hist` was set by a source scan, so
+    a `prev of v` inside a function nothing ever called switched on
+    recording for every name in the program. Both history-reading forms
+    compile to a NAMED opcode carrying a compile-time identifier, so the
+    reachable name set is exactly known: the compiler now arms only those
+    names, and assignments to any other name record nothing. `state_at`
+    (it queries every name), an open tape, and turning recording on
+    without naming a name (the REPL, `record_history of 1`) still arm the
+    wildcard. The dead-code repro went 53,120 kB -> 2,944 kB.
+
+  **Semantics are unchanged and the tape is untouched.** The pruning drops
+  only entries no query could reach, which is why
+  `tests/test_temporal_pruning.eigs` — including the backward-line-jump
+  counterexample that a line-keyed table gets wrong — passes on the
+  pre-fix binary too, and why a 200-seed differential fuzz of every query
+  form (`what`/`who`/`when`/`where`/`why`/`how`/`prev`, with and without
+  `at`, plus `state_at`) against the pre-fix binary shows zero
+  divergences on both execution tiers. `A` records are written
+  independently of the history table, so tapes recorded before and after
+  are byte-identical and a pre-fix tape replays byte-identically on the
+  fixed binary — no format-version bump (#411). New gates: suite [70c]
+  (semantics) and [70d] (`tests/test_temporal_memory.sh` — peak RSS
+  ceiling *and* flatness across an 8x iteration range, which goes red on
+  the pre-fix binary and on an answer-preserving-but-unbounded prune).
+
 ## [0.35.0] - 2026-08-02
 
 ### Added

docs/TRACE.md

@@ -289,6 +289,18 @@ in-run time travel.
   *first* temporal query starts recording at that point, so assigns
   executed earlier are not visible to it. Aliasing `state_at` through
   a dict or eval-built string also hides it from the compiler's scan.
+- The gate is **per name**, not whole-program (#827). Both history-reading
+  forms — `prev of x` and `<kw> is x at L` — compile to a NAMED opcode
+  carrying a compile-time identifier, so the set of names a temporal query
+  can ever reach is known exactly, and assignments to any other name record
+  nothing. This is what stops a `prev of v` sitting in a function nothing
+  ever calls from taxing every assignment in the program. Three things
+  force the wildcard instead, because they can reach a name the compiler
+  cannot enumerate: `state_at` (it queries every tracked name), an open
+  tape (`EIGS_TRACE` or an embed sink), and turning recording on without
+  naming a name (the REPL, `record_history of 1`). Arming only ever widens
+  within a session — a name armed mid-run by `eval` starts recording from
+  that point, the same edge the whole-program gate already had.
 - When the compiled program contains a `where`/`why`/`how ... at`
   query, each history entry also stamps an observer snapshot
   (entropy, dH) at assign time, so the observer-derived
@@ -297,14 +309,42 @@ in-run time travel.
   no such query in the program, no per-assign cost.
 - `state_at of line` walks every tracked name's history backward and
   returns a dict of each binding's value at or before `line`.
-- Backward queries (`at`, `state_at`) are pruned by a periodic
-  line-floor index: each 64-entry segment of a name's history caches
-  its minimum line stamp, so segments that cannot contain a hit are
-  skipped in one compare. Loop-heavy histories — thousands of assigns
-  stamped with the same few lines, the debugger-scrub worst case —
-  resolve in O(history/64) instead of O(history). The index adds one
-  `int` per 64 history entries and an O(1) min-update per assign.
-- Per-assign cost of the history: one cache line + a pointer compare.
+- **A backward query is TEMPORAL, not line-keyed.** `<kw> is x at L`
+  returns the value from the most recent assignment whose line is `<= L`
+  — which is *not* "the value at the greatest line `<= L`". Assign at
+  line 12, then at line 5, then ask at L=15: the answer is the line-5
+  value, because that assignment happened later. Any representation that
+  keys the history by line answers the line-12 value and is wrong.
+- **The history is bounded by the program TEXT, not by runtime** (#827).
+  It used to be append-only and uncapped, holding a reference to every
+  value ever assigned: a program that merely mentioned `prev of` grew
+  linearly until the machine died. It is now pruned at append time, with
+  no change to any answer, because most entries are provably unreachable:
+
+      entry i is dead  <=>  some later entry j has line[j] <= line[i]
+
+  (any `L` that admits `i` also admits `j`, and `j` wins for being later).
+  What survives are the strict suffix minima of the line sequence, so the
+  live entries are sorted by line and can never outnumber the distinct
+  source lines that assign that name. A loop that reassigns one name a
+  billion times keeps ONE entry — and pins one value instead of a billion.
+  Two facts that pruning would otherwise lose are carried explicitly, so
+  the answers are identical: each live entry stores its own
+  execution-order predecessor (`prev of x at L` wants a value that is
+  usually pruned), and a per-name `(line -> count)` histogram carries
+  `when is x at L`, which counts pruned assignments too.
+  **Nothing about the tape changed**: `A` records are written by
+  `trace_assign` independently of the history table, one per assignment
+  as before, and an open tape arms every name anyway. Tapes recorded
+  before and after #827 are byte-identical, so no format-version bump
+  (#411) — this was a retention bug, not a format one.
+- Backward queries (`at`, `state_at`) are therefore a binary search over
+  a line-sorted array — `O(log D)` where `D` is the number of distinct
+  assigning lines. This replaced the periodic line-floor segment index,
+  which existed only to make scanning an unbounded array survivable.
+- Per-assign cost of the history: one cache line + a pointer compare,
+  plus the pop-while that retires the entries the new assignment kills
+  (amortized O(1) — an entry is pushed once and popped once).
 - **The history is per-thread; the tape is per-process** (#739). The
   history table is keyed by *interned name pointer*, and the intern
   table lives on `EigsThread`, so two threads' `x` were never the same

src/builtins.c

@@ -3046,8 +3046,10 @@ Value* builtin_record_history(Value *arg) {
     }
     int prev = g_trace_hist;
     int on = (arg->data.num != 0.0) ? 1 : 0;
-    g_trace_hist = on;
-    g_trace_obs_hist = on;
+    /* #827: no name to narrow on — a self-hosted compiler calling this is
+     * standing in for the whole-program arming, so it gets the wildcard. */
+    if (on) { trace_arm_history_all(); g_trace_obs_hist = 1; }
+    else    trace_history_disable();
     return make_num((double)prev);
 }
 
@@ -3524,6 +3526,18 @@ Value* builtin_spawn(Value *arg) {
      * READ it (the value is already published to all workers via the first
      * write's happens-before through their pthread_create). */
     if (!g_vm_multithreaded) g_vm_multithreaded = 1;
+    /* #827: #739's per-thread history is filtered by a PROCESS-global armed-name
+     * set that the compiler grows. Single-threaded that is fine (compile, then
+     * run), but a worker calling eval/load_file compiles concurrently with other
+     * workers recording assignments — a realloc of the name array under a
+     * reader is a use-after-free, not just a torn read. So the last
+     * single-threaded act before the first spawn is to widen to the wildcard,
+     * permanently: from here the filter reads only the two ints (the same
+     * benign shape as g_trace_hist itself) and the name array is never touched
+     * again. Costs nothing that matters — the history is bounded either way
+     * now; the narrowing is a per-assign CPU optimization for the
+     * single-threaded long-running programs #827 was actually about. */
+    trace_arm_history_all_mt();
     int pc_rc = pthread_create(&h->tid, NULL, thread_entry, h);
     if (pc_rc != 0) {
         /* The thread never started. Returning a live-looking handle here

src/compiler.c

@@ -1908,7 +1908,7 @@ static void compile_node_inner(Compiler *c, ASTNode *node) {
          * seen; recording then starts at the aliasing program's own
          * temporal queries, or never. Documented in TRACE.md.) */
         if (strcmp(node->data.ident.name, "state_at") == 0)
-            g_trace_hist = 1;
+            trace_arm_history_all();   /* #827: state_at queries every name */
         /* Try local slot resolution for params (fast path) */
         if (c->enclosing) {
             uint32_t h = node->name_hash;
@@ -2842,9 +2842,18 @@ static void compile_node_inner(Compiler *c, ASTNode *node) {
         ASTNode *at_expr = node->data.interrogate.at_expr;
 
         /* `prev of x` and every `at <line>` form answer from the
-         * per-assign history — enable recording. */
-        if (kind == 6 || at_expr)
-            g_trace_hist = 1;
+         * per-assign history — enable recording. #827: arm only the NAME
+         * this query can reach. Both history-reading forms compile to a
+         * NAMED opcode carrying a compile-time identifier, so the reachable
+         * set is exact; a non-ident operand never reads the history at all
+         * (bare OP_INTERROGATE) but arms the wildcard anyway — widening is
+         * the safe direction. */
+        if (kind == 6 || at_expr) {
+            if (expr && expr->type == AST_IDENT)
+                trace_arm_history_name(expr->data.ident.name);
+            else
+                trace_arm_history_all();
+        }
 
         if (at_expr && expr && expr->type == AST_IDENT) {
             /* `<kw> is x at <expr>` — operand value is not needed; only

src/repl.c

@@ -585,7 +585,9 @@ static void repl_interactive(Env *env) {
      * by line — `x is 5` would record nothing and a later `prev of x` finds
      * no history. Interactive sessions record from the start (the piped
      * path is left untouched: byte-identical output is its contract). */
-    g_trace_hist = 1;
+    /* #827: a REPL line can name any binding assigned by an earlier line,
+     * so the narrow per-name arming cannot apply here — wildcard. */
+    trace_arm_history_all();
     g_trace_obs_hist = 1;
 
     hist_load();