diff --git a/.changeset/soft-agents-discover.md b/.changeset/soft-agents-discover.md new file mode 100644 index 000000000..3e7c8eb84 --- /dev/null +++ b/.changeset/soft-agents-discover.md @@ -0,0 +1,9 @@ +--- +"@sapiom/harness": minor +--- + +Discover markerless `defineAgent` and legacy `defineOrchestration` projects plus literal direct source invocations with bounded syntax-only analysis, conservative reconciliation, and live rail/system-graph updates without importing, bundling, type-checking, or executing project code. Public direct invocation edges use `basis: "static-invocation"`. Markerless agents inside nested Git repositories are discovered when that repository is selected directly; package-wide output-to-input data flow remains a separate evidence provider. + +**Breaking:** Public system-graph invocation edges that previously carried `basis: "static"` now carry `basis: "static-invocation"`. + +**Migration:** Consumers that validate or deserialize `GET /api/workspaces/:workspaceKey/system-graph` responses must accept `"static-invocation"` as the invocation-edge `basis` value before upgrading. diff --git a/packages/harness/docs/agent-discovery.md b/packages/harness/docs/agent-discovery.md index 859267ec7..74db04b7a 100644 --- a/packages/harness/docs/agent-discovery.md +++ b/packages/harness/docs/agent-discovery.md @@ -6,21 +6,31 @@ shows the wrong number of agents: 1. [How is an agent discovered?](#1-how-is-an-agent-discovered) 2. [What happens when I create a new agent?](#2-what-happens-when-i-create-a-new-agent) 3. [How is it tracked afterwards, and how does an entry ever leave?](#3-how-is-it-tracked-afterwards) -4. [What are the bounds, what do they cost, and what is *not* scanned?](#4-the-bounds-what-they-cost-and-what-is-not-scanned) +4. [What are the bounds, what do they cost, and what is _not_ scanned?](#4-the-bounds-what-they-cost-and-what-is-not-scanned) 5. [What changed, and what did not](#5-what-changed-and-what-did-not) -Every claim below has a file and line beside it. If the code and this document -disagree, the code is right and this document is a bug. +Important implementation claims below name their owning module. If the code and +this document disagree, the code is right and this document is a bug. --- ## 1. How is an agent discovered? -### What marks a directory as an agent +### What proves a directory is an agent -One file: **`sapiom.json`**, directly inside the directory -(`src/shared/types.ts:52`, `AGENT_PROJECT_MARKER`). Nothing else — not -`package.json`, not a `defineAgent` call, not a naming convention. +Studio uses two ordered proofs: + +1. A valid **`sapiom.json`** directly inside the directory is authoritative. + It wins without reading TypeScript and stops descent below that agent. +2. When the marker is absent or invalid, a regular, non-symlink `index.ts` may + prove exactly one exported agent definition created by `defineAgent` or the + legacy `defineOrchestration` API. This fallback is syntax-only: Studio parses + TypeScript ASTs and never bundles, typechecks, dynamically imports, or + executes project code merely to discover it. + +An unresolved/dynamic export, ambiguous definition, transient read failure, or +exhausted analysis budget is **incomplete**, not "not an agent". Existing rows +under that uncertain envelope survive until a later scan can prove the answer. "Has a `sapiom.json`" is stricter than it sounds (`inspectAgentProjectMarkerSync`, `src/core/agent-project-discovery.ts:237`): @@ -40,7 +50,14 @@ momentary I/O failure is never mistaken for "this agent was deleted" — see Everything the marker carries — `definitionId`, `name`, `templateId`, `forkId`, `starterId` — is provenance written by `link`, `clone` or `scaffold`. None of it -affects *whether* the directory is an agent. +affects _whether_ a valid marker is authoritative. A syntax-only row has null +cloud metadata. If a connected row also has syntax proof, its source name is +canonical while its retained marker/cloud slug remains a compatibility alias. + +The syntax resolver follows only supported relative TypeScript re-exports and +aliases within the selected workspace. It never crosses ignored directories, +symlink components, or a nested repository checkout. Bare-package or otherwise +unresolved export structure fails closed rather than guessing. ### Who walks the tree @@ -54,11 +71,12 @@ It is **breadth-first**, and that is load-bearing rather than stylistic — see §4. At each directory it asks the caller what to do, and stops descending on three answers: -| It stops at | Because | -|---|---| -| a directory with a valid marker | that is the agent; agents do not nest | -| a directory whose marker is `unreadable` | the whole subtree is treated as opaque for this pass | -| a directory that is **its own repository checkout** | it belongs to a repo you did not ask about — see §4 | +| It stops at | Because | +| --------------------------------------------------- | ---------------------------------------------------- | +| a directory with a valid marker | that is the agent; agents do not nest | +| a directory with one syntax-proven exported agent | that is the agent; agents do not nest | +| a directory whose marker is `unreadable` | the whole subtree is treated as opaque for this pass | +| a directory that is **its own repository checkout** | it belongs to a repo you did not ask about — see §4 | and it never enters `node_modules`, `.git`, `.sapiom`, `dist`, `build`, `.next` (`IGNORED_DIR_NAMES`, `agent-project-discovery.ts:154`), nor any symlink — @@ -70,21 +88,23 @@ cycle terminate. **This is the part that actually decides what you see.** A scan only ever covers the tree beneath the root it is given, so the whole question is "who chose the -root". There are exactly six ways an agent can enter the registry, five of them -a walk: - -| Reason | Root | Fires when | -|---|---|---| -| `boot` | the launch directory | server start (`src/server/index.ts:1174`) | -| `session-create` | the new session's `cwd` | `POST /api/sessions` (`index.ts:1305`) | -| `workspace-change` | a live session's `cwd` | the workspace watcher saw the marker set change (`index.ts:889`) | -| `agent-linked` | that one agent's directory | a deploy wrote a `definitionId` into its marker (`index.ts:1418`) | -| `agent-moved` | the destination's parent | a rail drag moved an agent on disk (`index.ts:1468`) | -| `requested` | whatever folder you named | `POST /api/workflows/scan` — the **"Add all N"** button (`index.ts:1365`) | - -The sixth is `POST /api/workflows/connect` (`workflow-registry.ts:433`), which -registers exactly one path and does not walk at all — the **"Add workspace"** -button. +root". There are eight scan reasons, plus one direct connection mutation: + +| Reason | Root | Fires when | +| ------------------ | -------------------------- | ------------------------------------------------------------------------- | +| `boot` | the launch directory | server start (`src/server/index.ts:1174`) | +| `session-create` | the new session's `cwd` | `POST /api/sessions` (`index.ts:1305`) | +| `workspace-change` | a live session's `cwd` | the workspace watcher saw the marker set change (`index.ts:889`) | +| `agent-linked` | that one agent's directory | a deploy wrote a `definitionId` into its marker (`index.ts:1418`) | +| `agent-connected` | that connected directory | a manual connection settles its current marker/source evidence | +| `agent-moved` | the destination's parent | a rail drag moved an agent on disk (`index.ts:1468`) | +| `graph-refresh` | the selected Project root | first graph access, a source/inventory change, or explicit graph refresh | +| `requested` | whatever folder you named | `POST /api/workflows/scan` — the **"Add all N"** button (`index.ts:1365`) | + +`POST /api/workflows/connect` first mutates exactly one path — +the **"Add workspace"** button. It applies the same marker-first, syntax-only +proof and freshness rules, then its `agent-connected` scan reconciles and +publishes the accepted inventory. Every one of those logs a line naming its reason, its root, what it found and what it cost (`logAgentScan`, `index.ts:305`): @@ -102,24 +122,24 @@ filesystem archaeology session. ## 2. What happens when I create a new agent? -An agent is created by writing `sapiom.json` into a directory — by -`sapiom agents init`, by a gallery-template clone (`@sapiom/agent-core`'s -`clone.ts`), or by hand. **Nothing about creation talks to the registry.** The -registry finds out the same way it finds out about anything: something scans. +An agent can appear by writing a marker, or by adding a markerless `index.ts` +whose exports satisfy the syntax proof above. **Nothing about creation talks to +the registry.** The registry finds out the same way it finds out about +anything: something scans. ### With the studio running (the normal case) -1. Your coding agent (or you) writes the marker somewhere under the session's - `cwd`. +1. Your coding agent (or you) writes the marker or relevant TypeScript somewhere + under the session's `cwd`. 2. The **workspace watcher** for that session sees a filesystem event (`SessionWorkspaceWatcher`, `src/core/workspace-watcher.ts:182`). A raw event - only *arms* a check — it does not itself mean anything, because recursive + only _arms_ a check — it does not itself mean anything, because recursive `fs.watch` on macOS reports `rename` for ordinary content writes too. -3. After a **250 ms debounce** (`workspace-watcher.ts:50`) it recomputes a - fingerprint of the marker directories under the cwd - (`snapshotWorkspaceWorkflows`, `:139`) and fires only if that fingerprint - actually changed. Editing a file in an existing agent does not fire it; - adding, removing or renaming an agent does. +3. After a short debounce it recomputes a bounded asynchronous fingerprint of + markers, candidate `index.ts` files, and the accepted relative dependency + observations needed to revisit split definitions. A raw relevant event + fail-closes affected graph navigation immediately; the expensive fingerprint + and reconciliation stay off the event callback. 4. That calls `rescanWorkspaceForSession` (`index.ts:889`), which prunes dead paths, rescans the session's `cwd`, rewrites every open session's `harness-context.json`, and broadcasts `workflows.changed`. The SPA refetches @@ -129,13 +149,14 @@ registry finds out the same way it finds out about anything: something scans. (`apps/web/src/features/billing/agents/invoice-chaser`) inside a live session's cwd appeared in `GET /api/workflows` within about a second, with no restart. -On Linux, or if the watcher errors, the same check runs on a 2 s poll instead -(`workspace-watcher.ts:53`) — same fingerprint, same firing rule, slightly -slower. +On Linux, or if the watcher errors, the same check runs on a 2 s poll instead — +same fingerprint and reconciliation rules, slightly slower. Missing, +unreadable, symlink, and non-file observations remain distinct so a recovery or +confirmed deletion cannot be swallowed as "unchanged". There is one more trigger for the same case: a session's **first** transition to `running` runs one rescan unconditionally (`index.ts:995`). The watcher captures -the markers already present as its baseline and only fires on a *later* change, +the markers already present as its baseline and only fires on a _later_ change, so a session that starts in a folder where the agent already exists (a just-cloned template) would otherwise never trigger anything. @@ -149,13 +170,14 @@ workspace / Add all** at it. That is not a failure mode; it is the design in §4 ### If the new agent is its own git repository -It is still registered. The marker is inspected *before* the repository boundary -is considered (`stopsAtRepositoryBoundary`, `agent-project-discovery.ts:420`, is -reached only after `onDirectory` has already declined to stop), so -"one git repo per agent" works exactly as it did. What the boundary declines to -enter is a checkout that merely *contains* agents. Verified against a real -server: a scaffold that runs `git init` in its own new directory registers -within the same second as one that does not. +A valid marker is inspected _before_ the repository boundary is considered, so +a marker-backed agent in a nested checkout is still registered by the containing +workspace scan. Static source discovery deliberately does not enter a foreign +checkout: select a markerless agent repository as its own workspace to prove it. +If a source-only folder previously discovered by its parent later runs +`git init`, the next parent scan retires that parent-owned row until the new +repository is selected directly. This asymmetry keeps syntax reads confined to +the selected repository while preserving the established marker behavior. --- @@ -168,9 +190,11 @@ temp-file rename so a crash mid-write cannot tear it through a single write queue, so a concurrent scan and prune cannot interleave and drop entries (`enqueue`, `:288`). -Each entry records where it came from in its `source` field: `"scan"` (a walk -found it) or `"connect"` (you named it). That distinction is what protects a -folder you added by hand from being reconciled away by an unrelated scan. +Each public entry records where it came from in its `source` field: `"scan"` (a +walk found it) or `"connect"` (you named it). Private source-name, marker-proof, +canonical-path, observation, and completeness evidence lives in the registry +and accepted inventory sidecars; it is never serialized by `/api/state` or +`/api/workflows`. ### What adds entries @@ -180,7 +204,7 @@ folder you added by hand from being reconciled away by an unrelated scan. survive it. That is deliberate — a scan of one project must not delete the agents of another — but it is also why the registry is the union of every root ever scanned over the life of an install, and why an over-broad scan is -expensive in a way a single bad session is not. It is a file you have to *clean*, +expensive in a way a single bad session is not. It is a file you have to _clean_, not one that resets. ### What removes entries @@ -201,12 +225,12 @@ Three things, and they are deliberately narrow: - on a session workspace change and after an agent move (`index.ts:895`, `:1467`). 2. **Scan reconciliation** (`isCoveredByScan`, `:171`). A `"scan"`-sourced entry - whose marker is gone or has become invalid is dropped — but *only* if this + whose marker is gone or has become invalid is dropped — but _only_ if this scan can prove it would have looked there. Three things protect an entry from that proof: it sits deeper than the scan actually reached (`budget.envelopeDepth`), it sits beneath a subtree that was unreadable on this pass, or it sits **inside a repository checkout this scan declined to - enter** (`isBehindRepositoryBoundary`, `:208`). Without that last one, + enter** (`isProtectedByIncompleteScan`, `workflow-registry.ts`). Without that last one, opening `~/src` after having opened `~/src/some-repo` would delete that repo's agents. 3. **Removing a project in the rail**, which is a client-side concept: it drops @@ -219,7 +243,7 @@ Three things, and they are deliberately narrow: practice that is within 30 s of the next read, without a restart. (Verified on a real server: an agent deleted outside any watched session's cwd disappeared from `GET /api/workflows` 33 s later with no scan and no restart.) -- **Renamed or moved by hand** — this is a delete *and* a create. The old path +- **Renamed or moved by hand** — this is a delete _and_ a create. The old path is pruned; the new one is registered by whichever scan next covers it. A live session's `boundWorkflowPath` pointing at the old path is cleared rather than left dangling (`index.ts:911`). @@ -228,7 +252,7 @@ Three things, and they are deliberately narrow: destination is rescanned in one step (`index.ts:1465`). Note the asymmetry: **removal needs proof, registration does not.** That is the -right way round for not losing your work, and it is exactly why the *breadth* of +right way round for not losing your work, and it is exactly why the _breadth_ of a scan matters so much — see §4. --- @@ -253,17 +277,33 @@ deterministic outer envelope for reconciliation. ### Directories entered: `AGENT_PROJECT_SCAN_MAX_NODES = 10_000` (`:76`) The bound that governs cost. Per directory entered = one `lstat` + one `readdir`, -~22–25 µs warm at every depth, so cost is linear and predictable in *directories -entered* and that is what is bounded. +~22–25 µs warm at every depth, so cost is linear and predictable in _directories +entered_ and that is what is bounded. Past the budget a scan is incomplete. Because the walk is breadth-first, the budget degrades by **depth**: every level above the cut is complete, and the scan reports how far it got as `budget.envelopeDepth`. Nothing beyond that is reconciled away as missing. -The watcher's fingerprint gets a tighter budget of its own — -`AGENT_PROJECT_WATCH_MAX_NODES = 2_500` (`:90`) — because that walk is -synchronous and re-runs on a 250 ms debounce while you type. +The watcher walk is asynchronous and bounded over the same 10,000-directory +candidate envelope, so polling cannot permanently miss a candidate the scanner +would admit. Dependency observations have their own bounded, scope-confined +metadata projection; multiple sessions and graphs sharing a canonical root +reuse one watcher rather than multiplying filesystem walks. + +### Syntax analysis budgets + +Syntax fallback has independent deterministic limits: + +- relative re-export depth: 8; +- per candidate: 32 unique TypeScript modules and 1 MiB; +- per workspace scan: 2,000 unique TypeScript modules and 16 MiB; +- parsed-summary LRU: 10,000 entries. + +Warm cache hits avoid physical reads/parses but spend the same logical scan and +candidate budgets as cold reads. Shared modules spend the workspace budget once +and each candidate budget once. Hitting any limit degrades completeness and +protects unresolved prior rows; it never turns uncertainty into deletion. ### Repository boundary (`isForeignRepositoryRoot`, `:150`) @@ -274,34 +314,33 @@ that repo. This is the round-2 addition, and it is the one that changes what you see. Measured on one real install (macOS/APFS, warm cache, at the 10,000-node budget). -Cells are *agents registered / distinct names among them / directories entered / -wall clock*: +Cells are _agents registered / distinct names among them / directories entered / +wall clock_: -| root | before | after | -|---|---|---| -| `~/sapiom/wf-demo-testing` (a launch dir) | 10 / 10, 17 dirs, 0 ms | 10 / 10, 17 dirs, 0 ms | -| `~/sapiom` (a parent of it) | 88 / 65, 10,000 dirs, 239 ms — **truncated at depth 5** | 68 / 64, 408 dirs, 8 ms — complete | -| `~/sapiom/sapiom-js` (a monorepo) | 25 / **2**, 9,016 dirs, 233 ms | 2 / 2, 444 dirs, 8 ms | -| `~/sapiom/Sapiom` | 0 / 0, 10,000 dirs, 200 ms — **truncated** | 0 / 0, 5,408 dirs, 107 ms — complete | +| root | before | after | +| ----------------------------------------- | ------------------------------------------------------- | ------------------------------------ | +| `~/sapiom/wf-demo-testing` (a launch dir) | 10 / 10, 17 dirs, 0 ms | 10 / 10, 17 dirs, 0 ms | +| `~/sapiom` (a parent of it) | 88 / 65, 10,000 dirs, 239 ms — **truncated at depth 5** | 68 / 64, 408 dirs, 8 ms — complete | +| `~/sapiom/sapiom-js` (a monorepo) | 25 / **2**, 9,016 dirs, 233 ms | 2 / 2, 444 dirs, 8 ms | +| `~/sapiom/Sapiom` | 0 / 0, 10,000 dirs, 200 ms — **truncated** | 0 / 0, 5,408 dirs, 107 ms — complete | The `sapiom-js` row is the whole argument. Of the 25 agents a scan of that repo used to register, **24 were the same agent** — one e2e fixture, reachable once per git worktree under `.trees/`. Two were real. Reading the distinct-name column -down the table: the boundary barely changes how many *agents* a scan finds and -collapses how many *rows* it writes. +down the table: the boundary barely changes how many _agents_ a scan finds and +collapses how many _rows_ it writes. Raising the node budget makes this worse, not better: uncapped, `~/sapiom` is 141 agents across 83,969 directories in 6.8 s — and still only 73 distinct names. -### What is *not* scanned, stated plainly +### What is _not_ scanned, stated plainly - anything below a directory that already has a marker; - `node_modules`, `.git`, `.sapiom`, `dist`, `build`, `.next`, and anything under them; - anything reached only through a symlink; - anything more than 8 levels below the root; -- anything past 10,000 directories on one walk (2,500 for the watcher's - fingerprint); +- anything past 10,000 directories on one scan or watcher candidate walk; - **anything inside a git checkout that is not the one you pointed at.** The last is the only one that loses whole agents rather than duplicates. On the @@ -310,8 +349,8 @@ agent under `~/sapiom/Sapiom`. Each of them lives inside a checkout below the scan root, and each is registered the moment that checkout is itself the root: launch the studio there, open a session there, or name it to **Add workspace**. -That is the trade the user asked for in as many words — *"It's okay if we don't -fully scan."* The rule it buys is worth stating on its own: +That is the trade the user asked for in as many words — _"It's okay if we don't +fully scan."_ The rule it buys is worth stating on its own: > **An agent is in your registry because you opened its folder, created it, or > asked for a scan that covers it — never because a walk wandered into a @@ -339,15 +378,18 @@ It is worth being blunt about this, because the opposite belief is also wrong. Round 1 **raised** the depth cap from 3 to 8, and added two things beside it: a 10,000-directory budget (`AGENT_PROJECT_SCAN_MAX_NODES`) and `envelopeDepth`, the scan's own report of how deep it actually got, so reconciliation could only -delete rows the scan could prove it had looked for. The watcher got a tighter -2,500-directory budget. +delete rows the scan could prove it had looked for. Round 1 gave the synchronous +marker watcher a tighter 2,500-directory budget; syntax reconciliation later +made that watcher asynchronous and aligned its candidate coverage with the +10,000-directory scan envelope so polling cannot miss an otherwise discoverable +agent forever. So: **there is still a depth scan, and after round 1 it reached further than -before, not less far.** What round 1 changed was *what limits it* — directories +before, not less far.** What round 1 changed was _what limits it_ — directories entered rather than levels descended — and it made a truncated scan honest about being truncated instead of silently deleting what it had not reached. -What round 1 did **not** change was scan *breadth*. A scan still followed its +What round 1 did **not** change was scan _breadth_. A scan still followed its root wherever that root led, across as many unrelated repositories as fitted in the budget. That is what produced a registry of 88 agents spanning twelve top-level directories nobody had opened, and six copies each of four agents from @@ -375,16 +417,32 @@ six checkouts of one repo. those projects, not waiting for a scan to undo it. - Removal still requires proof of absence. An unreadable directory keeps its entry. -- Duplicate agent *names* are still possible and still legitimate: open two +- Duplicate agent _names_ are still possible and still legitimate: open two worktrees of one repo as two projects and you will see the same agent twice, because you asked for both. What the boundary stops is getting them without asking. Disambiguating two legitimately-open copies in the rail is the SPA's job, not the registry's. +### Syntax discovery and live reconciliation + +- Marker-first semantics remain intact. Only an absent or invalid marker falls + through to source; an unreadable marker keeps the subtree opaque. +- Markerless current and legacy authoring APIs now enter the same registry, + rail, graph, and revision-matched navigation lifecycle as linked agents. +- Discovery results and completeness publish as one accepted generation after + context staging. Watcher races, overlapping roots, stale scans, and reverse + browser responses cannot publish an older inventory over a newer one. +- Known inventory renders from memory immediately in a degraded graph while + background discovery and bounded direct invocation extraction settle. No ordinary + GET waits on a filesystem scan, project execution, or remote metadata fetch. +- Syntax-only/null-cloud rows never authorize automatic legacy Canvas or + manifest extraction. Existing explicit valid-marker or cloud-link evidence + retains that legacy path, with authorization rechecked at process launch. + ### The one thing this document cannot tell you A registry that was already polluted stays polluted. The boundary changes what -*future* scans register; it does not retroactively remove what an earlier +_future_ scans register; it does not retroactively remove what an earlier over-broad scan wrote. Rescanning the root that caused it will now reconcile the duplicates away — but only that root, and only if you point at it again. Otherwise, remove those projects from the rail, or delete diff --git a/packages/harness/docs/workspace-system-graph.md b/packages/harness/docs/workspace-system-graph.md index 808f6bbd2..efb632307 100644 --- a/packages/harness/docs/workspace-system-graph.md +++ b/packages/harness/docs/workspace-system-graph.md @@ -34,11 +34,14 @@ GET /api/workspaces/:workspaceKey/system-graph POST /api/workspaces/:workspaceKey/system-graph/refresh ``` -`GET` returns the current process-memory snapshot. A cold read waits for the -initial projection, concurrent cold reads share that build, and later reads -reuse it. `POST .../refresh` reruns registry prerequisites, requests a fresh -projection, waits for that attempt, and is the explicit recovery action after -an error. Both successful routes return `200` with a +`GET` returns the current accepted process-memory snapshot. On a cold read, +known inventory nodes and revision-matched navigation render immediately in a +degraded projection; bounded direct invocation extraction and background discovery +may publish a later revision. Concurrent reads share that work, and ordinary +reads never await a filesystem baseline or discovery scan. `POST .../refresh` +reruns registry prerequisites, requests a fresh projection, waits for that +attempt, and is the explicit recovery action after an error. Both successful +routes return `200` with a `SystemGraphSnapshot`: ```ts @@ -105,9 +108,30 @@ within a bounded loop. `SystemGraph` is path-free and has `kind: "system"`. Its scope repeats only the opaque key. Nodes contain an `id`, Project-scoped `agentKey`, and display -`label`. Edges are static `invokes` relationships with a `blocking` or `async` -mode. Blocking and asynchronous calls between the same pair remain distinct in -the JSON even when the UI groups them into one connector. +`label`. Public direct-invocation edges are explicit and extensible: + +```ts +interface StaticInvocationGraphEdge { + from: string; + to: string; + kind: "invokes"; + basis: "static-invocation"; + mode: "blocking" | "async"; +} + +type SystemGraphEdge = StaticInvocationGraphEdge; +``` + +Blocking and asynchronous calls between the same pair remain distinct in the +JSON even when the UI groups them into one connector. + +Direct invocation analysis is syntax-only. It never creates a TypeScript +`Program` or `TypeChecker`, and it never imports, bundles, or executes customer +code. The provider scans each inventoried agent source root separately for +literal calls. It does not inspect the provenance of invocation inputs, follow +agent outputs through formatter/helper/router code, or scan arbitrary workspace +router modules outside those roots. Cross-agent output-to-input analysis will +use a separate package-level evidence provider. Projection can remain useful while reporting warnings: @@ -115,28 +139,26 @@ Projection can remain useful while reporting warnings: | ----------------------------- | ------------------------------------------------------------------------------------------------- | | `unresolved-target` | A literal target does not resolve to an agent in the selected Project. | | `dynamic-target` | Source contains a call whose target cannot be proven statically. | -| `duplicate-edge` | The same mode-specific relationship was discovered more than once. | -| `projection-failed` | A relationship projection failed and the remaining graph was preserved. | +| `duplicate-edge` | The same mode-specific direct invocation was discovered more than once. | +| `projection-failed` | A direct invocation projection failed and the remaining graph was preserved. | | `duplicate-agent-key` | More than one contained agent proposed the same key; local fallback identities disambiguate them. | | `inventory-extraction-failed` | One agent could not be enriched, so the remaining inventory was returned. | -Registry-known agents enter a working-tree package inventory and render -immediately; source inspection does not block the first graph. An unresolved -agent uses a safe provisional marker or `local:` identity. After the snapshot -and its navigation sidecar commit, source-name inspection runs in the -background. A valid current source definition name becomes canonical and -publishes a newer graph revision, while the older marker remains only a -compatibility alias. An absent or invalid name preserves the provisional node -and any unambiguous direct edges. - -Cacheability follows whether identity work has finished, not whether it found -a canonical name. While any source identity is pending, or a failed inspection -could still succeed against unchanged source, the snapshot is `degraded` and -uncached so a provisional identity cannot be frozen in place. Once every -identity has settled, the snapshot is `ready` and cached. Invalid and settled -unavailable identities keep their sanitized per-agent warnings; retryable -failures retain the graph's Retry affordance. A source edit invalidates the -affected identity and projects it again. +Registry and syntax-discovered agents enter a working-tree package inventory +and render immediately. A syntax-proven source definition name is canonical +without bundling or executing project code; a retained marker/cloud slug remains +only a compatibility alias. Unknown or invalid identity uses a safe provisional marker +or `local:` key. Marker-authorized legacy name inspection and direct invocation +extraction run in bounded background queues after the inventory projection +commits. Settled identities and invocation edges publish later revisions; +failures preserve provisional nodes and unambiguous direct edges. + +Cacheability keeps three private facts separate: workspace discovery must be +complete, identity work must be settled, and direct invocation extraction must +be complete. A settled unavailable identity may therefore remain provisional +while the snapshot is `ready`; an incomplete workspace walk, pending/retryable +identity, or incomplete invocation scan keeps it `degraded`. Warnings and the +Retry affordance remain visible without freezing evidence that may still change. Package inventory protocol 1 is deliberately limited to which agents exist, their stable identities, and their package-relative locations. It carries no @@ -161,10 +183,11 @@ The event is an invalidation hint. Clients compare its key and revision with the displayed snapshot and refetch when newer; the graph itself is not sent on the event bus. -Opening a Project graph starts one session-independent recursive filesystem -watcher for that Project. This is additional to session and Canvas watchers so -the graph stays current even when no coding-agent session is open. Source and -inventory events are debounced. Platforms without recursive watch support, or -watchers that later error, fall back to asynchronous polling. Removing a -Project retires its watcher and process-memory snapshot once Studio no longer -exposes that scope. +Opening a Project graph acquires a canonical-root watcher lease. Sessions and +graphs for the same root share its bounded asynchronous fingerprint rather than +multiplying recursive walks. Relevant raw events synchronously make old +navigation inert; source/inventory reconciliation is debounced, coalesced, and +generation-guarded. Platforms without recursive watch support, or watchers that +later error, fall back to asynchronous polling over the same admitted candidate +and dependency observations. Removing the final lease retires the watcher and +degrades its accepted freshness proof before a later reopen can reuse it. diff --git a/packages/harness/src/core/agent-project-discovery.test.ts b/packages/harness/src/core/agent-project-discovery.test.ts index 672c5cd22..ed3a97174 100644 --- a/packages/harness/src/core/agent-project-discovery.test.ts +++ b/packages/harness/src/core/agent-project-discovery.test.ts @@ -1,21 +1,27 @@ import * as fs from "node:fs/promises"; +import { execFile } from "node:child_process"; import * as os from "node:os"; import * as path from "node:path"; import { afterEach, beforeEach, describe, expect, it } from "vitest"; +import { promisify } from "node:util"; import { AGENT_PROJECT_MARKER } from "../shared/types.js"; import { AGENT_PROJECT_SCAN_MAX_DEPTH, AGENT_PROJECT_SCAN_MAX_NODES, AGENT_PROJECT_WATCH_MAX_NODES, + AgentProjectScanAllowance, AgentProjectScanBudget, type AgentProjectWalkAction, + inspectAgentProjectMarker, readAgentProjectMarker, readAgentProjectMarkerSync, walkAgentProjectTree, walkAgentProjectTreeAsync, } from "./agent-project-discovery.js"; +const execFileAsync = promisify(execFile); + describe("agent project marker reads", () => { let root: string; @@ -61,6 +67,63 @@ describe("agent project marker reads", () => { await fs.rm(outside, { force: true }); } }); + + it("reads zero external bytes when the admitted marker is swapped to a symlink", async () => { + const markerPath = path.join(root, AGENT_PROJECT_MARKER); + const admittedPath = `${markerPath}.admitted`; + const outside = path.join( + path.dirname(root), + `${path.basename(root)}-outside.json`, + ); + await fs.writeFile(markerPath, JSON.stringify({ definitionId: 1 })); + await fs.writeFile(outside, JSON.stringify({ definitionId: 999 })); + let bytesRead = 0; + + try { + const result = await inspectAgentProjectMarker(root, { + beforeOpen: async () => { + await fs.rename(markerPath, admittedPath); + await fs.symlink(outside, markerPath); + }, + onBytesRead: (bytes) => { + bytesRead += bytes; + }, + }); + expect(result.status).toBe("unreadable"); + expect(bytesRead).toBe(0); + } finally { + await fs.rm(outside, { force: true }); + } + }); + + it("does not block or read when an ancestor swap resolves the marker to a FIFO", async () => { + const agent = path.join(root, "agent"); + const admitted = path.join(root, "agent-admitted"); + const replacement = path.join(root, "replacement"); + await fs.mkdir(agent); + await fs.mkdir(replacement); + await fs.writeFile( + path.join(agent, AGENT_PROJECT_MARKER), + JSON.stringify({ definitionId: 1 }), + ); + await execFileAsync("mkfifo", [ + path.join(replacement, AGENT_PROJECT_MARKER), + ]); + let bytesRead = 0; + + const result = await inspectAgentProjectMarker(agent, { + beforeOpen: async () => { + await fs.rename(agent, admitted); + await fs.symlink(replacement, agent, "dir"); + }, + onBytesRead: (bytes) => { + bytesRead += bytes; + }, + }); + + expect(result.status).toBe("unreadable"); + expect(bytesRead).toBe(0); + }); }); /** @@ -81,7 +144,10 @@ describe("bounded agent-project traversal", () => { }); /** Records every directory the walk enters, in order, and descends always. */ - function recorder(): { visits: [string, number][]; onDirectory: (dir: string, depth: number) => AgentProjectWalkAction } { + function recorder(): { + visits: [string, number][]; + onDirectory: (dir: string, depth: number) => AgentProjectWalkAction; + } { const visits: [string, number][] = []; return { visits, @@ -93,14 +159,19 @@ describe("bounded agent-project traversal", () => { } async function mkdirs(...relative: string[]): Promise { - for (const rel of relative) await fs.mkdir(path.join(root, rel), { recursive: true }); + for (const rel of relative) + await fs.mkdir(path.join(root, rel), { recursive: true }); } it("visits shallowest-first, so a truncated walk loses the DEEPEST level, not a branch", async () => { await mkdirs("a/deep/deeper", "b/deep/deeper", "c/deep/deeper"); const rec = recorder(); - const budget = walkAgentProjectTree(root, rec, new AgentProjectScanBudget()); + const budget = walkAgentProjectTree( + root, + rec, + new AgentProjectScanBudget(), + ); const depths = rec.visits.map(([, depth]) => depth); // Non-decreasing depth is what "breadth-first" means operationally, and it @@ -130,6 +201,22 @@ describe("bounded agent-project traversal", () => { expect(budget.envelopeDepth).toBe(1); }); + it("shares one node allowance across direct-root reconciliation walks", () => { + const allowance = new AgentProjectScanAllowance(3); + const first = new AgentProjectScanBudget({ maxNodes: 10 }, allowance); + const second = new AgentProjectScanBudget({ maxNodes: 10 }, allowance); + + expect(first.admit(0)).toBe(true); + expect(first.admit(1)).toBe(true); + expect(second.admit(0)).toBe(true); + expect(second.admit(1)).toBe(false); + + expect(allowance.visited).toBe(3); + expect(first.visited).toBe(2); + expect(second.visited).toBe(1); + expect(second.truncatedAtDepth).toBe(1); + }); + it("truncates at the same place twice, so a fingerprint built from it holds still", async () => { for (const a of ["a", "b", "c", "d", "e", "f"]) { for (const b of ["m", "n", "o", "p"]) await mkdirs(`${a}/${b}`); @@ -137,7 +224,11 @@ describe("bounded agent-project traversal", () => { const runs = [0, 1].map(() => { const rec = recorder(); - walkAgentProjectTree(root, rec, new AgentProjectScanBudget({ maxNodes: 12 })); + walkAgentProjectTree( + root, + rec, + new AgentProjectScanBudget({ maxNodes: 12 }), + ); return rec.visits.map(([rel]) => rel); }); @@ -149,19 +240,31 @@ describe("bounded agent-project traversal", () => { await mkdirs("node_modules/pkg/nested", "src/agents/one", ".git/objects"); const rec = recorder(); - walkAgentProjectTree(root, rec, new AgentProjectScanBudget({ maxDepth: 20 })); + walkAgentProjectTree( + root, + rec, + new AgentProjectScanBudget({ maxDepth: 20 }), + ); const seen = rec.visits.map(([rel]) => rel); expect(seen).toContain(path.join("src", "agents", "one")); - expect(seen.some((rel) => rel.split(path.sep).includes("node_modules"))).toBe(false); - expect(seen.some((rel) => rel.split(path.sep).includes(".git"))).toBe(false); + expect( + seen.some((rel) => rel.split(path.sep).includes("node_modules")), + ).toBe(false); + expect(seen.some((rel) => rel.split(path.sep).includes(".git"))).toBe( + false, + ); }); it("obeys maxDepth exactly: a directory at maxDepth is entered, one below is not", async () => { await mkdirs("l1/l2/l3/l4"); const rec = recorder(); - walkAgentProjectTree(root, rec, new AgentProjectScanBudget({ maxDepth: 3 })); + walkAgentProjectTree( + root, + rec, + new AgentProjectScanBudget({ maxDepth: 3 }), + ); const seen = rec.visits.map(([rel]) => rel); expect(seen).toContain(path.join("l1", "l2", "l3")); @@ -208,8 +311,13 @@ describe("bounded agent-project traversal", () => { "a", path.join("a", "b"), ]); - expect(await walkAgentProjectTreeAsync(root, recorder(), new AgentProjectScanBudget({ maxDepth: 64 }))) - .toMatchObject({ visited: 3, truncatedAtDepth: null }); + expect( + await walkAgentProjectTreeAsync( + root, + recorder(), + new AgentProjectScanBudget({ maxDepth: 64 }), + ), + ).toMatchObject({ visited: 3, truncatedAtDepth: null }); }, ); @@ -218,7 +326,11 @@ describe("bounded agent-project traversal", () => { // inside PATH_MAX on every platform this runs on. await mkdirs(Array.from({ length: 100 }, (_, i) => `d${i}`).join("/")); - const budget = walkAgentProjectTree(root, recorder(), new AgentProjectScanBudget()); + const budget = walkAgentProjectTree( + root, + recorder(), + new AgentProjectScanBudget(), + ); // One directory per level, root inclusive — the chain is never wider. expect(budget.visited).toBe(budget.maxDepth + 1); expect(budget.truncated).toBe(false); @@ -227,29 +339,46 @@ describe("bounded agent-project traversal", () => { it.skipIf( process.platform === "win32" || (typeof process.getuid === "function" && process.getuid() === 0), - )("reports an unreadable directory once, and a vanished one not at all", async () => { - await mkdirs("locked/child", "gone"); - await fs.rm(path.join(root, "gone"), { recursive: true, force: true }); - await fs.chmod(path.join(root, "locked"), 0o000); - - const unreadable: string[] = []; - try { - walkAgentProjectTree(root, { - onDirectory: () => "descend", - onUnreadable: (dir) => unreadable.push(path.relative(root, dir)), - }); - } finally { - await fs.chmod(path.join(root, "locked"), 0o700); - } - - expect(unreadable).toEqual(["locked"]); - }); + )( + "reports an unreadable directory once, and a vanished one not at all", + async () => { + await mkdirs("locked/child", "gone"); + await fs.rm(path.join(root, "gone"), { recursive: true, force: true }); + await fs.chmod(path.join(root, "locked"), 0o000); + const permissionsAreEnforced = await fs + .readdir(path.join(root, "locked")) + .then( + () => false, + () => true, + ); + + const unreadable: string[] = []; + try { + walkAgentProjectTree(root, { + onDirectory: () => "descend", + onUnreadable: (dir) => unreadable.push(path.relative(root, dir)), + }); + } finally { + await fs.chmod(path.join(root, "locked"), 0o700); + } + + // Some CI/container filesystems grant the runner permission capabilities + // that make chmod(000) readable. Pin the contract only when the fixture is + // genuinely unreadable; the deterministic race tests above cover the + // fail-closed read path independently of host permission semantics. + expect(unreadable).toEqual(permissionsAreEnforced ? ["locked"] : []); + }, + ); it("sync and async walks agree on order and on what the budget bought", async () => { await mkdirs("z/1", "a/2/3", "m/4", "node_modules/x"); const syncRec = recorder(); - const syncBudget = walkAgentProjectTree(root, syncRec, new AgentProjectScanBudget()); + const syncBudget = walkAgentProjectTree( + root, + syncRec, + new AgentProjectScanBudget(), + ); const asyncRec = recorder(); const asyncBudget = await walkAgentProjectTreeAsync( root, @@ -269,13 +398,16 @@ describe("bounded agent-project traversal", () => { walkAgentProjectTree(root, { onDirectory: rec.onDirectory, - onRepositoryBoundary: (dir, depth) => boundaries.push([path.relative(root, dir), depth]), + onRepositoryBoundary: (dir, depth) => + boundaries.push([path.relative(root, dir), depth]), }); // The checkout itself IS entered (its marker still gets inspected, so a // repo that is itself an agent is registered) — nothing below it is. expect(rec.visits.map(([rel]) => rel)).toContain("vendor-repo"); - expect(rec.visits.some(([rel]) => rel.startsWith(`vendor-repo${path.sep}`))).toBe(false); + expect( + rec.visits.some(([rel]) => rel.startsWith(`vendor-repo${path.sep}`)), + ).toBe(false); expect(rec.visits.map(([rel]) => rel)).toContain(`src${path.sep}agents`); expect(boundaries).toEqual([["vendor-repo", 1]]); }); @@ -292,7 +424,9 @@ describe("bounded agent-project traversal", () => { // This is the six-copies case: the same agent reachable once per worktree. expect( - rec.visits.some(([rel]) => rel.startsWith(`worktrees${path.sep}feature-a${path.sep}`)), + rec.visits.some(([rel]) => + rel.startsWith(`worktrees${path.sep}feature-a${path.sep}`), + ), ).toBe(false); }); @@ -302,8 +436,12 @@ describe("bounded agent-project traversal", () => { walkAgentProjectTree(root, rec); - expect(rec.visits.map(([rel]) => rel)).toContain(`packages${path.sep}a${path.sep}agents`); - expect(rec.visits.map(([rel]) => rel)).toContain(`packages${path.sep}b${path.sep}agents`); + expect(rec.visits.map(([rel]) => rel)).toContain( + `packages${path.sep}a${path.sep}agents`, + ); + expect(rec.visits.map(([rel]) => rel)).toContain( + `packages${path.sep}b${path.sep}agents`, + ); }); it("crossRepositoryBoundaries re-enables the old reach — the measurement escape hatch only", async () => { @@ -314,7 +452,11 @@ describe("bounded agent-project traversal", () => { crossRepositoryBoundaries: true, }); - expect(rec.visits.some(([rel]) => rel.startsWith(`vendor-repo${path.sep}agents`))).toBe(true); + expect( + rec.visits.some(([rel]) => + rel.startsWith(`vendor-repo${path.sep}agents`), + ), + ).toBe(true); }); it("sync and async walks agree about repository boundaries too", async () => { @@ -323,13 +465,15 @@ describe("bounded agent-project traversal", () => { const syncBoundaries: string[] = []; walkAgentProjectTree(root, { onDirectory: syncRec.onDirectory, - onRepositoryBoundary: (dir) => syncBoundaries.push(path.relative(root, dir)), + onRepositoryBoundary: (dir) => + syncBoundaries.push(path.relative(root, dir)), }); const asyncRec = recorder(); const asyncBoundaries: string[] = []; await walkAgentProjectTreeAsync(root, { onDirectory: asyncRec.onDirectory, - onRepositoryBoundary: (dir) => asyncBoundaries.push(path.relative(root, dir)), + onRepositoryBoundary: (dir) => + asyncBoundaries.push(path.relative(root, dir)), }); expect(asyncRec.visits).toEqual(syncRec.visits); @@ -345,6 +489,6 @@ describe("bounded agent-project traversal", () => { // under a chosen project root (`/backend/src/agents/ads` is 4). expect(AGENT_PROJECT_SCAN_MAX_DEPTH).toBeGreaterThanOrEqual(6); // The watcher's synchronous fingerprint must stay the cheaper of the two. - expect(AGENT_PROJECT_WATCH_MAX_NODES).toBeLessThan(AGENT_PROJECT_SCAN_MAX_NODES); + expect(AGENT_PROJECT_WATCH_MAX_NODES).toBe(AGENT_PROJECT_SCAN_MAX_NODES); }); }); diff --git a/packages/harness/src/core/agent-project-discovery.ts b/packages/harness/src/core/agent-project-discovery.ts index 491ce0edf..78c8ab87d 100644 --- a/packages/harness/src/core/agent-project-discovery.ts +++ b/packages/harness/src/core/agent-project-discovery.ts @@ -76,18 +76,13 @@ export const AGENT_PROJECT_SCAN_MAX_DEPTH = 8; export const AGENT_PROJECT_SCAN_MAX_NODES = 10_000; /** - * The watcher fingerprint's budget, deliberately tighter than the scan's. - * - * `snapshotWorkspaceWorkflows` is synchronous and re-runs on a 250 ms debounce - * after any file change under the session's cwd, so its cost lands on the event - * loop while the user is typing. 2,500 dirs ~= 60 ms warm, which is the same - * order as what depth-3 cost on the widest real root measured (1,298 dirs / - * 32 ms): the watcher gets the full depth on an ordinary project and gets no - * slower on a huge one. A truncated fingerprint means a *deep* structural - * change may not arm a rescan until some shallower event does; discovery depth - * itself is the registry scan's budget, not this one. + * The watcher must observe every directory the accepted discovery envelope can + * later reconcile. Production fingerprints are async and shared per canonical + * root, so keeping the same 10k breadth-first allowance avoids a permanently + * blind suffix on polling-only platforms without multiplying event-loop work + * per session/graph caller. The synchronous helper remains test/compat only. */ -export const AGENT_PROJECT_WATCH_MAX_NODES = 2_500; +export const AGENT_PROJECT_WATCH_MAX_NODES = AGENT_PROJECT_SCAN_MAX_NODES; /** * The entry name that marks a directory as its own repository checkout: a @@ -182,6 +177,13 @@ export type AgentProjectMarkerInspection = | { status: "valid"; marker: AgentProjectMarker } | { status: "absent" | "invalid" | "unreadable" }; +export interface AgentProjectMarkerInspectionHooks { + /** Deterministic race seam: runs after lstat admission and before open. */ + beforeOpen?: (markerPath: string) => void | Promise; + /** Test-only observation that project-controlled bytes were actually read. */ + onBytesRead?: (bytes: number) => void; +} + export function isAgentProjectScanIgnoredDir(name: string): boolean { return IGNORED_DIR_NAMES.has(name); } @@ -227,6 +229,60 @@ function markerReadErrorStatus(error: unknown): "absent" | "unreadable" { return code === "ENOENT" || code === "ENOTDIR" ? "absent" : "unreadable"; } +const AGENT_PROJECT_MARKER_MAX_BYTES = 64 * 1024; +const MARKER_OPEN_FLAGS = + fs.constants.O_RDONLY | + (fs.constants.O_NOFOLLOW ?? 0) | + (fs.constants.O_NONBLOCK ?? 0); +const MARKER_FALLBACK_OPEN_FLAGS = + fs.constants.O_RDONLY | (fs.constants.O_NONBLOCK ?? 0); + +function sameMarkerIdentity( + expected: import("node:fs").Stats, + actual: import("node:fs").Stats, +): boolean { + return ( + actual.isFile() && + !actual.isSymbolicLink() && + expected.dev === actual.dev && + expected.ino === actual.ino && + expected.size === actual.size && + expected.mtimeMs === actual.mtimeMs && + actual.size <= AGENT_PROJECT_MARKER_MAX_BYTES + ); +} + +function openMarkerSync(markerPath: string): number { + try { + return fs.openSync(markerPath, MARKER_OPEN_FLAGS); + } catch (error) { + if ( + (error as NodeJS.ErrnoException).code !== "EINVAL" || + MARKER_OPEN_FLAGS === MARKER_FALLBACK_OPEN_FLAGS + ) { + throw error; + } + // Some platforms do not implement O_NOFOLLOW. The pre-read fstat identity + // check below is still fail-closed: a followed replacement can be opened, + // but it is never read unless it is the exact lstat-authorized inode. + return fs.openSync(markerPath, MARKER_FALLBACK_OPEN_FLAGS); + } +} + +async function openMarker(markerPath: string): Promise { + try { + return await fsp.open(markerPath, MARKER_OPEN_FLAGS); + } catch (error) { + if ( + (error as NodeJS.ErrnoException).code !== "EINVAL" || + MARKER_OPEN_FLAGS === MARKER_FALLBACK_OPEN_FLAGS + ) { + throw error; + } + return fsp.open(markerPath, MARKER_FALLBACK_OPEN_FLAGS); + } +} + export function readAgentProjectMarkerSync( dir: string, ): AgentProjectMarker | null { @@ -251,11 +307,30 @@ export function inspectAgentProjectMarkerSync( return { status: "invalid" }; } + let fd: number | null = null; try { - const marker = parseAgentProjectMarker(fs.readFileSync(markerPath, "utf8")); + fd = openMarkerSync(markerPath); + const openedStat = fs.fstatSync(fd); + if (!sameMarkerIdentity(markerStat, openedStat)) { + return { status: "unreadable" }; + } + const buffer = Buffer.alloc(openedStat.size + 1); + const bytesRead = fs.readSync(fd, buffer, 0, buffer.length, 0); + const finalStat = fs.fstatSync(fd); + if ( + bytesRead !== openedStat.size || + !sameMarkerIdentity(openedStat, finalStat) + ) { + return { status: "unreadable" }; + } + const marker = parseAgentProjectMarker( + buffer.subarray(0, bytesRead).toString("utf8"), + ); return marker ? { status: "valid", marker } : { status: "invalid" }; } catch (error) { return { status: markerReadErrorStatus(error) }; + } finally { + if (fd !== null) fs.closeSync(fd); } } @@ -268,6 +343,7 @@ export async function readAgentProjectMarker( export async function inspectAgentProjectMarker( dir: string, + hooks: AgentProjectMarkerInspectionHooks = {}, ): Promise { const markerPath = resolveAgentProjectMarkerPath(dir); if (!markerPath) return { status: "invalid" }; @@ -283,13 +359,32 @@ export async function inspectAgentProjectMarker( return { status: "invalid" }; } + let handle: fsp.FileHandle | null = null; try { + await hooks.beforeOpen?.(markerPath); + handle = await openMarker(markerPath); + const openedStat = await handle.stat(); + if (!sameMarkerIdentity(markerStat, openedStat)) { + return { status: "unreadable" }; + } + const buffer = Buffer.alloc(openedStat.size + 1); + const { bytesRead } = await handle.read(buffer, 0, buffer.length, 0); + if (bytesRead > 0) hooks.onBytesRead?.(bytesRead); + const finalStat = await handle.stat(); + if ( + bytesRead !== openedStat.size || + !sameMarkerIdentity(openedStat, finalStat) + ) { + return { status: "unreadable" }; + } const marker = parseAgentProjectMarker( - await fsp.readFile(markerPath, "utf8"), + buffer.subarray(0, bytesRead).toString("utf8"), ); return marker ? { status: "valid", marker } : { status: "invalid" }; } catch (error) { return { status: markerReadErrorStatus(error) }; + } finally { + await handle?.close().catch(() => {}); } } @@ -299,6 +394,22 @@ export interface AgentProjectScanLimits { maxNodes: number; } +/** One logical workspace reconciliation allowance shared by direct-root walks. */ +export class AgentProjectScanAllowance { + readonly maxNodes: number; + visited = 0; + + constructor(maxNodes = AGENT_PROJECT_SCAN_MAX_NODES) { + this.maxNodes = maxNodes; + } + + admit(): boolean { + if (this.visited >= this.maxNodes) return false; + this.visited += 1; + return true; + } +} + /** * One walk's traversal allowance, and its report on what it managed to cover. * @@ -326,7 +437,10 @@ export class AgentProjectScanBudget implements AgentProjectScanLimits { */ repositoryBoundaries: string[] = []; - constructor(limits: Partial = {}) { + constructor( + limits: Partial = {}, + private readonly sharedAllowance?: AgentProjectScanAllowance, + ) { this.maxDepth = limits.maxDepth ?? AGENT_PROJECT_SCAN_MAX_DEPTH; this.maxNodes = limits.maxNodes ?? AGENT_PROJECT_SCAN_MAX_NODES; } @@ -350,7 +464,10 @@ export class AgentProjectScanBudget implements AgentProjectScanLimits { /** Charges one directory. False (and records the cut) once spent. */ admit(depth: number): boolean { - if (this.visited >= this.maxNodes) { + if ( + this.visited >= this.maxNodes || + (this.sharedAllowance && !this.sharedAllowance.admit()) + ) { if (this.truncatedAtDepth === null) this.truncatedAtDepth = depth; return false; } @@ -371,6 +488,16 @@ export interface AgentProjectWalkVisitor< > { /** Every directory entered, root first, shallowest level first. */ onDirectory(dir: string, depth: number): Action; + /** + * A directory whose entries were read and whose repository boundary was + * admitted. Marker inspection belongs in `onDirectory`; source discovery + * belongs here so an unmarked `index.ts` never crosses into another checkout. + */ + onAdmittedDirectory?( + dir: string, + depth: number, + entries: fs.Dirent[], + ): Action; /** `dir`'s entries could not be listed, and not because it is gone. */ onUnreadable?(dir: string, depth: number): void; /** @@ -409,7 +536,10 @@ export interface AgentProjectWalkOptions { */ function scanSubdirNames(entries: fs.Dirent[]): string[] { return entries - .filter((entry) => entry.isDirectory() && !isAgentProjectScanIgnoredDir(entry.name)) + .filter( + (entry) => + entry.isDirectory() && !isAgentProjectScanIgnoredDir(entry.name), + ) .map((entry) => entry.name) .sort(); } @@ -428,7 +558,9 @@ function isConfirmedMissingDir(error: unknown): boolean { * entered while it belongs to the same checkout. */ function stopsAtRepositoryBoundary( - visitor: AgentProjectWalkVisitor>, + visitor: AgentProjectWalkVisitor< + AgentProjectWalkAction | Promise + >, entries: fs.Dirent[], dir: string, depth: number, @@ -460,7 +592,11 @@ export function walkAgentProjectTree( options: AgentProjectWalkOptions = {}, ): AgentProjectScanBudget { let frontier = [path.resolve(root)]; - for (let depth = 0; depth <= budget.maxDepth && frontier.length > 0; depth += 1) { + for ( + let depth = 0; + depth <= budget.maxDepth && frontier.length > 0; + depth += 1 + ) { const next: string[] = []; for (const dir of frontier) { if (!budget.admit(depth)) return budget; @@ -472,8 +608,12 @@ export function walkAgentProjectTree( if (!isConfirmedMissingDir(error)) visitor.onUnreadable?.(dir, depth); continue; } - if (stopsAtRepositoryBoundary(visitor, entries, dir, depth, options)) continue; - for (const name of scanSubdirNames(entries)) next.push(path.join(dir, name)); + if (stopsAtRepositoryBoundary(visitor, entries, dir, depth, options)) + continue; + if (visitor.onAdmittedDirectory?.(dir, depth, entries) === "stop") + continue; + for (const name of scanSubdirNames(entries)) + next.push(path.join(dir, name)); } frontier = next; } @@ -488,12 +628,18 @@ export function walkAgentProjectTree( */ export async function walkAgentProjectTreeAsync( root: string, - visitor: AgentProjectWalkVisitor>, + visitor: AgentProjectWalkVisitor< + AgentProjectWalkAction | Promise + >, budget: AgentProjectScanBudget = new AgentProjectScanBudget(), options: AgentProjectWalkOptions = {}, ): Promise { let frontier = [path.resolve(root)]; - for (let depth = 0; depth <= budget.maxDepth && frontier.length > 0; depth += 1) { + for ( + let depth = 0; + depth <= budget.maxDepth && frontier.length > 0; + depth += 1 + ) { const next: string[] = []; for (const dir of frontier) { if (!budget.admit(depth)) return budget; @@ -505,8 +651,16 @@ export async function walkAgentProjectTreeAsync( if (!isConfirmedMissingDir(error)) visitor.onUnreadable?.(dir, depth); continue; } - if (stopsAtRepositoryBoundary(visitor, entries, dir, depth, options)) continue; - for (const name of scanSubdirNames(entries)) next.push(path.join(dir, name)); + if (stopsAtRepositoryBoundary(visitor, entries, dir, depth, options)) + continue; + if ( + visitor.onAdmittedDirectory && + (await visitor.onAdmittedDirectory(dir, depth, entries)) === "stop" + ) { + continue; + } + for (const name of scanSubdirNames(entries)) + next.push(path.join(dir, name)); } frontier = next; } diff --git a/packages/harness/src/core/agent-source-discovery.test.ts b/packages/harness/src/core/agent-source-discovery.test.ts new file mode 100644 index 000000000..34a80212b --- /dev/null +++ b/packages/harness/src/core/agent-source-discovery.test.ts @@ -0,0 +1,1658 @@ +import { execFile } from "node:child_process"; +import * as fs from "node:fs/promises"; +import * as os from "node:os"; +import * as path from "node:path"; +import { promisify } from "node:util"; + +import { afterEach, beforeEach, describe, expect, it } from "vitest"; + +import { + AGENT_SOURCE_MAX_BYTES_PER_CANDIDATE, + AGENT_SOURCE_MAX_BYTES_PER_SCAN, + AGENT_SOURCE_MAX_IMPORT_DEPTH, + AGENT_SOURCE_MAX_MODULES_PER_CANDIDATE, + AGENT_SOURCE_MAX_MODULES_PER_SCAN, + AGENT_SOURCE_MODULE_CACHE_MAX_ENTRIES, + AgentSourceDiscovery, + AgentSourceModuleCache, + AgentSourceScanBudget, +} from "./agent-source-discovery.js"; + +const execFileAsync = promisify(execFile); + +async function write( + root: string, + relativePath: string, + source: string, +): Promise { + const file = path.join(root, relativePath); + await fs.mkdir(path.dirname(file), { recursive: true }); + await fs.writeFile(file, source); +} + +function padSourceToBytes(source: string, bytes: number): string { + const opening = "\n/*"; + const closing = "*/"; + const padding = bytes - Buffer.byteLength(source + opening + closing); + if (padding < 0) throw new Error("source exceeds requested fixture size"); + return `${source}${opening}${"x".repeat(padding)}${closing}`; +} + +describe("AgentSourceDiscovery", () => { + let root: string; + + beforeEach(async () => { + root = await fs.mkdtemp(path.join(os.tmpdir(), "harness-source-agent-")); + }); + + afterEach(async () => { + await fs.rm(root, { recursive: true, force: true }); + }); + + it.each([ + { + label: "named current export", + source: `import { defineAgent } from "@sapiom/agent"; +export const agent = defineAgent({ name: " payments ", entry: "start", steps: {} });`, + name: "payments", + }, + { + label: "aliased current default export", + source: `import { defineAgent as create } from "@sapiom/agent"; +const factory = create; +export default factory({ name: \`billing\`, entry: "start", steps: {} });`, + name: "billing", + }, + { + label: "current namespace export", + source: `import * as sdk from "@sapiom/agent"; +export const agent = sdk.defineAgent({ name: "support", entry: "start", steps: {} });`, + name: "support", + }, + { + label: "legacy named export", + source: `import { defineOrchestration as define } from "@sapiom/orchestration"; +const workflow = define({ name: "legacy", entry: "start", steps: {} }); +export { workflow as orchestration };`, + name: "legacy", + }, + ])("proves $label without executing it", async ({ source, name }) => { + await write(root, "index.ts", source); + + const result = await new AgentSourceDiscovery().inspectCandidate(root); + + expect(result).toMatchObject({ status: "agent", name, modules: 1 }); + }); + + it.each([ + { + label: "literal computed name", + properties: `["name"]: "computed"`, + name: "computed", + }, + { + label: "unknown computed property before a final static name", + properties: `[key]: "unknown", name: "final"`, + name: "final", + }, + { + label: "unknown computed property after a static name", + properties: `name: "overridable", [key]: "unknown"`, + name: null, + }, + ])("handles $label conservatively", async ({ properties, name }) => { + await write( + root, + "index.ts", + `import { defineAgent } from "@sapiom/agent"; +declare const key: string; +export const agent = defineAgent({ ${properties} });`, + ); + + await expect( + new AgentSourceDiscovery().inspectCandidate(root), + ).resolves.toMatchObject({ status: "agent", name }); + }); + + it("resolves named/default imports, factory aliases, and re-exports", async () => { + await write( + root, + "factory.ts", + `export { defineAgent as makeAgent } from "@sapiom/agent";`, + ); + await write( + root, + "agent.ts", + `import { makeAgent } from "./factory"; +const defined = makeAgent({ name: "reexported", entry: "start", steps: {} }); +export { defined as default };`, + ); + await write( + root, + "barrel.ts", + `export { default as workflow } from "./agent.ts";`, + ); + await write( + root, + "index.ts", + `export { workflow as default } from "./barrel";`, + ); + + const result = await new AgentSourceDiscovery().inspectCandidate(root); + + expect(result).toMatchObject({ + status: "agent", + name: "reexported", + modules: 4, + }); + }); + + it.each([ + [".js", ".ts"], + [".jsx", ".tsx"], + [".mjs", ".mts"], + [".cjs", ".cts"], + ])( + "maps a NodeNext %s specifier back to a TypeScript %s module", + async (specifierExtension, sourceExtension) => { + await write( + root, + `agent${sourceExtension}`, + `import { defineAgent as make } from "@sapiom/agent"; +export const agent = make({ name: "nodenext", entry: "start", steps: {} });`, + ); + await write( + root, + "index.ts", + `export { agent as default } from "./agent${specifierExtension}";`, + ); + + await expect( + new AgentSourceDiscovery().inspectCandidate(root), + ).resolves.toMatchObject({ status: "agent", name: "nodenext" }); + }, + ); + + it("does not resolve a NodeNext .jsx specifier to a .ts module", async () => { + await write( + root, + "agent.ts", + `import { defineAgent } from "@sapiom/agent"; +export const agent = defineAgent({ name: "wrong-extension" });`, + ); + await write(root, "index.ts", `export { agent } from "./agent.jsx";`); + + await expect( + new AgentSourceDiscovery().inspectCandidate(root), + ).resolves.toMatchObject({ status: "incomplete" }); + }); + + it("continues relative resolution past ordinary non-file candidates", async () => { + await fs.mkdir(path.join(root, "agent.ts")); + await write( + root, + "agent.tsx", + `import { defineAgent } from "@sapiom/agent"; +export const agent = defineAgent({ name: "tsx-fallback" });`, + ); + await write(root, "index.ts", `export { agent } from "./agent.js";`); + await expect( + new AgentSourceDiscovery().inspectCandidate(root), + ).resolves.toMatchObject({ status: "agent", name: "tsx-fallback" }); + + await fs.rm(path.join(root, "agent.ts"), { recursive: true }); + await fs.rm(path.join(root, "agent.tsx")); + await fs.mkdir(path.join(root, "agent"), { recursive: true }); + await write( + root, + "agent/index.ts", + `import { defineAgent } from "@sapiom/agent"; +export const agent = defineAgent({ name: "index-fallback" });`, + ); + await write(root, "index.ts", `export { agent } from "./agent";`); + await expect( + new AgentSourceDiscovery().inspectCandidate(root), + ).resolves.toMatchObject({ status: "agent", name: "index-fallback" }); + }); + + it.each([ + `export * from "zod";`, + `export * from "zod"; +import { defineAgent } from "@sapiom/agent"; +export const agent = defineAgent({ name: "local", entry: "start", steps: {} });`, + ])("fails closed for an unresolved external export-star", async (source) => { + await write(root, "index.ts", source); + + await expect( + new AgentSourceDiscovery().inspectCandidate(root), + ).resolves.toMatchObject({ status: "incomplete" }); + }); + + it.each([ + { + label: "import-equals declaration", + extra: `import legacy = require("./legacy");`, + }, + { + label: "exported import-equals declaration", + extra: `export import legacy = require("./legacy");`, + }, + { + label: "export-equals assignment", + extra: `export = agent;`, + }, + { + label: "module.exports assignment", + extra: `module.exports = agent;`, + }, + { + label: "exports property assignment", + extra: `exports.agent = agent;`, + }, + { + label: "Object.defineProperty exports mutation", + extra: `Object.defineProperty(exports, "agent", { value: agent });`, + }, + { + label: "conditional module.exports assignment", + extra: `declare const condition: boolean; +if (condition) module.exports = agent;`, + }, + { + label: "Object.assign exports mutation", + extra: `Object.assign(exports, { agent });`, + }, + { + label: "Reflect.defineProperty exports mutation", + extra: `Reflect.defineProperty(exports, "agent", { value: agent });`, + }, + { + label: "exports escape to an unknown helper", + extra: `declare function decorateExports(value: unknown): void; +decorateExports(exports);`, + }, + { + label: "immediately-invoked CommonJS mutation", + extra: `(() => { module.exports.second = agent; })();`, + }, + { + label: "class static CommonJS mutation", + extra: `class ExportDecorator { + static { exports.second = agent; } +}`, + }, + { + label: "named function CommonJS mutation invoked later", + extra: `function decorateExportsLater() { + module.exports.second = agent; +} +decorateExportsLater();`, + }, + { + label: "class constructor CommonJS mutation invoked later", + extra: `class ExportDecorator { + constructor() { exports.second = agent; } +} +new ExportDecorator();`, + }, + ])( + "fails closed without executing an unsupported $label beside a proven ESM agent", + async ({ extra }) => { + await write( + root, + "index.ts", + `import { defineAgent } from "@sapiom/agent"; +export const agent = defineAgent({ name: "esm-agent" }); +${extra} +throw new Error("source discovery must never execute project code");`, + ); + + await expect( + new AgentSourceDiscovery().inspectCandidate(root), + ).resolves.toMatchObject({ status: "incomplete" }); + }, + ); + + it("honors an explicit export that shadows an export-star name", async () => { + await write( + root, + "a.ts", + `import { defineAgent } from "@sapiom/agent"; +export const agent = defineAgent({ name: "explicit", entry: "start", steps: {} });`, + ); + await write( + root, + "b.ts", + `import { defineAgent } from "@sapiom/agent"; +export const agent = defineAgent({ name: "shadowed", entry: "start", steps: {} });`, + ); + await write( + root, + "index.ts", + `export { agent } from "./a.js"; +export * from "./b.js";`, + ); + + await expect( + new AgentSourceDiscovery().inspectCandidate(root), + ).resolves.toMatchObject({ status: "agent", name: "explicit" }); + }); + + it("fails closed when export-stars provide the same name from different modules", async () => { + await write( + root, + "a.ts", + `import { defineAgent } from "@sapiom/agent"; +export const candidate = defineAgent({ name: "hidden" });`, + ); + await write(root, "b.ts", `export const candidate = {};`); + await write( + root, + "barrel.ts", + `export * from "./a.js"; +export * from "./b.js";`, + ); + await write(root, "index.ts", `export * from "./barrel.js";`); + + await expect( + new AgentSourceDiscovery().inspectCandidate(root), + ).resolves.toMatchObject({ status: "incomplete" }); + + await write(root, "index.ts", `export { candidate } from "./barrel.js";`); + await expect( + new AgentSourceDiscovery().inspectCandidate(root), + ).resolves.toMatchObject({ status: "incomplete" }); + }); + + it("resolves disjoint export-stars", async () => { + await write(root, "ordinary.ts", `export const helper = {};`); + await write( + root, + "agent.ts", + `import { defineAgent } from "@sapiom/agent"; +export const agent = defineAgent({ name: "disjoint" });`, + ); + await write( + root, + "index.ts", + `export * from "./ordinary.js"; +export * from "./agent.js";`, + ); + + await expect( + new AgentSourceDiscovery().inspectCandidate(root), + ).resolves.toMatchObject({ status: "agent", name: "disjoint" }); + }); + + it("accepts a star diamond that resolves to the same ultimate agent", async () => { + await write( + root, + "shared.ts", + `import { defineAgent } from "@sapiom/agent"; +export const agent = defineAgent({ name: "diamond" });`, + ); + await write(root, "a.ts", `export * from "./shared";`); + await write(root, "b.ts", `export * from "./shared";`); + await write( + root, + "index.ts", + `export * from "./a"; +export * from "./b";`, + ); + + await expect( + new AgentSourceDiscovery().inspectCandidate(root), + ).resolves.toMatchObject({ status: "agent", name: "diamond" }); + }); + + it("fails closed for a star diamond with distinct ultimate agents", async () => { + for (const branch of ["a", "b"]) { + await write( + root, + `${branch}-agent.ts`, + `import { defineAgent } from "@sapiom/agent"; +export const agent = defineAgent({ name: "${branch}" });`, + ); + await write(root, `${branch}.ts`, `export * from "./${branch}-agent";`); + } + await write( + root, + "index.ts", + `export * from "./a"; +export * from "./b";`, + ); + + await expect( + new AgentSourceDiscovery().inspectCandidate(root), + ).resolves.toMatchObject({ status: "incomplete" }); + }); + + it("resolves export-star cycles once and still finds the reachable agent", async () => { + await write(root, "index.ts", `export * from "./a";`); + await write( + root, + "a.ts", + `export * from "./b"; +import { defineAgent } from "@sapiom/agent"; +export const agent = defineAgent({ name: "cycle", entry: "start", steps: {} });`, + ); + await write(root, "b.ts", `export * from "./a";`); + + const result = await new AgentSourceDiscovery().inspectCandidate(root); + + expect(result).toMatchObject({ + status: "agent", + name: "cycle", + modules: 3, + }); + }); + + it.each([ + { + label: "explicit re-export cycle", + files: { + "index.ts": `export { agent } from "./a";`, + "a.ts": `export { agent } from "./b";`, + "b.ts": `export { agent } from "./a";`, + }, + }, + { + label: "local alias cycle", + files: { + "index.ts": `const first = second; +const second = first; +export { first as agent };`, + }, + }, + ])("fails closed for an $label", async ({ files }) => { + for (const [file, source] of Object.entries(files)) { + await write(root, file, source); + } + + await expect( + new AgentSourceDiscovery().inspectCandidate(root), + ).resolves.toMatchObject({ status: "incomplete" }); + }); + + it.each([ + `const defineAgent = (value: unknown) => value; +export const agent = defineAgent({ name: "fake" });`, + `import { defineAgent } from "not-sapiom"; +export const agent = defineAgent({ name: "fake" });`, + `// defineAgent({ name: "comment" }) +export const text = "defineAgent({ name: 'string' })";`, + `import type { defineAgent } from "@sapiom/agent"; +export const value = { name: "type-only" };`, + ])("does not accept token-shaped false positives", async (source) => { + await write(root, "index.ts", source); + + await expect( + new AgentSourceDiscovery().inspectCandidate(root), + ).resolves.toMatchObject({ + status: "not-agent", + }); + }); + + it.each([ + { + label: "dynamic factory", + source: `import { defineAgent } from "@sapiom/agent"; +const factory = Math.random() ? defineAgent : (value: unknown) => value; +export const agent = factory({ name: "dynamic" });`, + }, + { + label: "unresolved export", + source: `export { agent } from "./missing";`, + }, + { + label: "invalid syntax", + source: `export const agent = (;`, + }, + { + label: "dynamic conditional export", + source: `import { defineAgent } from "@sapiom/agent"; +declare const condition: boolean; +export default condition ? defineAgent({ name: "dynamic" }) : {};`, + }, + { + label: "mutable factory alias", + source: `import { defineAgent } from "@sapiom/agent"; +let factory = defineAgent; +export default factory({ name: "mutable" });`, + }, + { + label: "external dynamic factory", + source: `import { maybeFactory } from "another-package"; +export default maybeFactory({ name: "unknown" });`, + }, + { + label: "external namespace dynamic factory", + source: `import * as external from "another-package"; +export default external.maybeFactory({ name: "unknown" });`, + }, + { + label: "external value re-export", + source: `export { agent } from "another-package";`, + }, + ])("fails closed for $label", async ({ source }) => { + await write(root, "index.ts", source); + + await expect( + new AgentSourceDiscovery().inspectCandidate(root), + ).resolves.toMatchObject({ + status: "incomplete", + }); + }); + + it("accepts ordinary function, class, enum, and namespace exports beside one agent", async () => { + await write( + root, + "index.ts", + `import { defineAgent } from "@sapiom/agent"; +export function helper() {} +export class Helper {} +export enum Kind { One } +export namespace Values { export const one = 1; } +export const agent = defineAgent({ name: "mixed", entry: "start", steps: {} });`, + ); + + await expect( + new AgentSourceDiscovery().inspectCandidate(root), + ).resolves.toMatchObject({ status: "agent", name: "mixed" }); + }); + + it("does not let unrelated destructuring poison a proven agent export", async () => { + await write( + root, + "index.ts", + `import { defineAgent } from "@sapiom/agent"; +declare const config: { env: string }; +const { env } = config; +void env; +export const agent = defineAgent({ name: "destructured-context" });`, + ); + + await expect( + new AgentSourceDiscovery().inspectCandidate(root), + ).resolves.toMatchObject({ + status: "agent", + name: "destructured-context", + }); + }); + + it("does not let unrelated mutable state or overloads poison a proven agent", async () => { + await write( + root, + "index.ts", + `import { defineAgent } from "@sapiom/agent"; +let counter = 0; +counter++; +function helper(value: string): string; +function helper(value: unknown) { return String(value); } +void helper(counter); +export const agent = defineAgent({ name: "independent" });`, + ); + + await expect( + new AgentSourceDiscovery().inspectCandidate(root), + ).resolves.toMatchObject({ status: "agent", name: "independent" }); + }); + + it("accepts a valid exported overload beside a proven agent", async () => { + await write( + root, + "index.ts", + `import { defineAgent } from "@sapiom/agent"; +export function helper(value: string): string; +export function helper(value: number): string; +export function helper(value: string | number) { return String(value); } +export const agent = defineAgent({ name: "exported-overload" });`, + ); + + await expect( + new AgentSourceDiscovery().inspectCandidate(root), + ).resolves.toMatchObject({ status: "agent", name: "exported-overload" }); + }); + + it.each([ + `import { defineAgent } from "@sapiom/agent"; +const defineAgent = (value: unknown) => value; +export const agent = defineAgent({ name: "duplicate" });`, + `import { defineAgent } from "@sapiom/agent"; +export let agent = defineAgent({ name: "mutable" }); +agent = {};`, + `import { defineAgent } from "@sapiom/agent"; +declare const flag: boolean; +const factory = defineAgent; +if (flag) factory = (value: unknown) => value; +export default factory({ name: "conditional-write" });`, + `import { defineAgent } from "@sapiom/agent"; +({ defineAgent } = { defineAgent: (value: unknown) => value }); +export default defineAgent({ name: "destructuring-write" });`, + `import { defineAgent } from "@sapiom/agent"; +declare const factories: unknown[]; +for (defineAgent of factories) {} +export default defineAgent({ name: "loop-write" });`, + `import { defineAgent } from "@sapiom/agent"; +declare const factories: unknown[]; +[...defineAgent] = factories; +export default defineAgent({ name: "array-rest-write" });`, + `import { defineAgent } from "@sapiom/agent"; +declare const factories: unknown[]; +[defineAgent = (value: unknown) => value] = factories; +export default defineAgent({ name: "array-default-write" });`, + `import { defineAgent } from "@sapiom/agent"; +declare const values: { x?: unknown }; +({ x: defineAgent = (value: unknown) => value } = values); +export default defineAgent({ name: "object-default-write" });`, + `import * as sdk from "@sapiom/agent"; +sdk.defineAgent = (value: unknown) => value; +export default sdk.defineAgent({ name: "namespace-property-write" });`, + `import * as sdk from "@sapiom/agent"; +sdk["defineAgent"] = (value: unknown) => value; +export default sdk.defineAgent({ name: "namespace-element-write" });`, + `export const { agent } = getValues();`, + ])( + "fails closed for mutable or unresolved top-level bindings", + async (source) => { + await write(root, "index.ts", source); + + await expect( + new AgentSourceDiscovery().inspectCandidate(root), + ).resolves.toMatchObject({ status: "incomplete" }); + }, + ); + + it("does not treat a function-local shadow write as a top-level mutation", async () => { + await write( + root, + "index.ts", + `import { defineAgent } from "@sapiom/agent"; +const factory = defineAgent; +function unrelated() { let factory = 1; factory = 2; return factory; } +export default factory({ name: "outer-stable" });`, + ); + + await expect( + new AgentSourceDiscovery().inspectCandidate(root), + ).resolves.toMatchObject({ status: "agent", name: "outer-stable" }); + }); + + it.each([ + `import { defineAgent } from "@sapiom/agent"; +const build = () => defineAgent({ name: "wrapped" }); +export default build();`, + `const helper = () => ({ ordinary: true }); +export default helper();`, + `import { defineAgent } from "@sapiom/agent"; +const box = { make: defineAgent }; +export default box.make({ name: "member-alias" });`, + `import { defineAgent } from "@sapiom/agent"; +const box = { make: () => defineAgent({ name: "member-wrapper" }) }; +export default box.make();`, + `class Factory { static make() { return {}; } } +export default Factory.make();`, + ])("fails closed for an uninspected local callable", async (source) => { + await write(root, "index.ts", source); + + await expect( + new AgentSourceDiscovery().inspectCandidate(root), + ).resolves.toMatchObject({ status: "incomplete" }); + }); + + it.each([ + `import { defineAgent } from "@sapiom/agent"; +const agent = defineAgent({ name: "one" }); +const ordinary = {}; +export { agent as default }; +export { ordinary as default };`, + `import { defineAgent } from "@sapiom/agent"; +const agent = defineAgent({ name: "one" }); +const ordinary = {}; +export { agent as candidate }; +export { ordinary as candidate };`, + ])("fails closed for duplicate explicit export names", async (source) => { + await write(root, "index.ts", source); + + await expect( + new AgentSourceDiscovery().inspectCandidate(root), + ).resolves.toMatchObject({ status: "incomplete" }); + }); + + it("deduplicates one definition exported under multiple names", async () => { + await write( + root, + "index.ts", + `import { defineAgent } from "@sapiom/agent"; +const agent = defineAgent({ name: "once", entry: "start", steps: {} }); +export { agent, agent as default };`, + ); + + await expect( + new AgentSourceDiscovery().inspectCandidate(root), + ).resolves.toMatchObject({ status: "agent", name: "once" }); + }); + + it("does not execute top-level project code or side-effect imports", async () => { + const sentinel = `__sapiom_source_discovery_${Date.now()}`; + await write(root, "side-effect.ts", `throw new Error("imported");`); + await write( + root, + "index.ts", + `import "./side-effect.js"; +import { defineAgent } from "@sapiom/agent"; +(globalThis as Record)[${JSON.stringify(sentinel)}] = true; +throw new Error("executed"); +export const agent = defineAgent({ name: "syntax-only", entry: "start", steps: {} });`, + ); + + await expect( + new AgentSourceDiscovery().inspectCandidate(root), + ).resolves.toMatchObject({ status: "agent", name: "syntax-only" }); + expect((globalThis as Record)[sentinel]).toBeUndefined(); + }); + + it("fails closed when the entry exports multiple distinct definitions", async () => { + await write( + root, + "index.ts", + `import { defineAgent } from "@sapiom/agent"; +export const one = defineAgent({ name: "one", entry: "start", steps: {} }); +export const two = defineAgent({ name: "two", entry: "start", steps: {} });`, + ); + + await expect( + new AgentSourceDiscovery().inspectCandidate(root), + ).resolves.toMatchObject({ + status: "incomplete", + reason: "ambiguous-export", + }); + }); + + it("does not follow a relative import that escapes the candidate root", async () => { + await write( + path.dirname(root), + `${path.basename(root)}-outside.ts`, + `import { defineAgent } from "@sapiom/agent"; +export const agent = defineAgent({ name: "outside", entry: "start", steps: {} });`, + ); + await write( + root, + "index.ts", + `export { agent } from "../${path.basename(root)}-outside";`, + ); + + await expect( + new AgentSourceDiscovery().inspectCandidate(root), + ).resolves.toMatchObject({ + status: "incomplete", + }); + }); + + it.skipIf(process.platform === "win32")( + "does not follow a symlinked TypeScript module", + async () => { + const outside = `${root}-outside.ts`; + await fs.writeFile( + outside, + `import { defineAgent } from "@sapiom/agent"; +export const agent = defineAgent({ name: "outside", entry: "start", steps: {} });`, + ); + await fs.symlink(outside, path.join(root, "agent.ts")); + await write(root, "index.ts", `export { agent } from "./agent";`); + + await expect( + new AgentSourceDiscovery().inspectCandidate(root), + ).resolves.toMatchObject({ + status: "incomplete", + }); + await fs.rm(outside, { force: true }); + }, + ); + + it("treats a removed or non-file entrypoint as definitively absent", async () => { + const discovery = new AgentSourceDiscovery(); + await write( + root, + "index.ts", + `import { defineAgent } from "@sapiom/agent"; +export const agent = defineAgent({ name: "before" });`, + ); + await expect(discovery.inspectCandidate(root)).resolves.toMatchObject({ + status: "agent", + }); + + await fs.rm(path.join(root, "index.ts")); + await expect(discovery.inspectCandidate(root)).resolves.toMatchObject({ + status: "absent", + }); + await fs.mkdir(path.join(root, "index.ts")); + await expect(discovery.inspectCandidate(root)).resolves.toMatchObject({ + status: "absent", + }); + }); + + it.skipIf(process.platform === "win32")( + "treats a symlinked entrypoint as definitively absent", + async () => { + const outside = `${root}-entry.ts`; + await fs.writeFile(outside, `export const ordinary = true;`); + await fs.symlink(outside, path.join(root, "index.ts")); + + await expect( + new AgentSourceDiscovery().inspectCandidate(root), + ).resolves.toMatchObject({ status: "absent" }); + await fs.rm(outside, { force: true }); + }, + ); + + it("allows a nested candidate to resolve shared source within the selected workspace", async () => { + const candidate = path.join(root, "apps", "payments"); + await write( + root, + "shared/agent.ts", + `import { defineAgent } from "@sapiom/agent"; +export const agent = defineAgent({ name: "shared", entry: "start", steps: {} });`, + ); + await write( + candidate, + "index.ts", + `export { agent } from "../../shared/agent.js";`, + ); + + await expect( + new AgentSourceDiscovery().inspectCandidate( + candidate, + new AgentSourceScanBudget(), + root, + ), + ).resolves.toMatchObject({ status: "agent", name: "shared" }); + }); + + it.each(["node_modules", "dist", ".sapiom"])( + "rejects source reached through ignored directory %s", + async (ignored) => { + const candidate = path.join(root, "apps", "payments"); + await write( + root, + `${ignored}/agent.ts`, + `import { defineAgent } from "@sapiom/agent"; +export const agent = defineAgent({ name: "ignored" });`, + ); + await write( + candidate, + "index.ts", + `export { agent } from "../../${ignored}/agent.js";`, + ); + + await expect( + new AgentSourceDiscovery().inspectCandidate( + candidate, + new AgentSourceScanBudget(), + root, + ), + ).resolves.toMatchObject({ status: "incomplete" }); + }, + ); + + it("rejects a relative import into a nested repository", async () => { + const candidate = path.join(root, "apps", "payments"); + await write(root, "nested/.git", "gitdir: elsewhere"); + await write( + root, + "nested/agent.ts", + `import { defineAgent } from "@sapiom/agent"; +export const agent = defineAgent({ name: "foreign" });`, + ); + await write( + candidate, + "index.ts", + `export { agent } from "../../nested/agent.js";`, + ); + + await expect( + new AgentSourceDiscovery().inspectCandidate( + candidate, + new AgentSourceScanBudget(), + root, + ), + ).resolves.toMatchObject({ status: "incomplete" }); + }); + + it("allows a repository when that repository itself is the selected workspace", async () => { + await write(root, ".git", "gitdir: elsewhere"); + await write( + root, + "index.ts", + `import { defineAgent } from "@sapiom/agent"; +export const agent = defineAgent({ name: "selected-repo" });`, + ); + + await expect( + new AgentSourceDiscovery().inspectCandidate(root), + ).resolves.toMatchObject({ status: "agent", name: "selected-repo" }); + }); + + it("rejects markerless source at a nested repository root under the selected workspace", async () => { + const candidate = path.join(root, "nested-repo"); + await write(candidate, ".git", "gitdir: elsewhere"); + await write( + candidate, + "agent.ts", + `import { defineAgent } from "@sapiom/agent"; +export const agent = defineAgent({ name: "nested-selected" });`, + ); + await write(candidate, "index.ts", `export * from "./agent.js";`); + + await expect( + new AgentSourceDiscovery().inspectCandidate( + candidate, + new AgentSourceScanBudget(), + root, + ), + ).resolves.toMatchObject({ status: "incomplete" }); + }); + + it.skipIf(process.platform === "win32")( + "rejects relative imports through an intermediate directory symlink", + async () => { + const candidate = path.join(root, "apps", "payments"); + const shared = path.join(root, "shared-real"); + await write( + shared, + "agent.ts", + `import { defineAgent } from "@sapiom/agent"; +export const agent = defineAgent({ name: "symlinked" });`, + ); + await fs.symlink(shared, path.join(root, "shared")); + await write( + candidate, + "index.ts", + `export { agent } from "../../shared/agent.js";`, + ); + + await expect( + new AgentSourceDiscovery().inspectCandidate( + candidate, + new AgentSourceScanBudget(), + root, + ), + ).resolves.toMatchObject({ status: "incomplete" }); + }, + ); + + it.skipIf(process.platform === "win32")( + "fails closed when an admitted ancestor is swapped to a symlink", + async () => { + const candidate = path.join(root, "apps", "candidate"); + const moved = path.join(root, "apps", "candidate-real"); + await write(candidate, "index.ts", `export { agent } from "./agent.js";`); + await write( + candidate, + "agent.ts", + `import { defineAgent } from "@sapiom/agent"; +export const agent = defineAgent({ name: "must-not-follow" });`, + ); + let swapped = false; + const discovery = new AgentSourceDiscovery(new AgentSourceModuleCache(), { + beforeModuleLookup: async (file) => { + if (!file.endsWith(`${path.sep}agent.ts`) || swapped) return; + swapped = true; + await fs.rename(candidate, moved); + await fs.symlink(moved, candidate, "dir"); + }, + }); + + await expect( + discovery.inspectCandidate( + candidate, + new AgentSourceScanBudget(), + root, + ), + ).resolves.toMatchObject({ status: "incomplete" }); + }, + ); + + it.skipIf(process.platform === "win32")( + "does not read or block on a FIFO reached through an ancestor swap", + async () => { + const candidate = path.join(root, "apps", "candidate"); + const moved = path.join(root, "apps", "candidate-real"); + const replacement = path.join(root, "apps", "replacement"); + await write(candidate, "index.ts", `export * from "./agent.js";`); + await write(candidate, "agent.ts", `export const ordinary = true;`); + await fs.mkdir(replacement, { recursive: true }); + await execFileAsync("mkfifo", [path.join(replacement, "agent.ts")]); + const byteReads: string[] = []; + let swapped = false; + const discovery = new AgentSourceDiscovery(new AgentSourceModuleCache(), { + beforeModuleRead: async (file) => { + if (!file.endsWith(`${path.sep}agent.ts`) || swapped) return; + swapped = true; + await fs.rename(candidate, moved); + await fs.symlink(replacement, candidate, "dir"); + }, + onModuleBytesRead: (file) => byteReads.push(file), + }); + + await expect( + discovery.inspectCandidate( + candidate, + new AgentSourceScanBudget(), + root, + ), + ).resolves.toMatchObject({ status: "incomplete" }); + expect(byteReads).not.toContain(path.join(candidate, "agent.ts")); + }, + ); + + it.skipIf(process.platform === "win32")( + "resolves split source from a symlinked selected workspace root", + async () => { + const link = `${root}-link`; + await write( + root, + "agent.ts", + `import { defineAgent } from "@sapiom/agent"; +export const agent = defineAgent({ name: "symlink-root" });`, + ); + await write(root, "index.ts", `export * from "./agent.js";`); + await fs.symlink(root, link); + + await expect( + new AgentSourceDiscovery().inspectCandidate(link), + ).resolves.toMatchObject({ status: "agent", name: "symlink-root" }); + await fs.rm(link, { force: true }); + }, + ); + + it("rejects a file that grows between stat and its bounded read", async () => { + const entry = path.join(root, "index.ts"); + await write(root, "index.ts", `export const ordinary = true;`); + let mutated = false; + const discovery = new AgentSourceDiscovery(new AgentSourceModuleCache(), { + beforeModuleRead: async (file) => { + if (file !== entry || mutated) return; + mutated = true; + await fs.writeFile(file, Buffer.alloc(2 * 1024 * 1024, 32)); + }, + }); + + await expect(discovery.inspectCandidate(root)).resolves.toMatchObject({ + status: "incomplete", + }); + }); + + it("rejects a stale cache hit when the module changes after its initial stat", async () => { + const entry = path.join(root, "index.ts"); + await write( + root, + "index.ts", + `import { defineAgent } from "@sapiom/agent"; +export const agent = defineAgent({ name: "cached" });`, + ); + const cache = new AgentSourceModuleCache(); + await new AgentSourceDiscovery(cache).inspectCandidate(root); + let mutated = false; + const discovery = new AgentSourceDiscovery(cache, { + beforeModuleRead: async (file) => { + if (file !== entry || mutated) return; + mutated = true; + await fs.writeFile(file, `export const ordinary = false;`); + }, + }); + + await expect(discovery.inspectCandidate(root)).resolves.toMatchObject({ + status: "incomplete", + }); + }); + + it("rejects cold and warm modules when a nested repository appears before read", async () => { + const candidate = path.join(root, "candidate"); + await write( + candidate, + "index.ts", + `import { defineAgent } from "@sapiom/agent"; +export const agent = defineAgent({ name: "before-boundary" });`, + ); + + for (const warm of [false, true]) { + await fs.rm(path.join(candidate, ".git"), { + recursive: true, + force: true, + }); + const cache = new AgentSourceModuleCache(); + if (warm) { + await expect( + new AgentSourceDiscovery(cache).inspectCandidate( + candidate, + new AgentSourceScanBudget(), + root, + ), + ).resolves.toMatchObject({ status: "agent" }); + } + let inserted = false; + const discovery = new AgentSourceDiscovery(cache, { + beforeModuleRead: async (file) => { + if (!file.endsWith(`${path.sep}index.ts`) || inserted) return; + inserted = true; + await fs.mkdir(path.join(candidate, ".git")); + }, + }); + + await expect( + discovery.inspectCandidate( + candidate, + new AgentSourceScanBudget(), + root, + ), + ).resolves.toMatchObject({ status: "incomplete" }); + } + }); + + it("accepts import depth 8 and fails closed at depth 9", async () => { + const writeChain = async (depth: number, name: string) => { + await write(root, "index.ts", `export * from "./m1";`); + for (let index = 1; index <= depth; index += 1) { + await write( + root, + `m${index}.ts`, + index === depth + ? `import { defineAgent } from "@sapiom/agent"; +export const agent = defineAgent({ name: "${name}" });` + : `export * from "./m${index + 1}";`, + ); + } + }; + + await writeChain(AGENT_SOURCE_MAX_IMPORT_DEPTH, "at-depth-limit"); + await expect( + new AgentSourceDiscovery().inspectCandidate(root), + ).resolves.toMatchObject({ status: "agent", name: "at-depth-limit" }); + + await fs.rm(root, { recursive: true, force: true }); + await fs.mkdir(root, { recursive: true }); + await writeChain(AGENT_SOURCE_MAX_IMPORT_DEPTH + 1, "too-deep"); + await expect( + new AgentSourceDiscovery().inspectCandidate(root), + ).resolves.toMatchObject({ status: "incomplete" }); + }); + + it("accepts exactly 32 candidate modules and rejects the 33rd", async () => { + const writeWideGraph = async (modules: number) => { + await write( + root, + "index.ts", + Array.from( + { length: modules - 1 }, + (_, index) => `export * from "./m${index}";`, + ).join("\n"), + ); + for (let index = 0; index < modules - 1; index += 1) { + await write( + root, + `m${index}.ts`, + index === 0 + ? `import { defineAgent } from "@sapiom/agent"; +export const agent = defineAgent({ name: "wide" });` + : `export const helper${index} = ${index};`, + ); + } + }; + + await writeWideGraph(AGENT_SOURCE_MAX_MODULES_PER_CANDIDATE); + await expect( + new AgentSourceDiscovery().inspectCandidate(root), + ).resolves.toMatchObject({ + status: "agent", + name: "wide", + modules: AGENT_SOURCE_MAX_MODULES_PER_CANDIDATE, + }); + + await writeWideGraph(AGENT_SOURCE_MAX_MODULES_PER_CANDIDATE + 1); + await expect( + new AgentSourceDiscovery().inspectCandidate(root), + ).resolves.toMatchObject({ + status: "incomplete", + reason: "budget", + modules: AGENT_SOURCE_MAX_MODULES_PER_CANDIDATE, + }); + }); + + it("accepts exactly 1 MiB per candidate and rejects one byte more", async () => { + const source = `import { defineAgent } from "@sapiom/agent"; +export const agent = defineAgent({ name: "byte-limit" });`; + await write( + root, + "index.ts", + padSourceToBytes(source, AGENT_SOURCE_MAX_BYTES_PER_CANDIDATE), + ); + await expect( + new AgentSourceDiscovery().inspectCandidate(root), + ).resolves.toMatchObject({ + status: "agent", + name: "byte-limit", + bytes: AGENT_SOURCE_MAX_BYTES_PER_CANDIDATE, + }); + + await write( + root, + "index.ts", + padSourceToBytes(source, AGENT_SOURCE_MAX_BYTES_PER_CANDIDATE + 1), + ); + await expect( + new AgentSourceDiscovery().inspectCandidate(root), + ).resolves.toMatchObject({ + status: "incomplete", + reason: "budget", + bytes: 0, + }); + }); + + it("pins the exact scan-wide module and byte envelopes", () => { + const modules = new AgentSourceScanBudget(); + for (let index = 0; index < AGENT_SOURCE_MAX_MODULES_PER_SCAN; index += 1) { + expect(modules.admit(`/module-${index}.ts`, 0)).toBe(true); + } + expect(modules.modules).toBe(AGENT_SOURCE_MAX_MODULES_PER_SCAN); + expect(modules.admit("/module-over-limit.ts", 0)).toBe(false); + expect(modules.truncated).toBe(true); + + const bytes = new AgentSourceScanBudget(); + expect(bytes.admit("/exact.ts", AGENT_SOURCE_MAX_BYTES_PER_SCAN)).toBe( + true, + ); + // Re-admitting a warm/shared canonical module is free logically. + expect(bytes.admit("/exact.ts", AGENT_SOURCE_MAX_BYTES_PER_SCAN)).toBe( + true, + ); + expect(bytes.bytes).toBe(AGENT_SOURCE_MAX_BYTES_PER_SCAN); + expect(bytes.admit("/one-byte-more.ts", 1)).toBe(false); + expect(bytes.truncated).toBe(true); + }); + + it("enforces the 2,000/2,001 scan module boundary end to end", async () => { + const candidates: string[] = []; + const source = `import { defineAgent } from "@sapiom/agent"; +export const agent = defineAgent({ name: "large-repo" });`; + for ( + let index = 0; + index < AGENT_SOURCE_MAX_MODULES_PER_SCAN + 1; + index += 1 + ) { + const candidate = path.join(root, `candidate-${index}`); + candidates.push(candidate); + await write(candidate, "index.ts", source); + } + const discovery = new AgentSourceDiscovery(); + const inspect = async () => { + const budget = new AgentSourceScanBudget(); + const statuses: string[] = []; + for (const candidate of candidates) { + statuses.push( + (await discovery.inspectCandidate(candidate, budget, root)).status, + ); + } + return { + statuses, + modules: budget.modules, + bytes: budget.bytes, + truncated: budget.truncated, + }; + }; + + const cold = await inspect(); + const warm = await inspect(); + expect(cold.statuses.slice(0, AGENT_SOURCE_MAX_MODULES_PER_SCAN)).toEqual( + Array.from({ length: AGENT_SOURCE_MAX_MODULES_PER_SCAN }, () => "agent"), + ); + expect(cold.statuses.at(-1)).toBe("incomplete"); + expect(cold.modules).toBe(AGENT_SOURCE_MAX_MODULES_PER_SCAN); + expect(cold.truncated).toBe(true); + expect(warm).toEqual(cold); + }, 20_000); + + it("enforces the exact 16 MiB/+1 scan byte boundary end to end", async () => { + const exactCandidates: string[] = []; + const source = `import { defineAgent } from "@sapiom/agent"; +export const agent = defineAgent({ name: "byte-scan" });`; + const candidateCount = + AGENT_SOURCE_MAX_BYTES_PER_SCAN / AGENT_SOURCE_MAX_BYTES_PER_CANDIDATE; + for (let index = 0; index < candidateCount; index += 1) { + const candidate = path.join(root, `byte-candidate-${index}`); + exactCandidates.push(candidate); + await write( + candidate, + "index.ts", + padSourceToBytes(source, AGENT_SOURCE_MAX_BYTES_PER_CANDIDATE), + ); + } + const overCandidate = path.join(root, "byte-candidate-over"); + await write(overCandidate, "index.ts", "x"); + const discovery = new AgentSourceDiscovery(); + const inspect = async () => { + const budget = new AgentSourceScanBudget(); + const exact = []; + for (const candidate of exactCandidates) { + exact.push(await discovery.inspectCandidate(candidate, budget, root)); + } + const over = await discovery.inspectCandidate( + overCandidate, + budget, + root, + ); + return { + exact: exact.map((result) => result.status), + over: over.status, + overReason: over.status === "incomplete" ? over.reason : null, + bytes: budget.bytes, + truncated: budget.truncated, + }; + }; + + const cold = await inspect(); + const warm = await inspect(); + expect(cold.exact).toEqual( + Array.from({ length: candidateCount }, () => "agent"), + ); + expect(cold).toMatchObject({ + over: "incomplete", + overReason: "budget", + bytes: AGENT_SOURCE_MAX_BYTES_PER_SCAN, + truncated: true, + }); + expect(warm).toEqual(cold); + }); + + it("bounds and memoizes missing-module lookup work", async () => { + await write( + root, + "index.ts", + Array.from( + { length: 20 }, + (_, index) => `export { value as value${index} } from "./missing.js";`, + ).join("\n"), + ); + const repeated = await new AgentSourceDiscovery( + new AgentSourceModuleCache(), + { maxLookups: 5 }, + ).inspectCandidate(root); + expect(repeated).toMatchObject({ status: "incomplete", lookups: 3 }); + + await write( + root, + "index.ts", + Array.from( + { length: 20 }, + (_, index) => + `export { value as value${index} } from "./missing-${index}.js";`, + ).join("\n"), + ); + const budget = new AgentSourceScanBudget({ maxLookups: 7 }); + const unique = await new AgentSourceDiscovery( + new AgentSourceModuleCache(), + { maxLookups: 5 }, + ).inspectCandidate(root, budget); + expect(unique).toMatchObject({ + status: "incomplete", + reason: "budget", + lookups: 5, + }); + expect(budget.lookups).toBe(5); + }); + + it("deduplicates repeated stars and bounds dense export-resolution work", async () => { + await write( + root, + "agent.ts", + `import { defineAgent } from "@sapiom/agent"; +export const agent = defineAgent({ name: "repeated-star" });`, + ); + await write( + root, + "index.ts", + Array.from({ length: 2_000 }, () => `export * from "./agent";`).join( + "\n", + ), + ); + let steps = 0; + const discovery = new AgentSourceDiscovery(new AgentSourceModuleCache(), { + onResolutionStep: () => { + steps += 1; + }, + }); + const inspect = async () => { + steps = 0; + const result = await discovery.inspectCandidate(root); + return { result, steps }; + }; + const cold = await inspect(); + const warm = await inspect(); + expect(cold.result).toMatchObject({ + status: "agent", + name: "repeated-star", + }); + expect(warm).toEqual(cold); + expect(cold.steps).toBeLessThan(20); + + const branches = Array.from( + { length: 12 }, + (_, index) => `branch-${index}`, + ); + for (const branch of branches) { + await write(root, `${branch}.ts`, `export * from "./agent";`); + } + await write( + root, + "index.ts", + branches.map((branch) => `export * from "./${branch}";`).join("\n"), + ); + let boundedSteps = 0; + await expect( + new AgentSourceDiscovery(new AgentSourceModuleCache(), { + maxResolutionSteps: 20, + onResolutionStep: () => { + boundedSteps += 1; + }, + }).inspectCandidate(root), + ).resolves.toMatchObject({ status: "incomplete", reason: "budget" }); + expect(boundedSteps).toBe(20); + }); + + it("charges and reads a canonical module only once scan-wide across candidates", async () => { + await write( + root, + "shared/agent.ts", + `import { defineAgent } from "@sapiom/agent"; +export const agent = defineAgent({ name: "shared" });`, + ); + const candidates = Array.from({ length: 12 }, (_, index) => + path.join(root, "apps", `candidate-${index}`), + ); + for (const candidate of candidates) { + await write( + candidate, + "index.ts", + `export { agent } from "../../shared/agent.js";`, + ); + } + const metadataLoads: string[] = []; + const contentLoads: string[] = []; + const discovery = new AgentSourceDiscovery(new AgentSourceModuleCache(), { + beforeModuleLookup: (file) => { + metadataLoads.push(file); + }, + beforeModuleRead: async (file) => { + contentLoads.push(file); + }, + }); + const budget = new AgentSourceScanBudget(); + + const results = []; + for (const candidate of candidates) { + results.push(await discovery.inspectCandidate(candidate, budget, root)); + } + + expect(results).toEqual( + candidates.map(() => + expect.objectContaining({ + status: "agent", + name: "shared", + modules: 2, + }), + ), + ); + expect(budget.modules).toBe(candidates.length + 1); + expect(new Set(metadataLoads).size).toBe(candidates.length + 1); + expect(metadataLoads).toHaveLength(candidates.length + 1); + expect(new Set(contentLoads).size).toBe(candidates.length + 1); + expect(contentLoads).toHaveLength(candidates.length + 1); + }); + + it("memoizes shared missing-resolution probes across candidate graphs", async () => { + await fs.mkdir(path.join(root, "shared"), { recursive: true }); + const candidates = Array.from({ length: 20 }, (_, index) => + path.join(root, "apps", `candidate-${index}`), + ); + for (const candidate of candidates) { + await write( + candidate, + "index.ts", + `export { agent } from "../../shared/missing";`, + ); + } + const metadataLoads: string[] = []; + const discovery = new AgentSourceDiscovery(new AgentSourceModuleCache(), { + beforeModuleLookup: (file) => { + metadataLoads.push(file); + }, + }); + const budget = new AgentSourceScanBudget(); + + for (const candidate of candidates) { + await expect( + discovery.inspectCandidate(candidate, budget, root), + ).resolves.toMatchObject({ status: "incomplete" }); + } + + // One entrypoint per candidate, four direct extensions, then the first + // index candidate proves the shared missing directory cannot be entered. + expect(metadataLoads).toHaveLength(candidates.length + 5); + expect(new Set(metadataLoads).size).toBe(metadataLoads.length); + expect(budget.lookups).toBe(metadataLoads.length); + }); + + it("charges cold, warm, and post-eviction scans identically", async () => { + await write(root, "index.ts", `export { agent } from "./agent";`); + await write( + root, + "agent.ts", + `import { defineAgent } from "@sapiom/agent"; +export const agent = defineAgent({ name: "stable", entry: "start", steps: {} });`, + ); + const cache = new AgentSourceModuleCache(2); + const discovery = new AgentSourceDiscovery(cache); + const inspect = async () => { + const budget = new AgentSourceScanBudget({ maxModules: 2 }); + const result = await discovery.inspectCandidate(root, budget); + return { result, modules: budget.modules, bytes: budget.bytes }; + }; + + const cold = await inspect(); + const warm = await inspect(); + expect(warm).toEqual(cold); + + const other = await fs.mkdtemp( + path.join(os.tmpdir(), "harness-source-cache-"), + ); + await write(other, "index.ts", `export const ordinary = {};`); + await discovery.inspectCandidate(other); + await fs.rm(other, { recursive: true, force: true }); + + expect(await inspect()).toEqual(cold); + }); + + it("enforces the 10,000-entry LRU ceiling with true recency", () => { + const cache = new AgentSourceModuleCache(); + const parsed = { + bindings: new Map(), + exports: new Map(), + exportStars: [], + parseable: true, + unresolved: false, + }; + for ( + let index = 0; + index < AGENT_SOURCE_MODULE_CACHE_MAX_ENTRIES; + index += 1 + ) { + cache.set(`/module-${index}.ts`, 1, index, 1, index + 1, parsed); + } + expect(cache.size).toBe(AGENT_SOURCE_MODULE_CACHE_MAX_ENTRIES); + expect(cache.get("/module-0.ts", 1, 0, 1, 1)).toBe(parsed); + + cache.set( + "/module-over-limit.ts", + 1, + AGENT_SOURCE_MODULE_CACHE_MAX_ENTRIES, + 1, + AGENT_SOURCE_MODULE_CACHE_MAX_ENTRIES + 1, + parsed, + ); + + expect(cache.size).toBe(AGENT_SOURCE_MODULE_CACHE_MAX_ENTRIES); + expect(cache.get("/module-1.ts", 1, 1, 1, 2)).toBeNull(); + expect(cache.get("/module-0.ts", 1, 0, 1, 1)).toBe(parsed); + }); + + it("invalidates a split re-export when only its dependency changes", async () => { + await write(root, "index.ts", `export { agent } from "./agent";`); + await write( + root, + "agent.ts", + `import { defineAgent } from "@sapiom/agent"; +export const agent = defineAgent({ name: "before", entry: "start", steps: {} });`, + ); + const discovery = new AgentSourceDiscovery(); + await expect(discovery.inspectCandidate(root)).resolves.toMatchObject({ + status: "agent", + name: "before", + }); + + await new Promise((resolve) => setTimeout(resolve, 5)); + await write(root, "agent.ts", `export const noAgent = true;`); + + await expect(discovery.inspectCandidate(root)).resolves.toMatchObject({ + status: "not-agent", + }); + }); + + it("fingerprints absent relative candidates so creating one changes the result", async () => { + await write(root, "index.ts", `export { agent } from "./agent.js";`); + const discovery = new AgentSourceDiscovery(); + const before = await discovery.inspectCandidate(root); + expect(before).toMatchObject({ status: "incomplete" }); + + await write( + root, + "agent.ts", + `import { defineAgent } from "@sapiom/agent"; +export const agent = defineAgent({ name: "appeared" });`, + ); + const after = await discovery.inspectCandidate(root); + + expect(after).toMatchObject({ status: "agent", name: "appeared" }); + expect(after.fingerprint).not.toBe(before.fingerprint); + }); + + it("uses collision-free JSON framing for fingerprint entries", async () => { + const delimited = path.join(root, "part|with|pipes"); + await write(delimited, "index.ts", `export const ordinary = true;`); + + const result = await new AgentSourceDiscovery().inspectCandidate(delimited); + const entries = JSON.parse(result.fingerprint) as string[]; + + expect(entries).toHaveLength(1); + expect(entries[0]).toContain("part|with|pipes"); + }); +}); diff --git a/packages/harness/src/core/agent-source-discovery.ts b/packages/harness/src/core/agent-source-discovery.ts new file mode 100644 index 000000000..27e5f5baa --- /dev/null +++ b/packages/harness/src/core/agent-source-discovery.ts @@ -0,0 +1,2064 @@ +/** + * Syntax-only discovery of an agent exported from a candidate `index.ts`. + * + * This module deliberately creates no TypeScript Program or TypeChecker and + * never imports, bundles, type-checks, or executes project code. It follows + * only relative TypeScript imports that are needed to resolve the entry + * module's exports, within the candidate directory and explicit budgets. + */ +import * as fs from "node:fs/promises"; +import { constants as fsConstants } from "node:fs"; +import * as path from "node:path"; +import ts from "typescript"; + +import { isAgentProjectScanIgnoredDir } from "./agent-project-discovery.js"; + +export const AGENT_SOURCE_ENTRYPOINT = "index.ts"; +export const AGENT_SOURCE_MAX_IMPORT_DEPTH = 8; +export const AGENT_SOURCE_MAX_MODULES_PER_CANDIDATE = 32; +export const AGENT_SOURCE_MAX_BYTES_PER_CANDIDATE = 1024 * 1024; +export const AGENT_SOURCE_MAX_MODULES_PER_SCAN = 2_000; +export const AGENT_SOURCE_MAX_BYTES_PER_SCAN = 16 * 1024 * 1024; +export const AGENT_SOURCE_MODULE_CACHE_MAX_ENTRIES = 10_000; +export const AGENT_SOURCE_MAX_LOOKUPS_PER_CANDIDATE = 256; +export const AGENT_SOURCE_MAX_LOOKUPS_PER_SCAN = 16_000; +export const AGENT_SOURCE_MAX_RESOLUTION_STEPS_PER_CANDIDATE = 4_096; + +const CURRENT_AGENT_MODULE = "@sapiom/agent"; +const LEGACY_AGENT_MODULE = "@sapiom/orchestration"; +const CURRENT_FACTORY = "defineAgent"; +const LEGACY_FACTORY = "defineOrchestration"; +const TYPESCRIPT_EXTENSIONS = [".ts", ".tsx", ".mts", ".cts"] as const; + +function compareText(left: string, right: string): number { + return left === right ? 0 : left < right ? -1 : 1; +} + +function isWithin(root: string, candidate: string): boolean { + const relative = path.relative(root, candidate); + return ( + relative === "" || + (relative !== ".." && + !relative.startsWith(`..${path.sep}`) && + !path.isAbsolute(relative)) + ); +} + +function isConfirmedMissing(error: unknown): boolean { + const code = (error as NodeJS.ErrnoException).code; + return code === "ENOENT" || code === "ENOTDIR"; +} + +function sameFileSnapshot( + left: import("node:fs").Stats, + right: import("node:fs").Stats, +): boolean { + return ( + left.dev === right.dev && + left.ino === right.ino && + left.size === right.size && + left.mtimeMs === right.mtimeMs + ); +} + +export interface AgentSourceScanLimits { + maxModules: number; + maxBytes: number; + maxLookups: number; +} + +interface AgentSourceDiscoveryOptions extends Partial { + maxResolutionSteps?: number; + /** Test seam for proving scan-wide physical metadata work is memoized. */ + beforeModuleLookup?: (file: string) => Promise | void; + /** Test seam for deterministic file-mutation coverage. */ + beforeModuleRead?: (file: string) => Promise; + /** Test seam fired only after an opened inode matches admitted metadata. */ + onModuleBytesRead?: (file: string) => void; + /** Test seam for proving syntax-resolution CPU work remains bounded. */ + onResolutionStep?: () => void; +} + +/** + * Logical scan-wide allowance. A cache hit still charges the first encounter, + * while one canonical module shared by candidate graphs charges this workspace + * scan once (and each candidate's separate budget once). Caching changes I/O + * cost, never which rows or completeness an unchanged scan produces. + */ +export class AgentSourceScanBudget implements AgentSourceScanLimits { + readonly maxModules: number; + readonly maxBytes: number; + readonly maxLookups: number; + modules = 0; + bytes = 0; + lookups = 0; + truncated = false; + private readonly admittedModules = new Set(); + private readonly admittedLookups = new Set(); + private readonly moduleMetadata = new Map>(); + private readonly moduleContent = new Map>(); + + constructor(limits: Partial = {}) { + this.maxModules = limits.maxModules ?? AGENT_SOURCE_MAX_MODULES_PER_SCAN; + this.maxBytes = limits.maxBytes ?? AGENT_SOURCE_MAX_BYTES_PER_SCAN; + this.maxLookups = limits.maxLookups ?? AGENT_SOURCE_MAX_LOOKUPS_PER_SCAN; + } + + canAdmit(canonicalPath: string, size: number): boolean { + if (this.admittedModules.has(canonicalPath)) return true; + return this.modules < this.maxModules && this.bytes + size <= this.maxBytes; + } + + admit(canonicalPath: string, size: number): boolean { + if (this.admittedModules.has(canonicalPath)) return true; + if (!this.canAdmit(canonicalPath, size)) { + this.truncated = true; + return false; + } + this.admittedModules.add(canonicalPath); + this.modules += 1; + this.bytes += size; + return true; + } + + admitLookup(key: string): boolean { + if (this.admittedLookups.has(key)) return true; + if (this.lookups >= this.maxLookups) { + this.truncated = true; + return false; + } + this.admittedLookups.add(key); + this.lookups += 1; + return true; + } + + metadata( + key: string, + load: () => Promise, + ): Promise { + const existing = this.moduleMetadata.get(key); + if (existing) return existing; + const pending = load(); + this.moduleMetadata.set(key, pending); + return pending; + } + + content( + key: string, + load: () => Promise, + ): Promise { + const existing = this.moduleContent.get(key); + if (existing) return existing; + const pending = load(); + this.moduleContent.set(key, pending); + return pending; + } +} + +interface CandidateBudget { + readonly modules: Set; + bytes: number; + lookups: number; + truncated: boolean; +} + +type ExpressionFact = + | { kind: "identifier"; name: string } + | { kind: "property"; target: ExpressionFact; name: string } + | { + kind: "call"; + callee: ExpressionFact; + position: number; + declaredName: string | null; + } + | { kind: "normal" } + | { kind: "callable" } + | { kind: "dynamic" }; + +type LocalBinding = + | { kind: "expression"; expression: ExpressionFact } + | { + kind: "import"; + moduleSpecifier: string; + importedName: string; + } + | { kind: "namespace"; moduleSpecifier: string } + | { kind: "factory" }; + +type ExportBinding = + | { kind: "local"; localName: string } + | { kind: "expression"; expression: ExpressionFact } + | { + kind: "reexport"; + moduleSpecifier: string; + importedName: string; + }; + +interface ParsedModule { + readonly bindings: ReadonlyMap; + readonly exports: ReadonlyMap; + readonly exportStars: readonly string[]; + readonly parseable: boolean; + readonly unresolved: boolean; +} + +type ModuleMetadata = + | { + status: "missing" | "not-file" | "symlink" | "incomplete"; + fingerprint: string; + } + | { + status: "file"; + lexicalPath: string; + canonicalPath: string; + stat: import("node:fs").Stats; + fingerprint: string; + }; + +type ModuleContent = + | { status: "loaded"; parsed: ParsedModule } + | { status: "incomplete"; fingerprint: string }; + +interface CacheEntry { + readonly canonicalPath: string; + readonly parsed: ParsedModule; + readonly dev: number; + readonly ino: number; +} + +/** LRU of compact syntax facts, keyed exactly by canonical path + size + mtime. */ +export class AgentSourceModuleCache { + private readonly entries = new Map(); + private readonly keyByPath = new Map(); + + constructor( + private readonly maxEntries = AGENT_SOURCE_MODULE_CACHE_MAX_ENTRIES, + ) {} + + get size(): number { + return this.entries.size; + } + + clear(): void { + this.entries.clear(); + this.keyByPath.clear(); + } + + get( + canonicalPath: string, + size: number, + mtimeMs: number, + dev: number, + ino: number, + ): ParsedModule | null { + const key = moduleCacheKey(canonicalPath, size, mtimeMs); + const entry = this.entries.get(key); + if (!entry || entry.dev !== dev || entry.ino !== ino) return null; + this.entries.delete(key); + this.entries.set(key, entry); + return entry.parsed; + } + + set( + canonicalPath: string, + size: number, + mtimeMs: number, + dev: number, + ino: number, + parsed: ParsedModule, + ): void { + if (this.maxEntries <= 0) return; + const key = moduleCacheKey(canonicalPath, size, mtimeMs); + const previousKey = this.keyByPath.get(canonicalPath); + if (previousKey && previousKey !== key) this.entries.delete(previousKey); + this.entries.delete(key); + this.entries.set(key, { canonicalPath, parsed, dev, ino }); + this.keyByPath.set(canonicalPath, key); + while (this.entries.size > this.maxEntries) { + const oldestKey = this.entries.keys().next().value as string | undefined; + if (oldestKey === undefined) break; + const oldest = this.entries.get(oldestKey); + this.entries.delete(oldestKey); + if (oldest && this.keyByPath.get(oldest.canonicalPath) === oldestKey) { + this.keyByPath.delete(oldest.canonicalPath); + } + } + } +} + +function moduleCacheKey( + canonicalPath: string, + size: number, + mtimeMs: number, +): string { + return `${canonicalPath}\0${size}\0${mtimeMs}`; +} + +function unwrapExpression(expression: ts.Expression): ts.Expression { + let current = expression; + while ( + ts.isParenthesizedExpression(current) || + ts.isAsExpression(current) || + ts.isTypeAssertionExpression(current) || + ts.isNonNullExpression(current) || + ts.isSatisfiesExpression(current) + ) { + current = current.expression; + } + return current; +} + +function propertyName(name: ts.PropertyName): string | null { + if ( + ts.isIdentifier(name) || + ts.isStringLiteral(name) || + ts.isNumericLiteral(name) + ) { + return name.text; + } + if (ts.isComputedPropertyName(name)) { + const expression = unwrapExpression(name.expression); + if ( + ts.isStringLiteral(expression) || + ts.isNoSubstitutionTemplateLiteral(expression) + ) { + return expression.text; + } + } + return null; +} + +function declaredAgentName(call: ts.CallExpression): string | null { + const argument = call.arguments[0]; + if (!argument) return null; + const unwrapped = unwrapExpression(argument); + if (!ts.isObjectLiteralExpression(unwrapped)) return null; + + let name: string | null = null; + for (const property of unwrapped.properties) { + if (ts.isSpreadAssignment(property)) { + name = null; + continue; + } + if (!property.name) continue; + const staticPropertyName = propertyName(property.name); + if ( + ts.isComputedPropertyName(property.name) && + staticPropertyName === null + ) { + name = null; + continue; + } + if (staticPropertyName !== "name") continue; + if (!ts.isPropertyAssignment(property)) { + name = null; + continue; + } + const value = unwrapExpression(property.initializer); + name = + ts.isStringLiteral(value) || ts.isNoSubstitutionTemplateLiteral(value) + ? value.text.trim() || null + : null; + } + return name; +} + +function expressionFact(expression: ts.Expression): ExpressionFact { + const current = unwrapExpression(expression); + if (ts.isIdentifier(current)) { + return { kind: "identifier", name: current.text }; + } + if (ts.isPropertyAccessExpression(current) && ts.isIdentifier(current.name)) { + return { + kind: "property", + target: expressionFact(current.expression), + name: current.name.text, + }; + } + if (ts.isElementAccessExpression(current)) { + const argument = current.argumentExpression + ? unwrapExpression(current.argumentExpression) + : null; + if ( + argument && + (ts.isStringLiteral(argument) || + ts.isNoSubstitutionTemplateLiteral(argument)) + ) { + return { + kind: "property", + target: expressionFact(current.expression), + name: argument.text, + }; + } + } + if (ts.isCallExpression(current)) { + return { + kind: "call", + callee: expressionFact(current.expression), + position: current.getStart(), + declaredName: declaredAgentName(current), + }; + } + if ( + ts.isStringLiteral(current) || + ts.isNoSubstitutionTemplateLiteral(current) || + ts.isNumericLiteral(current) || + current.kind === ts.SyntaxKind.TrueKeyword || + current.kind === ts.SyntaxKind.FalseKeyword || + current.kind === ts.SyntaxKind.NullKeyword || + ts.isObjectLiteralExpression(current) || + ts.isArrayLiteralExpression(current) || + ts.isClassExpression(current) + ) { + return { kind: "normal" }; + } + if (ts.isArrowFunction(current) || ts.isFunctionExpression(current)) { + return { kind: "callable" }; + } + return { kind: "dynamic" }; +} + +function bindingName(name: ts.BindingName): string | null { + return ts.isIdentifier(name) ? name.text : null; +} + +function bindingNames(name: ts.BindingName): string[] { + if (ts.isIdentifier(name)) return [name.text]; + const names: string[] = []; + for (const element of name.elements) { + if (ts.isOmittedExpression(element)) continue; + names.push(...bindingNames(element.name)); + } + return names; +} + +function hasExportModifier(node: ts.Node): boolean { + return Boolean( + ts.canHaveModifiers(node) && + ts + .getModifiers(node) + ?.some((modifier) => modifier.kind === ts.SyntaxKind.ExportKeyword), + ); +} + +function hasDefaultModifier(node: ts.Node): boolean { + return Boolean( + ts.canHaveModifiers(node) && + ts + .getModifiers(node) + ?.some((modifier) => modifier.kind === ts.SyntaxKind.DefaultKeyword), + ); +} + +function exportName(name: ts.ModuleExportName): string { + return name.text; +} + +function collectWrittenTarget(node: ts.Node, names: Set): void { + const current = ts.isExpression(node) ? unwrapExpression(node) : node; + if (ts.isIdentifier(current)) { + names.add(current.text); + return; + } + if (ts.isPropertyAccessExpression(current)) { + collectWrittenTarget(current.expression, names); + return; + } + if (ts.isElementAccessExpression(current)) { + collectWrittenTarget(current.expression, names); + return; + } + if (ts.isSpreadElement(current)) { + collectWrittenTarget(current.expression, names); + return; + } + if ( + ts.isBinaryExpression(current) && + current.operatorToken.kind >= ts.SyntaxKind.FirstAssignment && + current.operatorToken.kind <= ts.SyntaxKind.LastAssignment + ) { + collectWrittenTarget(current.left, names); + return; + } + if (ts.isObjectLiteralExpression(current)) { + for (const property of current.properties) { + if (ts.isShorthandPropertyAssignment(property)) { + names.add(property.name.text); + } else if (ts.isPropertyAssignment(property)) { + collectWrittenTarget(property.initializer, names); + } else if (ts.isSpreadAssignment(property)) { + collectWrittenTarget(property.expression, names); + } + } + return; + } + if (ts.isArrayLiteralExpression(current)) { + for (const element of current.elements) { + if (!ts.isOmittedExpression(element)) + collectWrittenTarget(element, names); + } + } +} + +function collectWrittenExpression( + expression: ts.Expression, + names: Set, +): void { + const current = unwrapExpression(expression); + if ( + ts.isBinaryExpression(current) && + current.operatorToken.kind >= ts.SyntaxKind.FirstAssignment && + current.operatorToken.kind <= ts.SyntaxKind.LastAssignment + ) { + collectWrittenTarget(current.left, names); + return; + } + if ( + (ts.isPrefixUnaryExpression(current) || + ts.isPostfixUnaryExpression(current)) && + (current.operator === ts.SyntaxKind.PlusPlusToken || + current.operator === ts.SyntaxKind.MinusMinusToken) && + ts.isExpression(current.operand) + ) { + collectWrittenTarget(current.operand, names); + } +} + +function collectTopLevelWrites(node: ts.Node, names: Set): void { + if ( + ts.isFunctionDeclaration(node) || + ts.isFunctionExpression(node) || + ts.isArrowFunction(node) || + ts.isMethodDeclaration(node) || + ts.isGetAccessorDeclaration(node) || + ts.isSetAccessorDeclaration(node) || + ts.isConstructorDeclaration(node) || + ts.isClassDeclaration(node) || + ts.isClassExpression(node) + ) { + return; + } + if ( + (ts.isForInStatement(node) || ts.isForOfStatement(node)) && + !ts.isVariableDeclarationList(node.initializer) + ) { + collectWrittenTarget(node.initializer, names); + } + if (ts.isExpression(node)) { + collectWrittenExpression(node, names); + } + node.forEachChild((child) => collectTopLevelWrites(child, names)); +} + +function isCommonJsExportReference(expression: ts.Expression): boolean { + const current = unwrapExpression(expression); + if (ts.isIdentifier(current)) return current.text === "exports"; + if (ts.isPropertyAccessExpression(current)) { + if ( + ts.isIdentifier(current.expression) && + current.expression.text === "module" && + current.name.text === "exports" + ) { + return true; + } + return isCommonJsExportReference(current.expression); + } + if (ts.isElementAccessExpression(current)) { + if ( + ts.isIdentifier(current.expression) && + current.expression.text === "module" && + current.argumentExpression && + ts.isStringLiteralLike(current.argumentExpression) && + current.argumentExpression.text === "exports" + ) { + return true; + } + return isCommonJsExportReference(current.expression); + } + return false; +} + +function containsUnsupportedCommonJsExport(node: ts.Node): boolean { + let found = false; + const visit = (current: ts.Node): void => { + if (found) return; + // This parser proves ESM exports only. A CommonJS reference anywhere in + // the module can become observable through a later top-level call/new or + // decorator, and proving that call graph without executing project code is + // outside this bounded resolver. Conservatively reject even dormant + // function/class bodies rather than falsely accepting a mixed module. + if (ts.isExpression(current) && isCommonJsExportReference(current)) { + found = true; + return; + } + current.forEachChild(visit); + }; + visit(node); + return found; +} + +function pushExport( + exports: Map, + name: string, + binding: ExportBinding, +): boolean { + const current = exports.get(name) ?? []; + if (current.length > 0) return false; + current.push(binding); + exports.set(name, current); + return true; +} + +function officialFactory( + moduleSpecifier: string, + importedName: string, +): boolean { + return ( + (moduleSpecifier === CURRENT_AGENT_MODULE && + importedName === CURRENT_FACTORY) || + (moduleSpecifier === LEGACY_AGENT_MODULE && importedName === LEGACY_FACTORY) + ); +} + +function parseModule(file: string, content: string): ParsedModule { + const sourceFile = ts.createSourceFile( + path.basename(file), + content, + ts.ScriptTarget.Latest, + true, + path.extname(file) === ".tsx" ? ts.ScriptKind.TSX : ts.ScriptKind.TS, + ); + const parseDiagnostics = ( + sourceFile as ts.SourceFile & { + readonly parseDiagnostics: readonly ts.Diagnostic[]; + } + ).parseDiagnostics; + if (parseDiagnostics.length > 0) { + return { + bindings: new Map(), + exports: new Map(), + exportStars: [], + parseable: false, + unresolved: true, + }; + } + + const bindings = new Map(); + const exports = new Map(); + const exportStars: string[] = []; + const writtenNames = new Set(); + const validFunctionOverloads = new Set(); + const handledFunctionOverloads = new Set(); + let unresolved = false; + + const functionGroups = new Map(); + for (const statement of sourceFile.statements) { + if (!ts.isFunctionDeclaration(statement) || !statement.name) continue; + const group = functionGroups.get(statement.name.text) ?? []; + group.push(statement); + functionGroups.set(statement.name.text, group); + } + for (const [name, group] of functionGroups) { + const modifierShape = group.map( + (declaration) => + `${hasExportModifier(declaration)}:${hasDefaultModifier(declaration)}`, + ); + if ( + group.length > 1 && + group.filter((declaration) => declaration.body).length === 1 && + modifierShape.every((shape) => shape === modifierShape[0]) + ) { + validFunctionOverloads.add(name); + } + } + + const setBinding = (name: string, binding: LocalBinding): void => { + if (bindings.has(name)) { + bindings.set(name, { + kind: "expression", + expression: { kind: "dynamic" }, + }); + return; + } + bindings.set(name, binding); + }; + + for (const statement of sourceFile.statements) { + collectTopLevelWrites(statement, writtenNames); + + // This resolver proves ESM exports only. CommonJS/import-equals structures + // can add or replace the public export surface, so ignoring them would turn + // an unresolved module into a definitive not-agent (or one-agent) result. + if ( + ts.isImportEqualsDeclaration(statement) || + containsUnsupportedCommonJsExport(statement) + ) { + unresolved = true; + continue; + } + + if (ts.isImportDeclaration(statement)) { + if (statement.importClause?.isTypeOnly) continue; + if (!ts.isStringLiteral(statement.moduleSpecifier)) continue; + const moduleSpecifier = statement.moduleSpecifier.text; + const clause = statement.importClause; + if (!clause) continue; + if (clause.name) { + setBinding(clause.name.text, { + kind: "import", + moduleSpecifier, + importedName: "default", + }); + } + const named = clause.namedBindings; + if (named && ts.isNamespaceImport(named)) { + setBinding(named.name.text, { kind: "namespace", moduleSpecifier }); + } else if (named && ts.isNamedImports(named)) { + for (const element of named.elements) { + if (element.isTypeOnly) continue; + const importedName = element.propertyName?.text ?? element.name.text; + setBinding( + element.name.text, + officialFactory(moduleSpecifier, importedName) + ? { kind: "factory" } + : { kind: "import", moduleSpecifier, importedName }, + ); + } + } + continue; + } + + if ( + (ts.isEnumDeclaration(statement) || ts.isModuleDeclaration(statement)) && + ts.isIdentifier(statement.name) + ) { + setBinding(statement.name.text, { + kind: "expression", + expression: { kind: "normal" }, + }); + if (hasExportModifier(statement)) { + unresolved ||= !pushExport(exports, statement.name.text, { + kind: "local", + localName: statement.name.text, + }); + } + continue; + } + + if (ts.isVariableStatement(statement)) { + const immutable = Boolean( + statement.declarationList.flags & ts.NodeFlags.Const, + ); + for (const declaration of statement.declarationList.declarations) { + const directName = bindingName(declaration.name); + const names = bindingNames(declaration.name); + if (names.length === 0) unresolved = true; + for (const name of names) { + setBinding(name, { + kind: "expression", + expression: + directName === name && immutable && declaration.initializer + ? expressionFact(declaration.initializer) + : { kind: "dynamic" }, + }); + if (hasExportModifier(statement)) { + unresolved ||= !pushExport(exports, name, { + kind: "local", + localName: name, + }); + } + } + } + continue; + } + + if ( + ts.isFunctionDeclaration(statement) || + ts.isClassDeclaration(statement) + ) { + if ( + ts.isFunctionDeclaration(statement) && + statement.name && + validFunctionOverloads.has(statement.name.text) + ) { + if (handledFunctionOverloads.has(statement.name.text)) continue; + handledFunctionOverloads.add(statement.name.text); + } + if (statement.name) { + setBinding(statement.name.text, { + kind: "expression", + expression: { + kind: ts.isFunctionDeclaration(statement) ? "callable" : "normal", + }, + }); + } + if (hasExportModifier(statement)) { + const exportedName = hasDefaultModifier(statement) + ? "default" + : statement.name?.text; + if (exportedName) { + unresolved ||= !pushExport( + exports, + exportedName, + statement.name + ? { kind: "local", localName: statement.name.text } + : { + kind: "expression", + expression: { + kind: ts.isFunctionDeclaration(statement) + ? "callable" + : "normal", + }, + }, + ); + } + } + continue; + } + + if (ts.isExportAssignment(statement)) { + if (!statement.isExportEquals) { + unresolved ||= !pushExport(exports, "default", { + kind: "expression", + expression: expressionFact(statement.expression), + }); + } else { + unresolved = true; + } + continue; + } + + if (!ts.isExportDeclaration(statement) || statement.isTypeOnly) continue; + const moduleSpecifier = + statement.moduleSpecifier && ts.isStringLiteral(statement.moduleSpecifier) + ? statement.moduleSpecifier.text + : null; + if (!statement.exportClause) { + if (moduleSpecifier) exportStars.push(moduleSpecifier); + continue; + } + if (!ts.isNamedExports(statement.exportClause)) { + unresolved = true; + continue; + } + for (const element of statement.exportClause.elements) { + if (element.isTypeOnly) continue; + const exportedName = exportName(element.name); + const importedName = element.propertyName + ? exportName(element.propertyName) + : exportedName; + if (moduleSpecifier) { + unresolved ||= !pushExport(exports, exportedName, { + kind: "reexport", + moduleSpecifier, + importedName, + }); + } else { + unresolved ||= !pushExport(exports, exportedName, { + kind: "local", + localName: importedName, + }); + } + } + } + + for (const name of writtenNames) { + if (bindings.has(name)) { + bindings.set(name, { + kind: "expression", + expression: { kind: "dynamic" }, + }); + } + } + + return { + bindings, + exports, + exportStars: [...new Set(exportStars)].sort(compareText), + parseable: true, + unresolved, + }; +} + +interface AgentProof { + readonly id: string; + readonly name: string | null; +} + +interface ResolvedValue { + readonly factory: boolean; + readonly namespaces: readonly string[]; + readonly agents: ReadonlyMap; + readonly incomplete: boolean; + readonly unresolvedExternal: boolean; + readonly callable: boolean; +} + +const EMPTY_VALUE: ResolvedValue = { + factory: false, + namespaces: [], + agents: new Map(), + incomplete: false, + unresolvedExternal: false, + callable: false, +}; + +function incompleteValue(): ResolvedValue { + return { ...EMPTY_VALUE, incomplete: true }; +} + +function mergeValues(values: readonly ResolvedValue[]): ResolvedValue { + const agents = new Map(); + const namespaces = new Set(); + let factory = false; + let incomplete = false; + let unresolvedExternal = false; + let callable = false; + for (const value of values) { + factory ||= value.factory; + incomplete ||= value.incomplete; + unresolvedExternal ||= value.unresolvedExternal; + callable ||= value.callable; + for (const namespace of value.namespaces) namespaces.add(namespace); + for (const [id, agent] of value.agents) agents.set(id, agent); + } + return { + factory, + namespaces: [...namespaces].sort(compareText), + agents, + incomplete, + unresolvedExternal, + callable, + }; +} + +function equivalentResolvedBinding( + values: readonly ResolvedValue[], +): ResolvedValue | null { + if ( + values.length === 0 || + values.some((value) => value.incomplete || value.unresolvedExternal) + ) { + return null; + } + const signature = (value: ResolvedValue): string => + JSON.stringify({ + factory: value.factory, + namespaces: [...value.namespaces].sort(compareText), + agents: [...value.agents.entries()] + .sort(([left], [right]) => compareText(left, right)) + .map(([id, proof]) => [id, proof.name]), + callable: value.callable, + }); + const first = values[0] as ResolvedValue; + const expected = signature(first); + return values.every((value) => signature(value) === expected) ? first : null; +} + +interface LoadedModule { + readonly file: string; + readonly parsed: ParsedModule; +} + +interface ResolverState { + readonly workspaceRoot: string; + readonly canonicalWorkspaceRoot: string; + readonly admittedDirectory: Map< + string, + { admitted: boolean; stat?: import("node:fs").Stats } + >; + readonly scanBudget: AgentSourceScanBudget; + readonly candidateBudget: CandidateBudget; + readonly loaded: Map; + readonly exactLoads: Map>; + readonly observedFingerprints: Set; + resolutionSteps: number; +} + +async function isAdmittedModulePath( + file: string, + state: ResolverState, +): Promise { + const relativeDirectory = path.relative( + state.workspaceRoot, + path.dirname(file), + ); + if ( + relativeDirectory === ".." || + relativeDirectory.startsWith(`..${path.sep}`) || + path.isAbsolute(relativeDirectory) + ) { + return false; + } + let directory = state.workspaceRoot; + const segments = relativeDirectory.split(path.sep).filter(Boolean); + for (const segment of ["", ...segments]) { + if (segment) directory = path.join(directory, segment); + if (isAgentProjectScanIgnoredDir(segment)) return false; + const cached = state.admittedDirectory.get(directory); + if (cached !== undefined) { + if (!cached.admitted || !cached.stat) return false; + try { + const current = await fs.lstat(directory); + if ( + current.isSymbolicLink() || + !current.isDirectory() || + !sameFileSnapshot(current, cached.stat) + ) { + state.admittedDirectory.set(directory, { admitted: false }); + return false; + } + if (directory !== state.workspaceRoot) { + try { + await fs.lstat(path.join(directory, ".git")); + state.admittedDirectory.set(directory, { admitted: false }); + return false; + } catch (error) { + if (!isConfirmedMissing(error)) return false; + } + } + continue; + } catch { + state.admittedDirectory.set(directory, { admitted: false }); + return false; + } + } + try { + const directoryStat = await fs.lstat(directory); + if (!directoryStat.isDirectory() || directoryStat.isSymbolicLink()) { + state.admittedDirectory.set(directory, { admitted: false }); + return false; + } + if (directory !== state.workspaceRoot) { + try { + await fs.lstat(path.join(directory, ".git")); + state.admittedDirectory.set(directory, { admitted: false }); + return false; + } catch (error) { + if (!isConfirmedMissing(error)) { + state.admittedDirectory.set(directory, { admitted: false }); + return false; + } + } + } + state.admittedDirectory.set(directory, { + admitted: true, + stat: directoryStat, + }); + } catch (error) { + state.admittedDirectory.set(directory, { admitted: false }); + return false; + } + } + return true; +} + +type LoadResult = + | { status: "loaded"; module: LoadedModule } + | { status: "missing" } + | { status: "not-file" } + | { status: "symlink" } + | { status: "incomplete" }; + +function relativeModuleCandidates( + importer: string, + moduleSpecifier: string, +): string[] { + const base = path.resolve(path.dirname(importer), moduleSpecifier); + const extension = path.extname(base); + if ( + TYPESCRIPT_EXTENSIONS.includes( + extension as (typeof TYPESCRIPT_EXTENSIONS)[number], + ) + ) { + return [base]; + } + const sourceBase = base.slice(0, -extension.length); + if (extension === ".jsx") return [`${sourceBase}.tsx`]; + if (extension === ".mjs") return [`${sourceBase}.mts`]; + if (extension === ".cjs") return [`${sourceBase}.cts`]; + if (extension === ".js") { + return [`${sourceBase}.ts`, `${sourceBase}.tsx`]; + } + if (extension !== "") return []; + return [ + ...TYPESCRIPT_EXTENSIONS.map( + (candidateExtension) => `${base}${candidateExtension}`, + ), + ...TYPESCRIPT_EXTENSIONS.map((candidateExtension) => + path.join(base, `index${candidateExtension}`), + ), + ]; +} + +function isRelativeModuleSpecifier(moduleSpecifier: string): boolean { + return ( + moduleSpecifier === "." || + moduleSpecifier === ".." || + moduleSpecifier.startsWith("./") || + moduleSpecifier.startsWith("../") + ); +} + +export type AgentSourceDiscoveryResult = + | { + status: "absent" | "not-agent"; + fingerprint: string; + observations: readonly string[]; + watchPaths: readonly string[]; + modules: number; + bytes: number; + lookups: number; + } + | { + status: "agent"; + name: string | null; + fingerprint: string; + observations: readonly string[]; + watchPaths: readonly string[]; + modules: number; + bytes: number; + lookups: number; + } + | { + status: "incomplete"; + reason: + | "ambiguous-export" + | "budget" + | "invalid-source" + | "unreadable-source" + | "unresolved-export"; + fingerprint: string; + observations: readonly string[]; + watchPaths: readonly string[]; + modules: number; + bytes: number; + lookups: number; + }; + +export class AgentSourceDiscovery { + constructor( + private readonly cache = new AgentSourceModuleCache(), + private readonly options: AgentSourceDiscoveryOptions = {}, + ) {} + + async inspectCandidate( + candidateRoot: string, + scanBudget: AgentSourceScanBudget = new AgentSourceScanBudget(), + workspaceRoot: string = candidateRoot, + ): Promise { + const absoluteRoot = path.resolve(candidateRoot); + const absoluteWorkspaceRoot = path.resolve(workspaceRoot); + let canonicalCandidateRoot: string; + let canonicalWorkspaceRoot: string; + try { + canonicalCandidateRoot = await fs.realpath(absoluteRoot); + canonicalWorkspaceRoot = await fs.realpath(absoluteWorkspaceRoot); + } catch (error) { + return discoveryResult( + "incomplete", + new Set([`${absoluteRoot}\0`]), + { modules: new Set(), bytes: 0, lookups: 0, truncated: false }, + isConfirmedMissing(error) ? "unresolved-export" : "unreadable-source", + ); + } + if ( + !isWithin(absoluteWorkspaceRoot, absoluteRoot) || + !isWithin(canonicalWorkspaceRoot, canonicalCandidateRoot) + ) { + return discoveryResult( + "incomplete", + new Set([`${absoluteRoot}\0`]), + { modules: new Set(), bytes: 0, lookups: 0, truncated: false }, + "unresolved-export", + ); + } + const state: ResolverState = { + workspaceRoot: canonicalWorkspaceRoot, + canonicalWorkspaceRoot, + admittedDirectory: new Map(), + scanBudget, + candidateBudget: { + modules: new Set(), + bytes: 0, + lookups: 0, + truncated: false, + }, + loaded: new Map(), + exactLoads: new Map(), + observedFingerprints: new Set(), + resolutionSteps: 0, + }; + const entryPath = path.join( + canonicalCandidateRoot, + AGENT_SOURCE_ENTRYPOINT, + ); + const entry = await this.loadExactModule(entryPath, state); + if (entry.status === "missing") { + state.observedFingerprints.add(`${entryPath}\0`); + return discoveryResult( + "absent", + state.observedFingerprints, + state.candidateBudget, + ); + } + if (entry.status === "not-file" || entry.status === "symlink") { + return discoveryResult( + "absent", + state.observedFingerprints, + state.candidateBudget, + ); + } + if (entry.status === "incomplete") { + return discoveryResult( + "incomplete", + state.observedFingerprints, + state.candidateBudget, + scanBudget.truncated || state.candidateBudget.truncated + ? "budget" + : "unreadable-source", + ); + } + if (!entry.module.parsed.parseable) { + return discoveryResult( + "incomplete", + state.observedFingerprints, + state.candidateBudget, + "invalid-source", + ); + } + if (entry.module.parsed.unresolved) { + return discoveryResult( + "incomplete", + state.observedFingerprints, + state.candidateBudget, + "unresolved-export", + ); + } + + const value = await this.resolveAllExports( + entry.module, + 0, + state, + new Set(), + true, + ); + if (value.incomplete || value.unresolvedExternal) { + return discoveryResult( + "incomplete", + state.observedFingerprints, + state.candidateBudget, + scanBudget.truncated || state.candidateBudget.truncated + ? "budget" + : "unresolved-export", + ); + } + if (value.agents.size === 0) { + return discoveryResult( + "not-agent", + state.observedFingerprints, + state.candidateBudget, + ); + } + if (value.agents.size !== 1) { + return discoveryResult( + "incomplete", + state.observedFingerprints, + state.candidateBudget, + "ambiguous-export", + ); + } + const proof = value.agents.values().next().value as AgentProof; + return discoveryResult( + "agent", + state.observedFingerprints, + state.candidateBudget, + proof.name, + ); + } + + private async loadExactModule( + file: string, + state: ResolverState, + ): Promise { + const lexicalPath = path.resolve(file); + const lookupKey = `${state.workspaceRoot}\0${lexicalPath}`; + const existing = state.exactLoads.get(lookupKey); + if (existing) return existing; + const maxLookups = + this.options.maxLookups ?? AGENT_SOURCE_MAX_LOOKUPS_PER_CANDIDATE; + if (state.candidateBudget.lookups >= maxLookups) { + state.candidateBudget.truncated = true; + return { status: "incomplete" }; + } + if (!state.scanBudget.admitLookup(lookupKey)) { + return { status: "incomplete" }; + } + state.candidateBudget.lookups += 1; + const pending = this.loadExactModuleUnmemoized(lexicalPath, state); + state.exactLoads.set(lookupKey, pending); + return pending; + } + + private async loadExactModuleUnmemoized( + file: string, + state: ResolverState, + ): Promise { + const lexicalPath = path.resolve(file); + const lookupKey = `${state.workspaceRoot}\0${lexicalPath}`; + const metadata = await state.scanBudget.metadata(lookupKey, () => + this.readModuleMetadata(lexicalPath, state), + ); + state.observedFingerprints.add(metadata.fingerprint); + if (metadata.status !== "file") return { status: metadata.status }; + + const { canonicalPath, stat } = metadata; + const alreadyLoaded = state.loaded.get(canonicalPath); + if (alreadyLoaded) return { status: "loaded", module: alreadyLoaded }; + + const isNewCandidateModule = + !state.candidateBudget.modules.has(canonicalPath); + const nextModuleCount = + state.candidateBudget.modules.size + (isNewCandidateModule ? 1 : 0); + const nextCandidateBytes = + state.candidateBudget.bytes + (isNewCandidateModule ? stat.size : 0); + const maxModules = + this.options.maxModules ?? AGENT_SOURCE_MAX_MODULES_PER_CANDIDATE; + const maxBytes = + this.options.maxBytes ?? AGENT_SOURCE_MAX_BYTES_PER_CANDIDATE; + if ( + nextModuleCount > maxModules || + nextCandidateBytes > maxBytes || + !state.scanBudget.canAdmit(canonicalPath, stat.size) + ) { + state.candidateBudget.truncated ||= + nextModuleCount > maxModules || nextCandidateBytes > maxBytes; + state.scanBudget.truncated ||= !state.scanBudget.canAdmit( + canonicalPath, + stat.size, + ); + return { status: "incomplete" }; + } + + if (isNewCandidateModule) { + state.candidateBudget.modules.add(canonicalPath); + state.candidateBudget.bytes = nextCandidateBytes; + } + state.scanBudget.admit(canonicalPath, stat.size); + + const contentKey = [ + canonicalPath, + stat.dev, + stat.ino, + stat.size, + stat.mtimeMs, + ].join("\0"); + const content = await state.scanBudget.content(contentKey, () => + this.readModuleContent(metadata, state), + ); + if (content.status === "incomplete") { + state.observedFingerprints.add(content.fingerprint); + return { status: "incomplete" }; + } + const loaded = { file: canonicalPath, parsed: content.parsed }; + state.loaded.set(canonicalPath, loaded); + return { status: "loaded", module: loaded }; + } + + private async readModuleMetadata( + lexicalPath: string, + state: ResolverState, + ): Promise { + try { + await this.options.beforeModuleLookup?.(lexicalPath); + } catch { + return { + status: "incomplete", + fingerprint: `${lexicalPath}\0`, + }; + } + if (!isWithin(state.workspaceRoot, lexicalPath)) { + return { + status: "incomplete", + fingerprint: `${lexicalPath}\0`, + }; + } + if (!(await isAdmittedModulePath(lexicalPath, state))) { + return { + status: "incomplete", + fingerprint: `${lexicalPath}\0`, + }; + } + let stat: import("node:fs").Stats; + try { + stat = await fs.lstat(lexicalPath); + } catch (error) { + return { + status: isConfirmedMissing(error) ? "missing" : "incomplete", + fingerprint: `${lexicalPath}\0${ + isConfirmedMissing(error) ? "" : "" + }`, + }; + } + if (stat.isSymbolicLink()) { + return { + status: "symlink", + fingerprint: `${lexicalPath}\0`, + }; + } + if (!stat.isFile()) { + return { + status: "not-file", + fingerprint: `${lexicalPath}\0`, + }; + } + let canonicalPath: string; + try { + canonicalPath = await fs.realpath(lexicalPath); + } catch { + return { + status: "incomplete", + fingerprint: `${lexicalPath}\0`, + }; + } + if (!isWithin(state.canonicalWorkspaceRoot, canonicalPath)) { + return { + status: "incomplete", + fingerprint: `${lexicalPath}\0`, + }; + } + return { + status: "file", + lexicalPath, + canonicalPath, + stat, + fingerprint: `${lexicalPath}\0${stat.dev}\0${stat.ino}\0${stat.size}\0${stat.mtimeMs}`, + }; + } + + private async readModuleContent( + metadata: Extract, + state: ResolverState, + ): Promise { + const { canonicalPath, lexicalPath, stat } = metadata; + try { + await this.options.beforeModuleRead?.(canonicalPath); + } catch { + return { + status: "incomplete", + fingerprint: `${lexicalPath}\0`, + }; + } + let parsed = this.cache.get( + canonicalPath, + stat.size, + stat.mtimeMs, + stat.dev, + stat.ino, + ); + if (parsed) { + try { + if (!(await isAdmittedModulePath(lexicalPath, state))) { + throw new Error("module ancestors changed"); + } + const verifiedStat = await fs.lstat(lexicalPath); + const verifiedCanonicalPath = await fs.realpath(lexicalPath); + if ( + verifiedStat.isSymbolicLink() || + !verifiedStat.isFile() || + !sameFileSnapshot(verifiedStat, stat) || + verifiedCanonicalPath !== canonicalPath + ) { + return { + status: "incomplete", + fingerprint: `${lexicalPath}\0`, + }; + } + } catch { + return { + status: "incomplete", + fingerprint: `${lexicalPath}\0`, + }; + } + } else { + try { + const handle = await fs.open( + canonicalPath, + fsConstants.O_RDONLY | + fsConstants.O_NOFOLLOW | + fsConstants.O_NONBLOCK, + ); + try { + const openedStat = await handle.stat(); + const admitted = await isAdmittedModulePath(lexicalPath, state); + const beforeReadPathStat = await fs.lstat(lexicalPath); + const beforeReadCanonicalPath = await fs.realpath(lexicalPath); + if ( + !openedStat.isFile() || + !sameFileSnapshot(openedStat, stat) || + !admitted || + beforeReadPathStat.isSymbolicLink() || + !beforeReadPathStat.isFile() || + !sameFileSnapshot(beforeReadPathStat, stat) || + beforeReadCanonicalPath !== canonicalPath + ) { + return { + status: "incomplete", + fingerprint: `${lexicalPath}\0`, + }; + } + const bytes = Buffer.alloc(stat.size + 1); + this.options.onModuleBytesRead?.(canonicalPath); + let offset = 0; + while (offset < bytes.length) { + const { bytesRead } = await handle.read( + bytes, + offset, + bytes.length - offset, + offset, + ); + if (bytesRead === 0) break; + offset += bytesRead; + } + const finalHandleStat = await handle.stat(); + const finalPathStat = await fs.lstat(lexicalPath); + const finalCanonicalPath = await fs.realpath(lexicalPath); + const finalAdmission = await isAdmittedModulePath(lexicalPath, state); + const stable = + openedStat.isFile() && + !finalPathStat.isSymbolicLink() && + finalPathStat.isFile() && + sameFileSnapshot(openedStat, stat) && + sameFileSnapshot(finalHandleStat, stat) && + sameFileSnapshot(finalPathStat, stat) && + finalCanonicalPath === canonicalPath && + finalAdmission && + offset === stat.size; + if (!stable) { + return { + status: "incomplete", + fingerprint: `${lexicalPath}\0`, + }; + } + parsed = parseModule( + canonicalPath, + bytes.subarray(0, offset).toString("utf8"), + ); + } finally { + await handle.close(); + } + } catch { + return { + status: "incomplete", + fingerprint: `${lexicalPath}\0`, + }; + } + this.cache.set( + canonicalPath, + stat.size, + stat.mtimeMs, + stat.dev, + stat.ino, + parsed, + ); + } + return { status: "loaded", parsed }; + } + + private async loadRelativeModule( + importer: string, + moduleSpecifier: string, + depth: number, + state: ResolverState, + ): Promise { + if (!isRelativeModuleSpecifier(moduleSpecifier)) { + return { status: "missing" }; + } + if (depth > AGENT_SOURCE_MAX_IMPORT_DEPTH) { + return { status: "incomplete" }; + } + for (const candidate of relativeModuleCandidates( + importer, + moduleSpecifier, + )) { + const result = await this.loadExactModule(candidate, state); + if ( + result.status === "loaded" || + result.status === "symlink" || + result.status === "incomplete" + ) { + return result; + } + } + return { status: "incomplete" }; + } + + private admitResolutionStep(state: ResolverState): boolean { + const maxSteps = + this.options.maxResolutionSteps ?? + AGENT_SOURCE_MAX_RESOLUTION_STEPS_PER_CANDIDATE; + if (state.resolutionSteps >= maxSteps) { + state.candidateBudget.truncated = true; + return false; + } + state.resolutionSteps += 1; + this.options.onResolutionStep?.(); + return true; + } + + private async collectExportedNames( + module: LoadedModule, + depth: number, + state: ResolverState, + stack: Set, + ): Promise<{ names: Set; incomplete: boolean }> { + if (!this.admitResolutionStep(state)) { + return { names: new Set(), incomplete: true }; + } + const stackKey = `${module.file}\0export-names`; + if (stack.has(stackKey)) return { names: new Set(), incomplete: false }; + if (!module.parsed.parseable || module.parsed.unresolved) { + return { names: new Set(), incomplete: true }; + } + const nextStack = new Set(stack).add(stackKey); + const explicitNames = new Set(module.parsed.exports.keys()); + const names = new Set(explicitNames); + const starNameOwners = new Map(); + let incomplete = false; + for (const moduleSpecifier of module.parsed.exportStars) { + if (!isRelativeModuleSpecifier(moduleSpecifier)) { + incomplete = true; + continue; + } + const loaded = await this.loadRelativeModule( + module.file, + moduleSpecifier, + depth + 1, + state, + ); + if (loaded.status !== "loaded") { + incomplete = true; + continue; + } + const child = await this.collectExportedNames( + loaded.module, + depth + 1, + state, + nextStack, + ); + incomplete ||= child.incomplete; + for (const name of child.names) { + if (name === "default" || explicitNames.has(name)) continue; + const previousOwner = starNameOwners.get(name); + if (previousOwner && previousOwner !== loaded.module.file) { + incomplete = true; + names.delete(name); + continue; + } + if (!previousOwner) { + starNameOwners.set(name, loaded.module.file); + names.add(name); + } + } + } + return { names, incomplete }; + } + + private async resolveAllExports( + module: LoadedModule, + depth: number, + state: ResolverState, + stack: Set, + includeDefault: boolean, + excludedNames: ReadonlySet = new Set(), + ): Promise { + if (!this.admitResolutionStep(state)) return incompleteValue(); + if (module.parsed.unresolved) return incompleteValue(); + const stackKey = `${module.file}\0*\0${includeDefault ? "all" : "named"}`; + if (stack.has(stackKey)) return EMPTY_VALUE; + const nextStack = new Set(stack).add(stackKey); + const values: ResolvedValue[] = []; + for (const [name] of [...module.parsed.exports.entries()].sort( + ([left], [right]) => compareText(left, right), + )) { + if (excludedNames.has(name)) continue; + if (!includeDefault && name === "default") continue; + values.push( + await this.resolveExport(module, name, depth, state, nextStack), + ); + } + const shadowedNames = new Set(excludedNames); + shadowedNames.add("default"); + for (const name of module.parsed.exports.keys()) shadowedNames.add(name); + const starOwners = new Map(); + for (const moduleSpecifier of module.parsed.exportStars) { + if (!isRelativeModuleSpecifier(moduleSpecifier)) { + values.push(incompleteValue()); + continue; + } + const loaded = await this.loadRelativeModule( + module.file, + moduleSpecifier, + depth + 1, + state, + ); + if (loaded.status !== "loaded") { + values.push(incompleteValue()); + continue; + } + if (!loaded.module.parsed.parseable) { + values.push(incompleteValue()); + continue; + } + const exportedNames = await this.collectExportedNames( + loaded.module, + depth + 1, + state, + nextStack, + ); + if (exportedNames.incomplete) values.push(incompleteValue()); + for (const name of exportedNames.names) { + if (shadowedNames.has(name)) continue; + const owners = starOwners.get(name) ?? []; + owners.push(loaded.module); + starOwners.set(name, owners); + } + } + for (const [name, owners] of [...starOwners.entries()].sort( + ([left], [right]) => compareText(left, right), + )) { + const distinctOwners = new Map( + owners.map((owner) => [owner.file, owner]), + ); + const resolvedOwners: ResolvedValue[] = []; + for (const owner of distinctOwners.values()) { + resolvedOwners.push( + await this.resolveExport(owner, name, depth + 1, state, nextStack), + ); + } + const resolved = equivalentResolvedBinding(resolvedOwners); + values.push(resolved ?? incompleteValue()); + } + return mergeValues(values); + } + + private async resolveExport( + module: LoadedModule, + exportName: string, + depth: number, + state: ResolverState, + stack: Set, + ): Promise { + if (!this.admitResolutionStep(state)) return incompleteValue(); + if (module.parsed.unresolved) return incompleteValue(); + const stackKey = `${module.file}\0export\0${exportName}`; + if (stack.has(stackKey)) return incompleteValue(); + const nextStack = new Set(stack).add(stackKey); + const explicit = module.parsed.exports.get(exportName) ?? []; + if (explicit.length > 0) { + const values: ResolvedValue[] = []; + for (const binding of explicit) { + if (binding.kind === "local") { + values.push( + await this.resolveLocal( + module, + binding.localName, + depth, + state, + nextStack, + ), + ); + } else if (binding.kind === "expression") { + values.push( + await this.resolveExpression( + module, + binding.expression, + depth, + state, + nextStack, + ), + ); + } else { + values.push( + await this.resolveImportedExport( + module, + binding.moduleSpecifier, + binding.importedName, + depth, + state, + nextStack, + ), + ); + } + } + return mergeValues(values); + } + + const values: ResolvedValue[] = []; + const owners = new Map(); + for (const moduleSpecifier of module.parsed.exportStars) { + if (!isRelativeModuleSpecifier(moduleSpecifier)) { + values.push(incompleteValue()); + continue; + } + const loaded = await this.loadRelativeModule( + module.file, + moduleSpecifier, + depth + 1, + state, + ); + if (loaded.status !== "loaded" || !loaded.module.parsed.parseable) { + values.push(incompleteValue()); + continue; + } + const exportedNames = await this.collectExportedNames( + loaded.module, + depth + 1, + state, + nextStack, + ); + if (exportedNames.incomplete) values.push(incompleteValue()); + if (exportedNames.names.has(exportName)) { + owners.set(loaded.module.file, loaded.module); + } + } + if (owners.size > 0) { + const resolvedOwners: ResolvedValue[] = []; + for (const owner of owners.values()) { + resolvedOwners.push( + await this.resolveExport( + owner, + exportName, + depth + 1, + state, + nextStack, + ), + ); + } + values.push( + equivalentResolvedBinding(resolvedOwners) ?? incompleteValue(), + ); + } + return mergeValues(values); + } + + private async resolveImportedExport( + module: LoadedModule, + moduleSpecifier: string, + importedName: string, + depth: number, + state: ResolverState, + stack: Set, + ): Promise { + if (!this.admitResolutionStep(state)) return incompleteValue(); + if (officialFactory(moduleSpecifier, importedName)) { + return { ...EMPTY_VALUE, factory: true }; + } + if (!isRelativeModuleSpecifier(moduleSpecifier)) { + if (importedName === CURRENT_FACTORY || importedName === LEGACY_FACTORY) { + return EMPTY_VALUE; + } + return { ...EMPTY_VALUE, unresolvedExternal: true }; + } + const loaded = await this.loadRelativeModule( + module.file, + moduleSpecifier, + depth + 1, + state, + ); + if (loaded.status !== "loaded" || !loaded.module.parsed.parseable) { + return incompleteValue(); + } + return this.resolveExport( + loaded.module, + importedName, + depth + 1, + state, + stack, + ); + } + + private async resolveLocal( + module: LoadedModule, + localName: string, + depth: number, + state: ResolverState, + stack: Set, + ): Promise { + if (!this.admitResolutionStep(state)) return incompleteValue(); + const stackKey = `${module.file}\0local\0${localName}`; + if (stack.has(stackKey)) return incompleteValue(); + const binding = module.parsed.bindings.get(localName); + if (!binding) return incompleteValue(); + const nextStack = new Set(stack).add(stackKey); + if (binding.kind === "factory") { + return { ...EMPTY_VALUE, factory: true }; + } + if (binding.kind === "namespace") { + return { ...EMPTY_VALUE, namespaces: [binding.moduleSpecifier] }; + } + if (binding.kind === "import") { + return this.resolveImportedExport( + module, + binding.moduleSpecifier, + binding.importedName, + depth, + state, + nextStack, + ); + } + return this.resolveExpression( + module, + binding.expression, + depth, + state, + nextStack, + ); + } + + private async resolveExpression( + module: LoadedModule, + expression: ExpressionFact, + depth: number, + state: ResolverState, + stack: Set, + ): Promise { + if (!this.admitResolutionStep(state)) return incompleteValue(); + if (expression.kind === "normal") return EMPTY_VALUE; + if (expression.kind === "callable") { + return { ...EMPTY_VALUE, callable: true }; + } + if (expression.kind === "dynamic") return incompleteValue(); + if (expression.kind === "identifier") { + return this.resolveLocal(module, expression.name, depth, state, stack); + } + if (expression.kind === "property") { + const target = await this.resolveExpression( + module, + expression.target, + depth, + state, + stack, + ); + const values: ResolvedValue[] = []; + if (target.incomplete) values.push(incompleteValue()); + if (target.unresolvedExternal) { + values.push({ ...EMPTY_VALUE, unresolvedExternal: true }); + } + for (const namespace of target.namespaces) { + values.push( + await this.resolveImportedExport( + module, + namespace, + expression.name, + depth, + state, + stack, + ), + ); + } + if (target.namespaces.length === 0) values.push(incompleteValue()); + return mergeValues(values); + } + + const callee = await this.resolveExpression( + module, + expression.callee, + depth, + state, + stack, + ); + if (!callee.factory) { + const knownLocalLookalike = + expression.callee.kind === "identifier" && + (expression.callee.name === CURRENT_FACTORY || + expression.callee.name === LEGACY_FACTORY); + if (callee.callable && !knownLocalLookalike) return incompleteValue(); + return callee.incomplete || callee.unresolvedExternal + ? { + ...EMPTY_VALUE, + incomplete: callee.incomplete, + unresolvedExternal: callee.unresolvedExternal, + } + : EMPTY_VALUE; + } + const proof: AgentProof = { + id: `${module.file}:${expression.position}`, + name: expression.declaredName, + }; + return { + factory: false, + namespaces: [], + agents: new Map([[proof.id, proof]]), + incomplete: callee.incomplete, + unresolvedExternal: callee.unresolvedExternal, + callable: false, + }; + } +} + +function discoveryResult( + status: "absent" | "not-agent", + fingerprints: ReadonlySet, + budget: CandidateBudget, +): AgentSourceDiscoveryResult; +function discoveryResult( + status: "agent", + fingerprints: ReadonlySet, + budget: CandidateBudget, + name: string | null, +): AgentSourceDiscoveryResult; +function discoveryResult( + status: "incomplete", + fingerprints: ReadonlySet, + budget: CandidateBudget, + reason: Extract< + AgentSourceDiscoveryResult, + { status: "incomplete" } + >["reason"], +): AgentSourceDiscoveryResult; +function discoveryResult( + status: "absent" | "not-agent" | "agent" | "incomplete", + fingerprints: ReadonlySet, + budget: CandidateBudget, + detail?: string | null, +): AgentSourceDiscoveryResult { + const sortedFingerprints = [...fingerprints].sort(compareText); + const watchPaths = sortedFingerprints + .filter((fingerprint) => { + const fields = fingerprint.split("\0"); + return ( + fields.length >= 5 || + fields[1] === "" || + fields[1] === "" || + fields[1] === "" || + fields[1] === "" || + fields[1] === "" || + fields[1] === "" + ); + }) + .map((fingerprint) => fingerprint.slice(0, fingerprint.indexOf("\0"))) + .filter((observedPath) => path.isAbsolute(observedPath)); + const common = { + observations: sortedFingerprints, + watchPaths: [...new Set(watchPaths)].sort(compareText), + fingerprint: JSON.stringify(sortedFingerprints), + modules: budget.modules.size, + bytes: budget.bytes, + lookups: budget.lookups, + }; + if (status === "agent") { + return { status, name: detail ?? null, ...common }; + } + if (status === "incomplete") { + return { + status, + reason: detail as Extract< + AgentSourceDiscoveryResult, + { status: "incomplete" } + >["reason"], + ...common, + }; + } + return { status, ...common }; +} diff --git a/packages/harness/src/core/canonical-graph-path.ts b/packages/harness/src/core/canonical-graph-path.ts new file mode 100644 index 000000000..ed88a5fd8 --- /dev/null +++ b/packages/harness/src/core/canonical-graph-path.ts @@ -0,0 +1,96 @@ +import { realpathSync } from "node:fs"; +import * as path from "node:path"; + +const MAX_CACHED_PATHS = 20_000; +const canonicalPaths = new Map(); +let probe: ((path: string) => void) | null = null; + +function isWindowsAbsolute(input: string): boolean { + return ( + /^[A-Za-z]:[\\/]/.test(input) || /^[\\/]{2}[^\\/]+[\\/][^\\/]+/.test(input) + ); +} + +function pathApi(input: string): typeof path.posix { + return isWindowsAbsolute(input) ? path.win32 : path.posix; +} + +function normalizedAbsolute(input: string): string { + const api = pathApi(input); + return api.resolve( + isWindowsAbsolute(input) ? input.replace(/\//g, "\\") : input, + ); +} + +function remember(key: string, value: string): void { + canonicalPaths.delete(key); + canonicalPaths.set(key, value); + while (canonicalPaths.size > MAX_CACHED_PATHS) { + const oldest = canonicalPaths.keys().next().value as string | undefined; + if (oldest === undefined) break; + canonicalPaths.delete(oldest); + } +} + +/** Seeds canonical evidence produced asynchronously by registry load/scan. */ +export function rememberCanonicalGraphPath( + input: string, + canonical: string, +): void { + const key = normalizedAbsolute(input); + const value = normalizedAbsolute(canonical); + remember(key, value); + remember(value, value); +} + +/** + * Canonicalizes an arbitrary graph path. Production registry paths hit the + * memory cache; the filesystem fallback remains for defensive arbitrary + * providers and watcher paths that have not yet been reconciled. + */ +export function canonicalGraphPath(input: string): string { + const windows = isWindowsAbsolute(input); + const api = pathApi(input); + const resolved = normalizedAbsolute(input); + const cached = canonicalPaths.get(resolved); + if (cached !== undefined) { + remember(resolved, cached); + return cached; + } + const matchesHost = windows === (process.platform === "win32"); + if (!matchesHost) return resolved; + let result = resolved; + try { + probe?.(resolved); + result = realpathSync.native(resolved); + } catch { + const missingSegments: string[] = []; + let ancestor = resolved; + let parent = api.dirname(ancestor); + while (parent !== ancestor) { + missingSegments.unshift(api.basename(ancestor)); + ancestor = parent; + try { + probe?.(ancestor); + result = api.join(realpathSync.native(ancestor), ...missingSegments); + break; + } catch { + parent = api.dirname(ancestor); + } + } + } + rememberCanonicalGraphPath(resolved, result); + return result; +} + +/** Test-only visibility into whether a hot projection touched the filesystem. */ +export function setCanonicalGraphPathProbeForTest( + next: ((path: string) => void) | null, +): void { + probe = next; +} + +export function clearCanonicalGraphPathCacheForTest(): void { + canonicalPaths.clear(); + probe = null; +} diff --git a/packages/harness/src/core/canvas-cache.test.ts b/packages/harness/src/core/canvas-cache.test.ts index 8731b649f..cd293a823 100644 --- a/packages/harness/src/core/canvas-cache.test.ts +++ b/packages/harness/src/core/canvas-cache.test.ts @@ -4,6 +4,7 @@ import * as os from "node:os"; import * as path from "node:path"; import { clearExtractionCache, + ExtractionLaunchCancelledError, extractWorkflowGraphCached, fingerprintWorkflowSources, } from "./canvas-cache.js"; @@ -27,7 +28,9 @@ async function tmpWorkflow(): Promise { return dir; } afterEach(async () => { - await Promise.all(tmpDirs.splice(0).map((d) => fs.rm(d, { recursive: true, force: true }))); + await Promise.all( + tmpDirs.splice(0).map((d) => fs.rm(d, { recursive: true, force: true })), + ); }); beforeEach(() => clearExtractionCache()); @@ -81,7 +84,10 @@ describe("extractWorkflowGraphCached", () => { it("never caches a failure — 'npm install' fixes don't touch any .ts file, so a cached failure would never self-invalidate", async () => { const dir = await tmpWorkflow(); - const extract = vi.fn().mockResolvedValueOnce(FAIL).mockResolvedValueOnce(OK); + const extract = vi + .fn() + .mockResolvedValueOnce(FAIL) + .mockResolvedValueOnce(OK); const first = await extractWorkflowGraphCached(dir, extract); const second = await extractWorkflowGraphCached(dir, extract); @@ -102,18 +108,58 @@ describe("extractWorkflowGraphCached", () => { expect(extract).toHaveBeenCalledTimes(2); expect(b.cached).toBe(false); }); + + it("rechecks launch authorization after fingerprinting and never starts an unauthorized extractor", async () => { + const dir = await tmpWorkflow(); + const extract = vi.fn().mockResolvedValue(OK); + const beforeLaunchAuthorization = vi.fn(); + const authorizeBeforeLaunch = vi.fn(async () => false); + + await expect( + extractWorkflowGraphCached(dir, extract, { + beforeLaunchAuthorization, + authorizeBeforeLaunch, + }), + ).rejects.toBeInstanceOf(ExtractionLaunchCancelledError); + + expect(beforeLaunchAuthorization).toHaveBeenCalledOnce(); + expect(authorizeBeforeLaunch).toHaveBeenCalledOnce(); + expect(extract).not.toHaveBeenCalled(); + }); + + it("does not require launch authorization when serving a cache hit", async () => { + const dir = await tmpWorkflow(); + const extract = vi.fn().mockResolvedValue(OK); + await extractWorkflowGraphCached(dir, extract); + const beforeLaunchAuthorization = vi.fn(); + const authorizeBeforeLaunch = vi.fn(async () => false); + + const hit = await extractWorkflowGraphCached(dir, extract, { + beforeLaunchAuthorization, + authorizeBeforeLaunch, + }); + + expect(hit.cached).toBe(true); + expect(extract).toHaveBeenCalledOnce(); + expect(beforeLaunchAuthorization).not.toHaveBeenCalled(); + expect(authorizeBeforeLaunch).not.toHaveBeenCalled(); + }); }); describe("fingerprintWorkflowSources", () => { it("is stable for an unchanged tree and skips node_modules", async () => { const dir = await tmpWorkflow(); await fs.mkdir(path.join(dir, "node_modules", "dep"), { recursive: true }); - await fs.writeFile(path.join(dir, "node_modules", "dep", "index.ts"), "ignored"); + await fs.writeFile( + path.join(dir, "node_modules", "dep", "index.ts"), + "ignored", + ); const first = await fingerprintWorkflowSources(dir); const second = await fingerprintWorkflowSources(dir); expect(second).toBe(first); - expect(first.startsWith("1:")).toBe(true); // only the workflow's own index.ts counted + expect(first).toContain('"index.ts","regular"'); + expect(first).not.toContain("node_modules"); }); it("changes when a source is edited", async () => { diff --git a/packages/harness/src/core/canvas-cache.ts b/packages/harness/src/core/canvas-cache.ts index a11a6f55f..5be4bcece 100644 --- a/packages/harness/src/core/canvas-cache.ts +++ b/packages/harness/src/core/canvas-cache.ts @@ -13,25 +13,37 @@ * * Process-lifetime only; a server restart re-extracts once per workflow. */ -import * as fs from "node:fs/promises"; import * as path from "node:path"; -import { extractWorkflowGraph, type ExtractionResult, type ExtractionSuccess } from "./canvas-graph.js"; -import { listSourceFiles } from "./canvas-interconnections.js"; +import { + extractWorkflowGraph, + type ExtractionResult, + type ExtractionSuccess, +} from "./canvas-graph.js"; +import { + listSourceFilesWithObservations, + workflowSourceFileMetadata, +} from "./canvas-interconnections.js"; -/** `:` over the workflow's own sources. */ -export async function fingerprintWorkflowSources(root: string): Promise { - const files = await listSourceFiles(root); - let maxMtimeMs = 0; - for (const file of files) { - try { - const stat = await fs.stat(file); - if (stat.mtimeMs > maxMtimeMs) maxMtimeMs = stat.mtimeMs; - } catch { - // A file deleted mid-walk still counts toward the file count; the next - // fingerprint won't include it, which is invalidation working as intended. - } +/** Stable no-follow metadata over the workflow's bounded admitted sources. */ +export async function fingerprintWorkflowSources( + root: string, +): Promise { + const sourceSet = await listSourceFilesWithObservations(root); + const parts: string[] = []; + for (const observedPath of sourceSet.observedPaths) { + const metadata = await workflowSourceFileMetadata(root, observedPath); + parts.push( + JSON.stringify([ + path.relative(path.resolve(root), observedPath), + metadata.status, + metadata.size ?? null, + metadata.mtimeMs ?? null, + metadata.dev ?? null, + metadata.ino ?? null, + ]), + ); } - return `${files.length}:${maxMtimeMs}`; + return parts.join("\n"); } interface CacheEntry { @@ -51,6 +63,26 @@ export interface CachedExtraction { fingerprint: string; } +export interface CachedExtractionOptions { + /** + * Revalidates provenance in the same continuation that starts a cache-miss + * extractor. It intentionally does not run for a cache hit: returning an + * already-derived process-local value launches no project code. + */ + authorizeBeforeLaunch?: () => boolean | Promise; + /** Test/lifecycle hook immediately before the final authorization check. */ + beforeLaunchAuthorization?: () => void | Promise; +} + +/** A cache miss whose launch proof expired is cancellation, not extraction + * failure. Callers use this to avoid writing a misleading error render. */ +export class ExtractionLaunchCancelledError extends Error { + constructor() { + super("Agent extraction authorization expired before launch"); + this.name = "ExtractionLaunchCancelledError"; + } +} + /** * `extractWorkflowGraph` behind the fingerprint cache. The `extract` * parameter exists for tests only (inject a spy to prove hit/miss behavior). @@ -58,11 +90,22 @@ export interface CachedExtraction { export async function extractWorkflowGraphCached( sourceDir: string, extract: (dir: string) => Promise = extractWorkflowGraph, + options: CachedExtractionOptions = {}, ): Promise { const key = path.resolve(sourceDir); const fingerprint = await fingerprintWorkflowSources(key); const hit = cache.get(key); - if (hit && hit.fingerprint === fingerprint) return { result: hit.result, cached: true, fingerprint }; + if (hit && hit.fingerprint === fingerprint) { + return { result: hit.result, cached: true, fingerprint }; + } + + await options.beforeLaunchAuthorization?.(); + if ( + options.authorizeBeforeLaunch && + !(await options.authorizeBeforeLaunch()) + ) { + throw new ExtractionLaunchCancelledError(); + } const result = await extract(key); if (result.ok) cache.set(key, { fingerprint, result }); diff --git a/packages/harness/src/core/canvas-interconnections.test.ts b/packages/harness/src/core/canvas-interconnections.test.ts index 87a9bbcfe..a05aca9c6 100644 --- a/packages/harness/src/core/canvas-interconnections.test.ts +++ b/packages/harness/src/core/canvas-interconnections.test.ts @@ -7,6 +7,7 @@ import { detectAgentInvocations, detectStepCapabilities, detectWorkflowLaunches, + listSourceFilesWithObservations, } from "./canvas-interconnections.js"; const FIXTURES_DIR = path.join( @@ -159,6 +160,12 @@ ctx.sapiom.agents.launch({ definition: "async-child" }); }, ], warnings: [], + observedPaths: [ + dir, + path.join(dir, "nested"), + path.join(dir, "nested", "index.ts"), + ], + complete: true, }); }); @@ -309,10 +316,72 @@ function helper(agents: { launch(spec: unknown): unknown }) { await expect(detectAgentInvocations(dir, new Set())).resolves.toEqual({ invocations: [], warnings: [], + observedPaths: [dir, path.join(dir, "index.ts")], + complete: true, }); }); }); +describe("listSourceFilesWithObservations", () => { + it("bounds a broad empty-directory tree and marks the scan incomplete", async () => { + const dir = await tmpProject({ "index.ts": "export const value = 1;\n" }); + await Promise.all( + Array.from({ length: 20 }, (_, index) => + fs.mkdir(path.join(dir, `directory-${String(index).padStart(2, "0")}`)), + ), + ); + + const result = await listSourceFilesWithObservations(dir, { + maxDirectories: 4, + }); + + expect(result.complete).toBe(false); + expect( + result.observedPaths.filter((observed) => + result.files.includes(observed) ? false : true, + ), + ).toHaveLength(4); + expect(result.files).toEqual([path.join(dir, "index.ts")]); + }); + + it("uses a deterministic depth boundary instead of recursing indefinitely", async () => { + const dir = await tmpProject({ + "one/two/three/index.ts": "export const value = 1;\n", + }); + + const result = await listSourceFilesWithObservations(dir, { maxDepth: 1 }); + + expect(result.complete).toBe(false); + expect(result.files).toEqual([]); + expect(result.observedPaths).toEqual([dir, path.join(dir, "one")]); + }); + + it("bounds candidates and observations in one very large directory", async () => { + const dir = await tmpProject( + Object.fromEntries( + Array.from({ length: 40 }, (_, index) => [ + `source-${String(index).padStart(2, "0")}.ts`, + `export const value${index} = ${index};\n`, + ]), + ), + ); + + const first = await listSourceFilesWithObservations(dir, { + maxFiles: 4, + maxEntries: 6, + }); + const second = await listSourceFilesWithObservations(dir, { + maxFiles: 4, + maxEntries: 6, + }); + + expect(first.complete).toBe(false); + expect(first.files).toEqual([]); + expect(first.observedPaths).toEqual([dir]); + expect(second).toEqual(first); + }); +}); + describe("detectStepCapabilities", () => { it("attributes each ctx.sapiom.*() call to the step whose defineStep block it sits in", async () => { const dir = await tmpProject({ @@ -366,7 +435,7 @@ function categorize(input) { expect(caps).toEqual([{ capability: "rules.classify", fromStepId: null }]); }); - it("keeps the blocking run chip until the per-agent Canvas can render it as a relationship", async () => { + it("keeps the blocking run chip until the per-agent Canvas can render it as an invocation", async () => { const dir = await tmpProject({ "index.ts": ` const kickoff = defineStep({ diff --git a/packages/harness/src/core/canvas-interconnections.ts b/packages/harness/src/core/canvas-interconnections.ts index 6b587a872..453ad3615 100644 --- a/packages/harness/src/core/canvas-interconnections.ts +++ b/packages/harness/src/core/canvas-interconnections.ts @@ -17,9 +17,10 @@ * claim about what a step calls. * * This deliberately does not create a Program or TypeChecker. Supported direct - * calls are syntax-accurate (comments and strings cannot become relationships), + * calls are syntax-accurate (comments and strings cannot become invocations), * while dynamic targets are returned as explicit extraction warnings. */ +import { constants as fsConstants } from "node:fs"; import * as fs from "node:fs/promises"; import * as path from "node:path"; import ts from "typescript"; @@ -35,8 +36,11 @@ const SKIP_DIR_NAMES = new Set([ "build", ".sapiom", ]); -const SOURCE_EXTENSIONS = new Set([".ts", ".tsx"]); -const MAX_FILES_PER_WORKFLOW = 200; +const SOURCE_EXTENSIONS = new Set([".ts", ".tsx", ".mts", ".cts"]); +export const RELATIONSHIP_SCAN_MAX_FILES = 200; +export const RELATIONSHIP_SCAN_MAX_DIRECTORIES = 2_000; +export const RELATIONSHIP_SCAN_MAX_DEPTH = 16; +export const RELATIONSHIP_SCAN_MAX_ENTRIES = 10_000; const MAX_FILE_BYTES = 512 * 1024; // Matches `sapiom..(` chains — e.g. `ctx.sapiom.web.search(`, @@ -49,7 +53,7 @@ const CAPABILITY_CALL_PATTERN = // Async launches already render as launched-agent nodes on the per-agent // Canvas. Keep blocking `agents.run` in that Canvas's existing capability-chip -// projection until it gains a blocking relationship node of its own. +// projection until it gains a blocking invocation node of its own. const NON_CAPABILITY_CALLS = new Set([ "agents.launch", "orchestrations.launch", @@ -71,34 +75,321 @@ const STEP_NAME_PATTERN = /(? { +export interface WorkflowSourceFileSet { + files: string[]; + /** Directories plus candidate files whose metadata defines membership. */ + observedPaths: string[]; + /** False when an opaque path or a deterministic work cap hid sources. */ + complete: boolean; +} + +export interface WorkflowSourceWalkLimits { + maxFiles?: number; + maxDirectories?: number; + maxDepth?: number; + maxEntries?: number; +} + +export async function listSourceFilesWithObservations( + root: string, + limits: WorkflowSourceWalkLimits = {}, +): Promise { + const absoluteRoot = path.resolve(root); const files: string[] = []; - async function walk(dir: string): Promise { - if (files.length >= MAX_FILES_PER_WORKFLOW) return; - let entries: import("node:fs").Dirent[]; + const observedPaths: string[] = []; + const pending: Array<{ dir: string; depth: number }> = [ + { dir: absoluteRoot, depth: 0 }, + ]; + let directories = 0; + let entriesVisited = 0; + let complete = true; + const maxFiles = Math.max(1, limits.maxFiles ?? RELATIONSHIP_SCAN_MAX_FILES); + const maxDirectories = Math.max( + 1, + limits.maxDirectories ?? RELATIONSHIP_SCAN_MAX_DIRECTORIES, + ); + const maxDepth = Math.max(0, limits.maxDepth ?? RELATIONSHIP_SCAN_MAX_DEPTH); + const maxEntries = Math.max( + 1, + limits.maxEntries ?? RELATIONSHIP_SCAN_MAX_ENTRIES, + ); + + while (pending.length > 0) { + if (directories >= maxDirectories) { + complete = false; + break; + } + const current = pending.shift()!; + const { dir, depth } = current; + const entries: import("node:fs").Dirent[] = []; + let directoryTruncated = false; try { - entries = await fs.readdir(dir, { withFileTypes: true }); + const directory = await fs.opendir(dir); + try { + const remainingEntries = maxEntries - entriesVisited; + for (let index = 0; index < remainingEntries; index += 1) { + const entry = await directory.read(); + if (!entry) break; + entries.push(entry); + entriesVisited += 1; + } + // Do not perform an unbounded count merely to distinguish exactly-at- + // cap from over-cap. Treat the boundary conservatively as incomplete; + // the containing directory metadata still detects membership changes. + if (entriesVisited >= maxEntries) { + complete = false; + directoryTruncated = true; + } + } finally { + await directory.close().catch(() => {}); + } } catch { - return; + complete = false; + continue; + } + directories += 1; + observedPaths.push(dir); + if (directoryTruncated) { + // Filesystem directory iteration order is not portable. Never project a + // cap-sized prefix whose membership could differ across hosts/passes; + // keep the containing directory observation and discard this partial + // directory atomically. + entries.length = 0; } entries.sort((left, right) => left.name.localeCompare(right.name)); + if ( + dir !== absoluteRoot && + entries.some((entry) => entry.name === ".git") + ) { + continue; + } for (const entry of entries) { - if (files.length >= MAX_FILES_PER_WORKFLOW) return; + const candidate = path.join(dir, entry.name); if (entry.isDirectory()) { if (SKIP_DIR_NAMES.has(entry.name)) continue; - await walk(path.join(dir, entry.name)); + if (depth >= maxDepth) { + complete = false; + continue; + } + pending.push({ dir: candidate, depth: depth + 1 }); } else if (SOURCE_EXTENSIONS.has(path.extname(entry.name))) { - files.push(path.join(dir, entry.name)); + if (!entry.isFile() || entry.isSymbolicLink()) { + complete = false; + continue; + } + if (files.length >= maxFiles) { + complete = false; + continue; + } + files.push(candidate); + observedPaths.push(candidate); + } else if (entry.isSymbolicLink()) { + // A symlink may hide a directory of project sources. Never follow it, + // and keep the invocation projection explicitly degraded. + complete = false; } } + if (entriesVisited >= maxEntries) break; + } + return { files, observedPaths, complete }; +} + +export async function listSourceFiles(root: string): Promise { + return (await listSourceFilesWithObservations(root)).files; +} + +export interface WorkflowSourceReadHooks { + /** Deterministic race seam after initial admission but before open. */ + beforeOpen?: (file: string) => void | Promise; + /** Called only after bytes were read from the admitted opened handle. */ + onBytesRead?: (file: string, bytes: number) => void; +} + +export interface WorkflowSourceFileMetadata { + status: "regular" | "directory" | "absent" | "unreadable" | "inadmissible"; + size?: number; + mtimeMs?: number; + dev?: number; + ino?: number; +} + +function confinedSourceFile(root: string, file: string): boolean { + const relative = path.relative(root, file); + return ( + relative !== "" && + relative !== ".." && + !relative.startsWith(`..${path.sep}`) && + !path.isAbsolute(relative) + ); +} + +async function admittedSourceAncestors( + root: string, + file: string, +): Promise { + if (!confinedSourceFile(root, file)) return false; + const relativeDirectory = path.relative(root, path.dirname(file)); + let current = root; + for (const segment of [ + "", + ...relativeDirectory.split(path.sep).filter(Boolean), + ]) { + if (segment) current = path.join(current, segment); + let stat: import("node:fs").Stats; + try { + stat = await fs.lstat(current); + } catch { + return false; + } + if (!stat.isDirectory() || stat.isSymbolicLink()) return false; + if (current === root) continue; + try { + await fs.lstat(path.join(current, ".git")); + return false; + } catch (error) { + const code = (error as NodeJS.ErrnoException).code; + if (code !== "ENOENT" && code !== "ENOTDIR") return false; + } + } + return true; +} + +/** Metadata-only, no-follow admission used by cache and watcher fingerprints. */ +export async function workflowSourceFileMetadata( + root: string, + file: string, +): Promise { + const absoluteRoot = path.resolve(root); + const absoluteFile = path.resolve(file); + if (absoluteFile === absoluteRoot) { + try { + const stat = await fs.lstat(absoluteRoot); + return stat.isDirectory() && !stat.isSymbolicLink() + ? { + status: "directory", + size: stat.size, + mtimeMs: stat.mtimeMs, + dev: stat.dev, + ino: stat.ino, + } + : { status: "inadmissible" }; + } catch (error) { + const code = (error as NodeJS.ErrnoException).code; + return { + status: + code === "ENOENT" || code === "ENOTDIR" ? "absent" : "unreadable", + }; + } + } + if (!(await admittedSourceAncestors(absoluteRoot, absoluteFile))) { + return { status: "inadmissible" }; + } + try { + const stat = await fs.lstat(absoluteFile); + if (stat.isDirectory() && !stat.isSymbolicLink()) { + return { + status: "directory", + size: stat.size, + mtimeMs: stat.mtimeMs, + dev: stat.dev, + ino: stat.ino, + }; + } + if (!stat.isFile() || stat.isSymbolicLink()) { + return { status: "inadmissible" }; + } + return { + status: "regular", + size: stat.size, + mtimeMs: stat.mtimeMs, + dev: stat.dev, + ino: stat.ino, + }; + } catch (error) { + const code = (error as NodeJS.ErrnoException).code; + return { + status: code === "ENOENT" || code === "ENOTDIR" ? "absent" : "unreadable", + }; + } +} + +function sameOpenedSource( + expected: WorkflowSourceFileMetadata, + actual: import("node:fs").Stats, +): boolean { + return ( + expected.status === "regular" && + actual.isFile() && + !actual.isSymbolicLink() && + actual.dev === expected.dev && + actual.ino === expected.ino && + actual.size === expected.size && + actual.mtimeMs === expected.mtimeMs + ); +} + +/** Bounded opened-handle read that never follows a final or ancestor symlink. */ +export async function readWorkflowSourceFile( + root: string, + file: string, + hooks: WorkflowSourceReadHooks = {}, +): Promise { + const absoluteRoot = path.resolve(root); + const absoluteFile = path.resolve(file); + const admitted = await workflowSourceFileMetadata(absoluteRoot, absoluteFile); + if ( + admitted.status !== "regular" || + (admitted.size ?? MAX_FILE_BYTES + 1) > MAX_FILE_BYTES + ) { + return null; + } + await hooks.beforeOpen?.(absoluteFile); + let handle: Awaited> | null = null; + try { + handle = await fs.open( + absoluteFile, + fsConstants.O_RDONLY | + (fsConstants.O_NOFOLLOW ?? 0) | + (fsConstants.O_NONBLOCK ?? 0), + ); + const opened = await handle.stat(); + if ( + !sameOpenedSource(admitted, opened) || + !(await admittedSourceAncestors(absoluteRoot, absoluteFile)) + ) { + return null; + } + const chunks: Buffer[] = []; + let total = 0; + while (total <= MAX_FILE_BYTES) { + const chunk = Buffer.allocUnsafe( + Math.min(64 * 1024, MAX_FILE_BYTES + 1 - total), + ); + const { bytesRead } = await handle.read(chunk, 0, chunk.length, null); + if (bytesRead === 0) break; + total += bytesRead; + hooks.onBytesRead?.(absoluteFile, bytesRead); + if (total > MAX_FILE_BYTES) return null; + chunks.push(chunk.subarray(0, bytesRead)); + } + const final = await handle.stat(); + if ( + !sameOpenedSource(admitted, final) || + !(await admittedSourceAncestors(absoluteRoot, absoluteFile)) + ) { + return null; + } + return Buffer.concat(chunks, total).toString("utf8"); + } catch { + return null; + } finally { + await handle?.close().catch(() => {}); } - await walk(root); - return files; } // --- attribution: which step's defineStep(...) block a call sits in --------- @@ -228,6 +519,8 @@ export interface AgentInvocationDetectionWarning { export interface AgentInvocationScanResult { invocations: DetectedAgentInvocation[]; warnings: AgentInvocationDetectionWarning[]; + observedPaths: string[]; + complete: boolean; } export interface DetectedCapability { @@ -242,6 +535,10 @@ export interface WorkflowSourceScan { invocations: DetectedAgentInvocation[]; invocationWarnings: AgentInvocationDetectionWarning[]; capabilities: DetectedCapability[]; + /** Confined metadata paths actually considered by the bounded extractor. */ + observedPaths: string[]; + /** False when an opaque path or work cap prevented a complete scan. */ + complete: boolean; } interface SupportedNamespaces { @@ -523,7 +820,7 @@ function scanAgentInvocationsInFile( file: string, content: string, blocks: readonly StepBlock[], -): AgentInvocationScanResult { +): Pick { const sourceFile = ts.createSourceFile( path.basename(file), content, @@ -582,17 +879,17 @@ function evidenceOrder(left: SourceEvidence, right: SourceEvidence): number { export async function scanWorkflowSources( root: string, knownStepIds: ReadonlySet, + readHooks: WorkflowSourceReadHooks = {}, ): Promise { const invocations: DetectedAgentInvocation[] = []; const invocationWarnings: AgentInvocationDetectionWarning[] = []; const capabilities: DetectedCapability[] = []; - for (const file of await listSourceFiles(root)) { - let content: string; - try { - const stat = await fs.stat(file); - if (stat.size > MAX_FILE_BYTES) continue; - content = await fs.readFile(file, "utf8"); - } catch { + const sourceSet = await listSourceFilesWithObservations(root); + let complete = sourceSet.complete; + for (const file of sourceSet.files) { + const content = await readWorkflowSourceFile(root, file, readHooks); + if (content === null) { + complete = false; continue; } @@ -626,7 +923,14 @@ export async function scanWorkflowSources( const launches = invocations .filter((invocation) => invocation.mode === "async") .map(({ slug, fromStepId }) => ({ slug, fromStepId })); - return { launches, invocations, invocationWarnings, capabilities }; + return { + launches, + invocations, + invocationWarnings, + capabilities, + observedPaths: sourceSet.observedPaths, + complete, + }; } /** Direct agent invocations plus deterministic warnings for supported calls @@ -634,11 +938,14 @@ export async function scanWorkflowSources( export async function detectAgentInvocations( root: string, knownStepIds: ReadonlySet, + readHooks: WorkflowSourceReadHooks = {}, ): Promise { - const scan = await scanWorkflowSources(root, knownStepIds); + const scan = await scanWorkflowSources(root, knownStepIds, readHooks); return { invocations: scan.invocations, warnings: scan.invocationWarnings, + observedPaths: scan.observedPaths, + complete: scan.complete, }; } diff --git a/packages/harness/src/core/canvas-render.test.ts b/packages/harness/src/core/canvas-render.test.ts index af426fd83..2aff528f9 100644 --- a/packages/harness/src/core/canvas-render.test.ts +++ b/packages/harness/src/core/canvas-render.test.ts @@ -1,4 +1,4 @@ -import { describe, expect, it, afterEach, beforeEach } from "vitest"; +import { describe, expect, it, afterEach, beforeEach, vi } from "vitest"; import * as fs from "node:fs/promises"; import * as os from "node:os"; import * as path from "node:path"; @@ -13,7 +13,10 @@ import { type RenderableWorkflow, } from "./canvas-render.js"; -const FIXTURES_DIR = path.join(path.dirname(fileURLToPath(import.meta.url)), "__fixtures__"); +const FIXTURES_DIR = path.join( + path.dirname(fileURLToPath(import.meta.url)), + "__fixtures__", +); const ORDER_TRIAGE = path.join(FIXTURES_DIR, "order-triage"); const NO_DEFINITION = path.join(FIXTURES_DIR, "no-definition"); const HUB = path.join(FIXTURES_DIR, "hub"); @@ -27,7 +30,9 @@ async function tmpCwd(): Promise { } beforeEach(() => clearExtractionCache()); afterEach(async () => { - await Promise.all(tmpDirs.splice(0).map((d) => fs.rm(d, { recursive: true, force: true }))); + await Promise.all( + tmpDirs.splice(0).map((d) => fs.rm(d, { recursive: true, force: true })), + ); }); async function readRender(cwd: string, workflowPath: string): Promise { @@ -48,8 +53,13 @@ describe("slugForWorkflowPath", () => { describe("renderCanvasForSession", () => { it("renders the bound workflow's real step names into its own per-workflow render file — index.html is never touched", async () => { const cwd = await tmpCwd(); - const workflows: RenderableWorkflow[] = [{ path: ORDER_TRIAGE, name: "order-triage", definitionId: null }]; - const outcome = await renderCanvasForSession({ cwd, boundWorkflowPath: ORDER_TRIAGE }, workflows); + const workflows: RenderableWorkflow[] = [ + { path: ORDER_TRIAGE, name: "order-triage", definitionId: null }, + ]; + const outcome = await renderCanvasForSession( + { cwd, boundWorkflowPath: ORDER_TRIAGE }, + workflows, + ); expect(outcome.mode).toBe("single"); expect(outcome.workflowPath).toBe(ORDER_TRIAGE); @@ -58,25 +68,45 @@ describe("renderCanvasForSession", () => { expect(outcome.renderPath).toBe(renderFileFor(cwd, ORDER_TRIAGE)); const html = await readRender(cwd, ORDER_TRIAGE); - for (const step of ["intake", "classify", "route", "auto_resolve", "escalate"]) { + for (const step of [ + "intake", + "classify", + "route", + "auto_resolve", + "escalate", + ]) { expect(html).toContain(`>${step}<`); } - await expect(fs.access(path.join(cwd, CANVAS_DIR, "index.html"))).rejects.toThrow(); + await expect( + fs.access(path.join(cwd, CANVAS_DIR, "index.html")), + ).rejects.toThrow(); }); it("embeds the step-graph JSON block the Steps tab reads — parseable, with the real nodes/edges", async () => { const cwd = await tmpCwd(); - const workflows: RenderableWorkflow[] = [{ path: ORDER_TRIAGE, name: "order-triage", definitionId: null }]; - await renderCanvasForSession({ cwd, boundWorkflowPath: ORDER_TRIAGE }, workflows); + const workflows: RenderableWorkflow[] = [ + { path: ORDER_TRIAGE, name: "order-triage", definitionId: null }, + ]; + await renderCanvasForSession( + { cwd, boundWorkflowPath: ORDER_TRIAGE }, + workflows, + ); const html = await readRender(cwd, ORDER_TRIAGE); - const match = html.match(/" in any label from breaking out — // and it round-trips through JSON.parse back to real data. - const graph = JSON.parse(match![1]) as { nodes: { id: string }[]; edges: unknown[] }; - expect(graph.nodes.map((n) => n.id)).toEqual(expect.arrayContaining(["intake", "classify", "route"])); + const graph = JSON.parse(match![1]) as { + nodes: { id: string }[]; + edges: unknown[]; + }; + expect(graph.nodes.map((n) => n.id)).toEqual( + expect.arrayContaining(["intake", "classify", "route"]), + ); expect(graph.edges.length).toBeGreaterThan(0); }); @@ -105,26 +135,69 @@ describe("renderCanvasForSession", () => { it("serves the second render of an unchanged workflow from the extraction cache", async () => { const cwd = await tmpCwd(); - const workflows: RenderableWorkflow[] = [{ path: ORDER_TRIAGE, name: "order-triage", definitionId: null }]; - await renderCanvasForSession({ cwd, boundWorkflowPath: ORDER_TRIAGE }, workflows); - const second = await renderCanvasForSession({ cwd, boundWorkflowPath: ORDER_TRIAGE }, workflows); + const workflows: RenderableWorkflow[] = [ + { path: ORDER_TRIAGE, name: "order-triage", definitionId: null }, + ]; + await renderCanvasForSession( + { cwd, boundWorkflowPath: ORDER_TRIAGE }, + workflows, + ); + const second = await renderCanvasForSession( + { cwd, boundWorkflowPath: ORDER_TRIAGE }, + workflows, + ); expect(second.cachedExtraction).toBe(true); await expect(readRender(cwd, ORDER_TRIAGE)).resolves.toContain(">intake<"); }); + it("writes nothing when unprompted launch authorization expires after fingerprinting", async () => { + const cwd = await tmpCwd(); + const workflows: RenderableWorkflow[] = [ + { path: ORDER_TRIAGE, name: "order-triage", definitionId: null }, + ]; + const beforeExtractionLaunchAuthorization = vi.fn(); + const authorizeBeforeExtraction = vi.fn(async () => false); + + const outcome = await renderCanvasForSession( + { cwd, boundWorkflowPath: ORDER_TRIAGE }, + workflows, + { + beforeExtractionLaunchAuthorization, + authorizeBeforeExtraction, + }, + ); + + expect(beforeExtractionLaunchAuthorization).toHaveBeenCalledOnce(); + expect(authorizeBeforeExtraction).toHaveBeenCalledOnce(); + expect(outcome).toMatchObject({ + mode: "single", + authorizationExpired: true, + extractionFailed: [], + }); + await expect(fs.access(renderFileFor(cwd, ORDER_TRIAGE))).rejects.toThrow(); + }); + it("renders an old-SDK (legacy-branded) workflow — the dual-brand extraction end to end", async () => { const cwd = await tmpCwd(); - const workflows: RenderableWorkflow[] = [{ path: LEGACY_FLOW, name: "legacy-flow", definitionId: null }]; - const outcome = await renderCanvasForSession({ cwd, boundWorkflowPath: LEGACY_FLOW }, workflows); + const workflows: RenderableWorkflow[] = [ + { path: LEGACY_FLOW, name: "legacy-flow", definitionId: null }, + ]; + const outcome = await renderCanvasForSession( + { cwd, boundWorkflowPath: LEGACY_FLOW }, + workflows, + ); expect(outcome.extractionFailed).toEqual([]); const html = await readRender(cwd, LEGACY_FLOW); - for (const step of ["receive", "confirm", "award"]) expect(html).toContain(`>${step}<`); + for (const step of ["receive", "confirm", "award"]) + expect(html).toContain(`>${step}<`); }); it("includes detected launches as dashed launched-workflow nodes in the bound workflow's own diagram", async () => { const cwd = await tmpCwd(); - const workflows: RenderableWorkflow[] = [{ path: HUB, name: "hub", definitionId: null }]; + const workflows: RenderableWorkflow[] = [ + { path: HUB, name: "hub", definitionId: null }, + ]; await renderCanvasForSession({ cwd, boundWorkflowPath: HUB }, workflows); const html = await readRender(cwd, HUB); @@ -137,8 +210,13 @@ describe("renderCanvasForSession", () => { it("degrades to an honest error panel when the bound workflow fails to extract — never crashes, never falls back to an LLM prompt", async () => { const cwd = await tmpCwd(); - const workflows: RenderableWorkflow[] = [{ path: NO_DEFINITION, name: "broken-flow", definitionId: null }]; - const outcome = await renderCanvasForSession({ cwd, boundWorkflowPath: NO_DEFINITION }, workflows); + const workflows: RenderableWorkflow[] = [ + { path: NO_DEFINITION, name: "broken-flow", definitionId: null }, + ]; + const outcome = await renderCanvasForSession( + { cwd, boundWorkflowPath: NO_DEFINITION }, + workflows, + ); expect(outcome.mode).toBe("single"); expect(outcome.extractionFailed).toEqual(["broken-flow"]); @@ -154,8 +232,13 @@ describe("renderCanvasForSession", () => { it("is a cheap no-op when unbound: no extraction, no write — the server serves the empty state itself", async () => { const cwd = await tmpCwd(); - const workflows: RenderableWorkflow[] = [{ path: ORDER_TRIAGE, name: "order-triage", definitionId: null }]; - const outcome = await renderCanvasForSession({ cwd, boundWorkflowPath: null }, workflows); + const workflows: RenderableWorkflow[] = [ + { path: ORDER_TRIAGE, name: "order-triage", definitionId: null }, + ]; + const outcome = await renderCanvasForSession( + { cwd, boundWorkflowPath: null }, + workflows, + ); expect(outcome).toEqual({ mode: "empty", extractionFailed: [] }); await expect(fs.access(path.join(cwd, CANVAS_DIR))).rejects.toThrow(); // nothing written at all @@ -163,8 +246,13 @@ describe("renderCanvasForSession", () => { it("treats a boundWorkflowPath that matches no known workflow as unbound", async () => { const cwd = await tmpCwd(); - const workflows: RenderableWorkflow[] = [{ path: ORDER_TRIAGE, name: "order-triage", definitionId: null }]; - const outcome = await renderCanvasForSession({ cwd, boundWorkflowPath: "/no/such/workflow" }, workflows); + const workflows: RenderableWorkflow[] = [ + { path: ORDER_TRIAGE, name: "order-triage", definitionId: null }, + ]; + const outcome = await renderCanvasForSession( + { cwd, boundWorkflowPath: "/no/such/workflow" }, + workflows, + ); expect(outcome.mode).toBe("empty"); }); @@ -174,8 +262,13 @@ describe("renderCanvasForSession", () => { const notADir = path.join(parent, "not-a-directory"); await fs.writeFile(notADir, "x"); - const workflows: RenderableWorkflow[] = [{ path: ORDER_TRIAGE, name: "order-triage", definitionId: null }]; - const outcome = await renderCanvasForSession({ cwd: notADir, boundWorkflowPath: ORDER_TRIAGE }, workflows); + const workflows: RenderableWorkflow[] = [ + { path: ORDER_TRIAGE, name: "order-triage", definitionId: null }, + ]; + const outcome = await renderCanvasForSession( + { cwd: notADir, boundWorkflowPath: ORDER_TRIAGE }, + workflows, + ); expect(outcome.mode).toBe("single"); expect(outcome.writeError).toBeTruthy(); }); @@ -188,11 +281,17 @@ describe("renderCanvasForSession", () => { const renderPath = renderFileFor(cwd, NO_DEFINITION); await fs.mkdir(path.dirname(renderPath), { recursive: true }); await fs.writeFile(renderPath, GOOD_RENDER, "utf8"); - const workflows: RenderableWorkflow[] = [{ path: NO_DEFINITION, name: "broken-flow", definitionId: null }]; + const workflows: RenderableWorkflow[] = [ + { path: NO_DEFINITION, name: "broken-flow", definitionId: null }, + ]; - const outcome = await renderCanvasForSession({ cwd, boundWorkflowPath: NO_DEFINITION }, workflows, { - preserveExistingOnFailure: true, - }); + const outcome = await renderCanvasForSession( + { cwd, boundWorkflowPath: NO_DEFINITION }, + workflows, + { + preserveExistingOnFailure: true, + }, + ); expect(outcome.extractionFailed).toEqual(["broken-flow"]); expect(outcome.preservedExisting).toBe(true); @@ -201,11 +300,17 @@ describe("renderCanvasForSession", () => { it("still writes the honest error page when nothing exists to preserve", async () => { const cwd = await tmpCwd(); - const workflows: RenderableWorkflow[] = [{ path: NO_DEFINITION, name: "broken-flow", definitionId: null }]; + const workflows: RenderableWorkflow[] = [ + { path: NO_DEFINITION, name: "broken-flow", definitionId: null }, + ]; - const outcome = await renderCanvasForSession({ cwd, boundWorkflowPath: NO_DEFINITION }, workflows, { - preserveExistingOnFailure: true, - }); + const outcome = await renderCanvasForSession( + { cwd, boundWorkflowPath: NO_DEFINITION }, + workflows, + { + preserveExistingOnFailure: true, + }, + ); expect(outcome.preservedExisting).toBeUndefined(); expect(await readRender(cwd, NO_DEFINITION)).toContain("render failed"); @@ -216,9 +321,14 @@ describe("renderCanvasForSession", () => { const renderPath = renderFileFor(cwd, NO_DEFINITION); await fs.mkdir(path.dirname(renderPath), { recursive: true }); await fs.writeFile(renderPath, GOOD_RENDER, "utf8"); - const workflows: RenderableWorkflow[] = [{ path: NO_DEFINITION, name: "broken-flow", definitionId: null }]; + const workflows: RenderableWorkflow[] = [ + { path: NO_DEFINITION, name: "broken-flow", definitionId: null }, + ]; - const outcome = await renderCanvasForSession({ cwd, boundWorkflowPath: NO_DEFINITION }, workflows); + const outcome = await renderCanvasForSession( + { cwd, boundWorkflowPath: NO_DEFINITION }, + workflows, + ); expect(outcome.preservedExisting).toBeUndefined(); expect(await readRender(cwd, NO_DEFINITION)).toContain("render failed"); @@ -227,7 +337,11 @@ describe("renderCanvasForSession", () => { }); describe("deriveWorkflowCanvas (the session-free entry point behind IA-01)", () => { - const ORDER: RenderableWorkflow = { path: ORDER_TRIAGE, name: "order-triage", definitionId: null }; + const ORDER: RenderableWorkflow = { + path: ORDER_TRIAGE, + name: "order-triage", + definitionId: null, + }; it("produces the byte-identical document the session-bound render writes to disk", async () => { // The proof that the workflow-keyed route (server/workflow-graph.ts) and @@ -235,7 +349,9 @@ describe("deriveWorkflowCanvas (the session-free entry point behind IA-01)", () // both sides come out of this one derivation, so a board read by agent path // IS the board a bound session sees. const cwd = await tmpCwd(); - await renderCanvasForSession({ cwd, boundWorkflowPath: ORDER_TRIAGE }, [ORDER]); + await renderCanvasForSession({ cwd, boundWorkflowPath: ORDER_TRIAGE }, [ + ORDER, + ]); const written = await readRender(cwd, ORDER_TRIAGE); const derived = await deriveWorkflowCanvas(ORDER); @@ -249,11 +365,17 @@ describe("deriveWorkflowCanvas (the session-free entry point behind IA-01)", () it("writes nothing — no render file, no canvas dir", async () => { const cwd = await tmpCwd(); await deriveWorkflowCanvas(ORDER); - await expect(fs.stat(path.join(cwd, CANVAS_DIR))).rejects.toMatchObject({ code: "ENOENT" }); + await expect(fs.stat(path.join(cwd, CANVAS_DIR))).rejects.toMatchObject({ + code: "ENOENT", + }); }); it("reports an extraction failure as status 'error' with the reason, never a throw", async () => { - const derived = await deriveWorkflowCanvas({ path: NO_DEFINITION, name: "broken-flow", definitionId: null }); + const derived = await deriveWorkflowCanvas({ + path: NO_DEFINITION, + name: "broken-flow", + definitionId: null, + }); expect(derived.status).toBe("error"); expect(derived.graph).toBeNull(); @@ -268,7 +390,9 @@ describe("dependencies not installed yet (fresh scaffold, pre-`npm install`)", ( // not-yet-installed project. (The repo's own __fixtures__ resolve the SDK via // the monorepo's node_modules, so they are NOT deps-missing.) async function depsMissingProject(): Promise { - const dir = await fs.mkdtemp(path.join(os.tmpdir(), "canvas-render-noinstall-")); + const dir = await fs.mkdtemp( + path.join(os.tmpdir(), "canvas-render-noinstall-"), + ); tmpDirs.push(dir); await fs.writeFile( path.join(dir, "index.ts"), @@ -281,9 +405,14 @@ describe("dependencies not installed yet (fresh scaffold, pre-`npm install`)", ( it("shows the calm 'preparing' placeholder — not the esbuild error — and flags depsMissing", async () => { const cwd = await tmpCwd(); const project = await depsMissingProject(); - const workflows: RenderableWorkflow[] = [{ path: project, name: "enrich-list-leads", definitionId: null }]; + const workflows: RenderableWorkflow[] = [ + { path: project, name: "enrich-list-leads", definitionId: null }, + ]; - const outcome = await renderCanvasForSession({ cwd, boundWorkflowPath: project }, workflows); + const outcome = await renderCanvasForSession( + { cwd, boundWorkflowPath: project }, + workflows, + ); expect(outcome.depsMissing).toBe(true); expect(outcome.extractionFailed).toEqual([]); @@ -302,11 +431,17 @@ describe("dependencies not installed yet (fresh scaffold, pre-`npm install`)", ( const renderPath = renderFileFor(cwd, project); await fs.mkdir(path.dirname(renderPath), { recursive: true }); await fs.writeFile(renderPath, "", "utf8"); - const workflows: RenderableWorkflow[] = [{ path: project, name: "enrich-list-leads", definitionId: null }]; - - const outcome = await renderCanvasForSession({ cwd, boundWorkflowPath: project }, workflows, { - preserveExistingOnFailure: true, - }); + const workflows: RenderableWorkflow[] = [ + { path: project, name: "enrich-list-leads", definitionId: null }, + ]; + + const outcome = await renderCanvasForSession( + { cwd, boundWorkflowPath: project }, + workflows, + { + preserveExistingOnFailure: true, + }, + ); expect(outcome.depsMissing).toBe(true); expect(outcome.preservedExisting).toBe(true); @@ -316,11 +451,17 @@ describe("dependencies not installed yet (fresh scaffold, pre-`npm install`)", ( it("surfaceErrorOnMissingDeps forces the honest error (the install-timeout safety valve)", async () => { const cwd = await tmpCwd(); const project = await depsMissingProject(); - const workflows: RenderableWorkflow[] = [{ path: project, name: "enrich-list-leads", definitionId: null }]; - - const outcome = await renderCanvasForSession({ cwd, boundWorkflowPath: project }, workflows, { - surfaceErrorOnMissingDeps: true, - }); + const workflows: RenderableWorkflow[] = [ + { path: project, name: "enrich-list-leads", definitionId: null }, + ]; + + const outcome = await renderCanvasForSession( + { cwd, boundWorkflowPath: project }, + workflows, + { + surfaceErrorOnMissingDeps: true, + }, + ); // Extraction runs and genuinely fails to resolve the SDK — the error panel // (and its Retry/Ask actions) is restored, and depsMissing is NOT set. @@ -331,11 +472,16 @@ describe("dependencies not installed yet (fresh scaffold, pre-`npm install`)", ( }); describe("deterministic enrichment merged into renders", () => { - const workflows: RenderableWorkflow[] = [{ path: ORDER_TRIAGE, name: "order-triage", definitionId: null }]; + const workflows: RenderableWorkflow[] = [ + { path: ORDER_TRIAGE, name: "order-triage", definitionId: null }, + ]; it("always annotates a successful render with a derived summary — no cache, no AI, no stale chip", async () => { const cwd = await tmpCwd(); - const outcome = await renderCanvasForSession({ cwd, boundWorkflowPath: ORDER_TRIAGE }, workflows); + const outcome = await renderCanvasForSession( + { cwd, boundWorkflowPath: ORDER_TRIAGE }, + workflows, + ); expect(outcome.enrichmentApplied).toBe(true); const html = await readRender(cwd, ORDER_TRIAGE); @@ -343,23 +489,36 @@ describe("deterministic enrichment merged into renders", () => { expect(html).toContain("3 steps · 1 branch point · 2 success outcomes"); expect(html).not.toContain("stale — Refresh"); // Never writes an enrichment cache dir — the annotation is recomputed each render. - await expect(fs.access(path.join(cwd, CANVAS_DIR, "cache"))).rejects.toThrow(); + await expect( + fs.access(path.join(cwd, CANVAS_DIR, "cache")), + ).rejects.toThrow(); }); it("is byte-stable across re-renders — same graph in, identical annotated HTML out", async () => { const cwd = await tmpCwd(); - await renderCanvasForSession({ cwd, boundWorkflowPath: ORDER_TRIAGE }, workflows); + await renderCanvasForSession( + { cwd, boundWorkflowPath: ORDER_TRIAGE }, + workflows, + ); const first = await readRender(cwd, ORDER_TRIAGE); clearExtractionCache(); - await renderCanvasForSession({ cwd, boundWorkflowPath: ORDER_TRIAGE }, workflows); + await renderCanvasForSession( + { cwd, boundWorkflowPath: ORDER_TRIAGE }, + workflows, + ); expect(await readRender(cwd, ORDER_TRIAGE)).toBe(first); }); it("carries no annotations on an extraction-failure panel", async () => { const cwd = await tmpCwd(); - const broken: RenderableWorkflow[] = [{ path: NO_DEFINITION, name: "broken-flow", definitionId: null }]; + const broken: RenderableWorkflow[] = [ + { path: NO_DEFINITION, name: "broken-flow", definitionId: null }, + ]; - const outcome = await renderCanvasForSession({ cwd, boundWorkflowPath: NO_DEFINITION }, broken); + const outcome = await renderCanvasForSession( + { cwd, boundWorkflowPath: NO_DEFINITION }, + broken, + ); expect(outcome.enrichmentApplied).toBeUndefined(); const html = await readRender(cwd, NO_DEFINITION); expect(html).toContain("render failed"); diff --git a/packages/harness/src/core/canvas-render.ts b/packages/harness/src/core/canvas-render.ts index 946b59fb4..b620bcaf8 100644 --- a/packages/harness/src/core/canvas-render.ts +++ b/packages/harness/src/core/canvas-render.ts @@ -27,7 +27,10 @@ import * as path from "node:path"; import { CANVAS_RENDERS_DIR } from "../shared/types.js"; import { agentDepsInstalled } from "./agent-deps.js"; import { renderCanvasDocument } from "./canvas-template.js"; -import { extractWorkflowGraphCached } from "./canvas-cache.js"; +import { + ExtractionLaunchCancelledError, + extractWorkflowGraphCached, +} from "./canvas-cache.js"; import type { CanvasGraph } from "./canvas-graph.js"; import type { CanvasEnrichment } from "./canvas-enrichment.js"; import { deriveEnrichment } from "./canvas-derive.js"; @@ -63,13 +66,20 @@ export function slugForWorkflowPath(workflowPath: string): string { .toLowerCase() .replace(/[^a-z0-9]+/g, "-") .replace(/^-+|-+$/g, "") || "workflow"; - const hash = createHash("sha256").update(path.resolve(workflowPath)).digest("hex").slice(0, 8); + const hash = createHash("sha256") + .update(path.resolve(workflowPath)) + .digest("hex") + .slice(0, 8); return `${base}-${hash}`; } /** Absolute path of `workflowPath`'s render file under `cwd`'s canvas dir. */ export function renderFileFor(cwd: string, workflowPath: string): string { - return path.join(cwd, CANVAS_RENDERS_DIR, `${slugForWorkflowPath(workflowPath)}.html`); + return path.join( + cwd, + CANVAS_RENDERS_DIR, + `${slugForWorkflowPath(workflowPath)}.html`, + ); } export interface CanvasRenderOutcome { @@ -96,6 +106,9 @@ export interface CanvasRenderOutcome { * esbuild error panel. The server arms an install watcher on this to * re-render once dependencies land — see server/index.ts. */ depsMissing?: boolean; + /** The unprompted launch proof expired after async dependency/fingerprint + * work. No extractor ran and no render file was written. */ + authorizationExpired?: boolean; } export interface RenderCanvasOptions { @@ -117,6 +130,11 @@ export interface RenderCanvasOptions { * wait forever. Normal renders leave it off. */ surfaceErrorOnMissingDeps?: boolean; + /** Unprompted-render provenance check, evaluated on a cache miss immediately + * before the extractor child starts. Manual Visualize calls omit it. */ + authorizeBeforeExtraction?: () => boolean | Promise; + /** Lifecycle/test hook immediately before the launch-boundary recheck. */ + beforeExtractionLaunchAuthorization?: () => void | Promise; } function badgesFor(workflow: RenderableWorkflow): string[] { @@ -146,11 +164,22 @@ function buildSingleBody( ): string { if (!graph) { return assembleCanvasBody({ - panels: [buildErrorPanelHtml(workflow.name, reason ?? "unknown extraction failure")], + panels: [ + buildErrorPanelHtml( + workflow.name, + reason ?? "unknown extraction failure", + ), + ], }); } return assembleCanvasBody({ - panels: [buildWorkflowPanelHtml(graph, { title: workflow.name, badges: badgesFor(workflow) }, enrichment)], + panels: [ + buildWorkflowPanelHtml( + graph, + { title: workflow.name, badges: badgesFor(workflow) }, + enrichment, + ), + ], }); } @@ -165,7 +194,9 @@ export async function renderCanvasForSession( workflows: readonly RenderableWorkflow[], options: RenderCanvasOptions = {}, ): Promise { - const bound = session.boundWorkflowPath ? workflows.find((w) => w.path === session.boundWorkflowPath) : undefined; + const bound = session.boundWorkflowPath + ? workflows.find((w) => w.path === session.boundWorkflowPath) + : undefined; if (!bound) { return { mode: "empty", extractionFailed: [] }; } @@ -184,7 +215,7 @@ export interface WorkflowCanvasDerivation { /** "ok": extracted and rendered. "preparing": dependencies aren't installed * yet, so extraction was skipped and the calm placeholder was built * instead. "error": extraction ran and failed — the honest error panel. */ - status: "ok" | "preparing" | "error"; + status: "ok" | "preparing" | "error" | "cancelled"; graph: CanvasGraph | null; enrichment: CanvasEnrichment | null; /** The extraction failure reason ("error" only); null otherwise. */ @@ -213,7 +244,10 @@ export async function deriveWorkflowCanvas( // it self-resolves when install finishes. So skip extraction entirely and // build a calm "preparing" placeholder. A bundle failure WITH deps installed // stays a genuine error (below). - if (!options.surfaceErrorOnMissingDeps && !(await agentDepsInstalled(workflow.path))) { + if ( + !options.surfaceErrorOnMissingDeps && + !(await agentDepsInstalled(workflow.path)) + ) { return { status: "preparing", graph: null, @@ -224,14 +258,38 @@ export async function deriveWorkflowCanvas( }; } - const { result, cached } = await extractWorkflowGraphCached(workflow.path); + let extracted: Awaited>; + try { + extracted = await extractWorkflowGraphCached(workflow.path, undefined, { + authorizeBeforeLaunch: options.authorizeBeforeExtraction, + beforeLaunchAuthorization: options.beforeExtractionLaunchAuthorization, + }); + } catch (error) { + if (!(error instanceof ExtractionLaunchCancelledError)) throw error; + return { + status: "cancelled", + graph: null, + enrichment: null, + reason: null, + cached: false, + // This document is intentionally never written by the render path. It + // keeps the session-free return shape total for defensive callers. + document: renderCanvasDocument(buildPreparingPanelHtml(workflow.name)), + }; + } + const { result, cached } = extracted; // Enrichment only decorates a successful extraction — deriving annotations // for an error panel whose steps we can't even show would be noise. Derived // deterministically from the freshly extracted graph, so it's always in sync // with the diagram and can never go stale. const enrichment = result.ok ? deriveEnrichment(result.graph) : null; - const body = buildSingleBody(workflow, result.ok ? result.graph : null, result.ok ? null : result.reason, enrichment); + const body = buildSingleBody( + workflow, + result.ok ? result.graph : null, + result.ok ? null : result.reason, + enrichment, + ); return { status: result.ok ? "ok" : "error", @@ -259,6 +317,15 @@ export async function renderWorkflowRenderFile( const renderPath = renderFileFor(cwd, bound.path); const derived = await deriveWorkflowCanvas(bound, options); + if (derived.status === "cancelled") { + return { + mode: "single", + workflowPath: bound.path, + extractionFailed: [], + authorizationExpired: true, + }; + } + const outcome: CanvasRenderOutcome = { mode: "single", workflowPath: bound.path, @@ -274,7 +341,11 @@ export async function renderWorkflowRenderFile( // good) diagram for this workflow — an agent mid-edit whose sources are // transiently un-buildable, or deps that went missing. `depsMissing` is still // flagged above so the server's install watcher re-renders when they return. - if (options.preserveExistingOnFailure && derived.status !== "ok" && (await pathExists(renderPath))) { + if ( + options.preserveExistingOnFailure && + derived.status !== "ok" && + (await pathExists(renderPath)) + ) { outcome.preservedExisting = true; return outcome; } diff --git a/packages/harness/src/core/definition-name.ts b/packages/harness/src/core/definition-name.ts index ee4a18def..5a380cb8a 100644 --- a/packages/harness/src/core/definition-name.ts +++ b/packages/harness/src/core/definition-name.ts @@ -12,7 +12,10 @@ * a bundle error, the check process timing out) comes back as null and the * caller falls back to a weaker name source. */ -import { extractWorkflowGraphCached } from "./canvas-cache.js"; +import { + extractWorkflowGraphCached, + type CachedExtractionOptions, +} from "./canvas-cache.js"; import { listSourceFiles } from "./canvas-interconnections.js"; export type ManifestNameInspection = @@ -27,8 +30,9 @@ export type ManifestNameInspection = * (core/canvas-cache.ts), so a re-run is free to succeed — and several causes * do resolve on their own terms: dependencies get installed, a check process * that crashed or timed out under load succeeds on the next attempt. None of - * those fire the graph watcher, which only reacts to `.ts`/`.tsx` outside - * ignored directories, so the caller must keep offering a manual retry. + * those fire the graph watcher, which only reacts to supported TypeScript + * source files outside ignored directories, so the caller must keep offering + * a manual retry. * * `reason` cannot answer this — it is free-form text assembled from an agent's * own error message, a stderr tail, or a timeout string — so the only claim @@ -49,6 +53,8 @@ async function couldStillBeNamed(projectDir: string): Promise { } } +export type ManifestNameInspectionOptions = CachedExtractionOptions; + /** * Inspect the declared manifest name while preserving the difference between * a valid unnamed agent and an extraction failure. Inventory uses the richer @@ -58,9 +64,13 @@ async function couldStillBeNamed(projectDir: string): Promise { export async function inspectManifestName( projectDir: string, extract: typeof extractWorkflowGraphCached = extractWorkflowGraphCached, + options: ManifestNameInspectionOptions = {}, ): Promise { try { - const { result } = await extract(projectDir); + const { result } = + options.authorizeBeforeLaunch || options.beforeLaunchAuthorization + ? await extract(projectDir, undefined, options) + : await extract(projectDir); if (!result.ok) { if (result.code === "NO_DEFINITION") return { status: "absent" }; return { diff --git a/packages/harness/src/core/system-graph-inventory.test.ts b/packages/harness/src/core/system-graph-inventory.test.ts index 9f10ea642..02f700bd1 100644 --- a/packages/harness/src/core/system-graph-inventory.test.ts +++ b/packages/harness/src/core/system-graph-inventory.test.ts @@ -3,7 +3,7 @@ import * as os from "node:os"; import * as path from "node:path"; import { describe, expect, it, vi } from "vitest"; -import type { WorkflowInfo } from "../shared/types.js"; +import type { RegistryWorkflowInfo as WorkflowInfo } from "./workflow-registry.js"; import { dirtyGraphSourceRoots, graphSourceRootsWithinScope, @@ -13,8 +13,11 @@ import { type HarnessRegistryInventoryProviderOptions, type WorkspaceScope, } from "./system-graph-inventory.js"; -import type { ManifestNameInspection } from "./definition-name.js"; import { workspaceRelativeLocalKey } from "../shared/system-graph.js"; +import type { + ManifestNameInspection, + ManifestNameInspectionOptions, +} from "./definition-name.js"; const WORKSPACE = "/private/workspaces/acme"; const SCOPE: WorkspaceScope = { @@ -33,6 +36,7 @@ function workflow( path: relativePath ? `${WORKSPACE}/${relativePath}` : WORKSPACE, definitionId: definitionSlug ? 1 : null, definitionSlug, + markerPresent: true, source: "scan", ...overrides, }; @@ -45,7 +49,28 @@ function provider( return new HarnessRegistryInventoryProvider({ listWorkflows: () => workflows, fingerprintSource: async (sourceRoot) => `fingerprint:${sourceRoot}`, + revalidateMarker: async () => true, ...options, + inventorySnapshot: + options.inventorySnapshot ?? + (async (scope) => ({ + workflows, + status: await (options.inventoryStatus?.(scope) ?? "complete"), + generation: 1, + canonicalScopeRoot: scope.root, + canonicalWorkflowRoots: workflows.map((item) => ({ + workflowPath: item.path, + canonicalRoot: item.path, + identityEvidence: Object.prototype.hasOwnProperty.call( + item, + "sourceDefinitionName", + ) + ? ("source" as const) + : item.markerPresent === true + ? ("marker" as const) + : ("unknown" as const), + })), + })), }); } @@ -60,6 +85,190 @@ async function enrich( } describe("HarnessRegistryInventoryProvider", () => { + it("uses syntax-proven source identity immediately without extraction", async () => { + const inspectManifestName = vi.fn(async () => { + throw new Error("source-only rows must never execute extraction"); + }); + const inventory = provider( + [ + workflow("Billing package", "billing", "old-marker", { + sourceDefinitionName: "CurrentSourceName", + }), + ], + { + inspectManifestName, + inventoryStatus: async () => "degraded" as const, + }, + ); + + const result = await inventory.listAgents(SCOPE); + + expect(result.inventory).toMatchObject({ + status: "degraded", + agents: [ + { + agentKey: "CurrentSourceName", + identityStatus: "canonical", + }, + ], + }); + expect(result.context[0]?.resolutionAliases).toContain("old-marker"); + expect(result.discoveryComplete).toBe(false); + expect(result.startEnrichment).toBeUndefined(); + expect(inspectManifestName).not.toHaveBeenCalled(); + }); + + it("never extracts a markerless manual row even after discovery is complete", async () => { + const inspectManifestName = vi.fn(async () => ({ + status: "found" as const, + name: "Executed", + })); + let status: "complete" | "degraded" = "degraded"; + const inventory = provider( + [ + workflow("Pending", "pending", null, { + markerPresent: undefined, + source: "connect", + }), + ], + { inspectManifestName, inventoryStatus: () => status }, + ); + + const degraded = await inventory.listAgents(SCOPE); + degraded.startEnrichment?.(); + await Promise.resolve(); + expect(degraded.startEnrichment).toBeUndefined(); + expect(inspectManifestName).not.toHaveBeenCalled(); + + status = "complete"; + const complete = await inventory.listAgents(SCOPE); + expect(complete.startEnrichment).toBeUndefined(); + complete.startEnrichment?.(); + expect(inspectManifestName).not.toHaveBeenCalled(); + }); + + it("enriches a markerless row with retained cloud-link authorization", async () => { + const inspectManifestName = vi.fn(async () => ({ + status: "found" as const, + name: "CloudLinked", + })); + const inventory = provider( + [ + workflow("Linked", "linked", "old-alias", { + markerPresent: undefined, + source: "connect", + }), + ], + { inspectManifestName }, + ); + + const initial = await inventory.listAgents(SCOPE); + expect(initial.startEnrichment).toBeTypeOf("function"); + initial.startEnrichment?.(); + await vi.waitFor(() => expect(inspectManifestName).toHaveBeenCalledOnce()); + await expect(inventory.listAgents(SCOPE)).resolves.toMatchObject({ + discoveryComplete: true, + inventory: { + agents: [{ agentKey: "CloudLinked", identityStatus: "canonical" }], + }, + }); + }); + + it("enriches a marker-proven row despite unrelated degraded discovery", async () => { + const inspectManifestName = vi.fn(async () => ({ + status: "found" as const, + name: "CurrentSource", + })); + const inventory = provider( + [ + workflow("Linked", "linked", null, { + markerPresent: true, + }), + ], + { + inspectManifestName, + inventoryStatus: () => "degraded", + }, + ); + + const degraded = await inventory.listAgents(SCOPE); + expect(degraded.inventory.status).toBe("degraded"); + expect(degraded.startEnrichment).toBeTypeOf("function"); + degraded.startEnrichment?.(); + await vi.waitFor(() => expect(inspectManifestName).toHaveBeenCalledOnce()); + }); + + it("prefers one atomic inventory snapshot over independently racing reads", async () => { + const listWorkflows = vi.fn(() => [workflow("wrong", "wrong", "wrong")]); + const inventoryStatus = vi.fn(() => "complete" as const); + const inventory = new HarnessRegistryInventoryProvider({ + listWorkflows, + inventoryStatus, + inventorySnapshot: () => ({ + workflows: [ + workflow("Atomic", "atomic", null, { + sourceDefinitionName: "atomic-source", + }), + ], + status: "degraded", + generation: 7, + }), + }); + + const result = await inventory.listAgents(SCOPE); + + expect(result.inventory).toMatchObject({ + status: "degraded", + agents: [{ agentKey: "atomic-source", path: "atomic" }], + }); + expect(listWorkflows).not.toHaveBeenCalled(); + expect(inventoryStatus).not.toHaveBeenCalled(); + }); + + it("retires an active legacy extraction when syntax evidence arrives", async () => { + let release!: () => void; + const gate = new Promise((resolve) => { + release = resolve; + }); + let workflows: WorkflowInfo[] = [workflow("Legacy", "agent", "old-marker")]; + const inspectManifestName = vi.fn(async () => { + await gate; + return { status: "found" as const, name: "StaleExtracted" }; + }); + const changed = vi.fn(); + const inventory = new HarnessRegistryInventoryProvider({ + listWorkflows: () => workflows, + inventoryStatus: () => "complete", + fingerprintSource: async () => "fingerprint", + revalidateMarker: async () => true, + inspectManifestName, + onIdentityChange: changed, + }); + const legacy = await inventory.listAgents(SCOPE); + legacy.startEnrichment?.(); + await vi.waitFor(() => expect(inspectManifestName).toHaveBeenCalledOnce()); + + workflows = [ + workflow("Source", "agent", "old-marker", { + sourceDefinitionName: "CurrentSyntax", + }), + ]; + const syntax = await inventory.listAgents(SCOPE); + expect(syntax.inventory.agents[0]).toMatchObject({ + agentKey: "CurrentSyntax", + identityStatus: "canonical", + }); + release(); + await new Promise((resolve) => setTimeout(resolve, 0)); + + expect(changed).not.toHaveBeenCalled(); + await expect(inventory.listAgents(SCOPE)).resolves.toMatchObject({ + inventory: { + agents: [{ agentKey: "CurrentSyntax", identityStatus: "canonical" }], + }, + }); + }); + it("derives inventory roots using POSIX, drive, and UNC workspace flavor", () => { expect(inventorySourceRoot("/workspace", "nested/agent")).toBe( "/workspace/nested/agent", @@ -402,16 +611,13 @@ describe("HarnessRegistryInventoryProvider", () => { it("settles an unnameable identity without hiding its warning", async () => { const changed = vi.fn(); - const inventory = provider( - [workflow("Dashboard", "dashboard", null)], - { - inspectManifestName: async () => ({ - status: "failed", - retryable: false, - }), - onIdentityChange: changed, - }, - ); + const inventory = provider([workflow("Dashboard", "dashboard", null)], { + inspectManifestName: async () => ({ + status: "failed", + retryable: false, + }), + onIdentityChange: changed, + }); const result = await enrich( inventory, @@ -532,6 +738,147 @@ describe("HarnessRegistryInventoryProvider", () => { expect(maximum).toBe(4); }); + it("retires queued and active marker inspections synchronously on a raw scope edit", async () => { + let release!: () => void; + const gate = new Promise((resolve) => { + release = resolve; + }); + const inspectManifestName = vi.fn(async (sourceRoot: string) => { + await gate; + return { status: "found" as const, name: path.basename(sourceRoot) }; + }); + const changed = vi.fn(); + const inventory = provider( + Array.from({ length: 5 }, (_, index) => + workflow(`Agent ${index}`, `agent-${index}`, null), + ), + { inspectManifestName, onIdentityChange: changed }, + ); + + (await inventory.listAgents(SCOPE)).startEnrichment?.(); + await vi.waitFor(() => + expect(inspectManifestName).toHaveBeenCalledTimes(4), + ); + inventory.invalidateScope(WORKSPACE); + release(); + await new Promise((resolve) => setTimeout(resolve, 20)); + + expect(inspectManifestName).toHaveBeenCalledTimes(4); + expect(changed).not.toHaveBeenCalled(); + }); + + it("queues fresh proof behind an invalidated active inspection and converges", async () => { + let releaseFirst!: () => void; + const firstGate = new Promise((resolve) => { + releaseFirst = resolve; + }); + const inspectManifestName = vi.fn(async () => { + if (inspectManifestName.mock.calls.length === 1) await firstGate; + return { status: "found" as const, name: "FreshIdentity" }; + }); + const changed = vi.fn(); + const inventory = provider([workflow("Agent", "agent", null)], { + inspectManifestName, + onIdentityChange: changed, + }); + + (await inventory.listAgents(SCOPE)).startEnrichment?.(); + await vi.waitFor(() => expect(inspectManifestName).toHaveBeenCalledOnce()); + + // A raw edit in another active scope conservatively invalidates the shared + // provider epoch. A later ordinary read of this scope must enqueue fresh + // work instead of mistaking the stale active task for the same request. + inventory.invalidateScope("/private/workspaces/unrelated"); + (await inventory.listAgents(SCOPE)).startEnrichment?.(); + releaseFirst(); + + await vi.waitFor(() => + expect(inspectManifestName).toHaveBeenCalledTimes(2), + ); + await vi.waitFor(() => expect(changed).toHaveBeenCalledTimes(1)); + await expect(inventory.listAgents(SCOPE)).resolves.toMatchObject({ + inventory: { + agents: [{ agentKey: "FreshIdentity", identityStatus: "canonical" }], + }, + }); + }); + + it("revalidates marker proof after fingerprinting immediately before inspection", async () => { + let releaseFingerprint!: () => void; + const fingerprintGate = new Promise((resolve) => { + releaseFingerprint = resolve; + }); + let markerPresent = true; + const fingerprintEntered = vi.fn(); + const inspectManifestName = vi.fn(async () => ({ + status: "found" as const, + name: "must-not-run", + })); + const changed = vi.fn(); + const inventory = provider([workflow("Agent", "agent", null)], { + fingerprintSource: async () => { + fingerprintEntered(); + await fingerprintGate; + return "fingerprint"; + }, + revalidateMarker: async () => markerPresent, + inspectManifestName, + onIdentityChange: changed, + }); + + (await inventory.listAgents(SCOPE)).startEnrichment?.(); + await vi.waitFor(() => expect(fingerprintEntered).toHaveBeenCalledOnce()); + markerPresent = false; + releaseFingerprint(); + await new Promise((resolve) => setTimeout(resolve, 20)); + + expect(inspectManifestName).not.toHaveBeenCalled(); + expect(changed).not.toHaveBeenCalled(); + }); + + it("threads current marker proof to the inspector's actual child-launch boundary", async () => { + let releaseInnerFingerprint!: () => void; + const innerFingerprintGate = new Promise((resolve) => { + releaseInnerFingerprint = resolve; + }); + let markerPresent = true; + const innerFingerprintEntered = vi.fn(); + const actualExtractorLaunch = vi.fn(); + const inspectManifestName = vi.fn( + async ( + _sourceRoot: string, + options?: ManifestNameInspectionOptions, + ): Promise => { + innerFingerprintEntered(); + await innerFingerprintGate; + if (!(await options?.authorizeBeforeLaunch?.())) { + return { status: "failed", retryable: true }; + } + actualExtractorLaunch(); + return { status: "found", name: "must-not-run" }; + }, + ); + const changed = vi.fn(); + const inventory = provider([workflow("Agent", "agent", null)], { + revalidateMarker: async () => markerPresent, + inspectManifestName, + onIdentityChange: changed, + }); + + (await inventory.listAgents(SCOPE)).startEnrichment?.(); + await vi.waitFor(() => + expect(innerFingerprintEntered).toHaveBeenCalledOnce(), + ); + markerPresent = false; + releaseInnerFingerprint(); + await vi.waitFor(() => expect(changed).toHaveBeenCalledOnce()); + + expect(actualExtractorLaunch).not.toHaveBeenCalled(); + expect( + (await inventory.listAgents(SCOPE)).inventory.agents[0], + ).toMatchObject({ identityIssue: "identity-unavailable" }); + }); + it("surfaces settled identities within a bounded window while slower work continues", async () => { let releaseSlow!: () => void; const slow = new Promise((resolve) => { @@ -785,6 +1132,7 @@ describe("HarnessRegistryInventoryProvider", () => { const inventory = provider([workflow("Agent", "agent", "marker")], { inspectManifestName, fingerprintSource: async () => fingerprint, + revalidateMarker: async () => true, onIdentityChange: changed, }); const before = await enrich( @@ -879,6 +1227,7 @@ describe("HarnessRegistryInventoryProvider", () => { listWorkflows: () => workflows, inspectManifestName, fingerprintSource: async () => fingerprint, + revalidateMarker: async () => true, onIdentityChange: changed, }); await enrich(inventory, await inventory.listAgents(SCOPE), changed); diff --git a/packages/harness/src/core/system-graph-inventory.ts b/packages/harness/src/core/system-graph-inventory.ts index 6ef122352..b166c03c3 100644 --- a/packages/harness/src/core/system-graph-inventory.ts +++ b/packages/harness/src/core/system-graph-inventory.ts @@ -1,5 +1,4 @@ import { createHash } from "node:crypto"; -import { realpathSync } from "node:fs"; import * as path from "node:path"; import { @@ -14,9 +13,16 @@ import { type GraphWarning, type WorkspaceKey, } from "../shared/system-graph.js"; -import type { WorkflowInfo } from "../shared/types.js"; +import type { RegistryWorkflowInfo as WorkflowInfo } from "./workflow-registry.js"; import { fingerprintWorkflowSources } from "./canvas-cache.js"; -import type { ManifestNameInspection } from "./definition-name.js"; +import { canonicalGraphPath } from "./canonical-graph-path.js"; +import type { + ManifestNameInspection, + ManifestNameInspectionOptions, +} from "./definition-name.js"; +import { inspectAgentProjectMarker } from "./agent-project-discovery.js"; + +export { canonicalGraphPath } from "./canonical-graph-path.js"; export interface WorkspaceScope { workspaceKey: WorkspaceKey; @@ -66,6 +72,13 @@ export interface AgentInventoryResult { * the public contract continues to answer only what exists and where. */ identitySettled: boolean; + /** + * Private discovery coverage: false when the accepted workspace scan could + * not prove that every eligible agent was considered. This is independent + * of identity settlement because a settled provisional identity is still a + * cacheable result, while an incomplete workspace walk is not. + */ + discoveryComplete: boolean; /** Starts source identity work only after the provisional graph is committed. */ startEnrichment?: () => void; } @@ -79,6 +92,7 @@ export interface AgentInventoryProvider { type ManifestNameInspector = ( sourceRoot: string, + options?: ManifestNameInspectionOptions, ) => Promise; export interface HarnessRegistryInventoryProviderOptions { @@ -86,6 +100,36 @@ export interface HarnessRegistryInventoryProviderOptions { | readonly WorkflowInfo[] | Promise; inspectManifestName?: ManifestNameInspector; + /** Hardened launch-time marker proof; injectable for filesystem-free tests. */ + revalidateMarker?: (sourceRoot: string) => Promise; + /** Last discovery completeness for this exact selected scope. */ + inventoryStatus?: ( + scope: WorkspaceScope, + ) => "complete" | "degraded" | Promise<"complete" | "degraded">; + /** Atomic registry/cache projection; preferred over separate legacy reads. */ + inventorySnapshot?: (scope: WorkspaceScope) => + | { + workflows: readonly WorkflowInfo[]; + status: "complete" | "degraded"; + generation: number; + canonicalScopeRoot?: string; + canonicalWorkflowRoots?: readonly { + workflowPath: string; + canonicalRoot: string; + identityEvidence: "marker" | "source" | "not-agent" | "unknown"; + }[]; + } + | Promise<{ + workflows: readonly WorkflowInfo[]; + status: "complete" | "degraded"; + generation: number; + canonicalScopeRoot?: string; + canonicalWorkflowRoots?: readonly { + workflowPath: string; + canonicalRoot: string; + identityEvidence: "marker" | "source" | "not-agent" | "unknown"; + }[]; + }>; /** * Called with coalesced identity changes. Settled roots are surfaced within * a short bounded window, while a fully drained queue flushes immediately. @@ -127,35 +171,6 @@ function pathApi(input: string): typeof path.posix { * Resolve with the input path's own flavor so mixed Windows separators remain * comparable even when the test process (or a future remote host) is POSIX. */ -export function canonicalGraphPath(input: string): string { - const windows = isWindowsAbsolute(input); - const api = pathApi(input); - const normalizedInput = windows ? input.replace(/\//g, "\\") : input; - const resolved = api.resolve(normalizedInput); - const matchesHost = windows === (process.platform === "win32"); - if (!matchesHost) return resolved; - try { - return realpathSync.native(resolved); - } catch { - // Watchers can report a path after an atomic rename or deletion, so the - // leaf itself may no longer exist. Resolve the nearest existing ancestor. - const missingSegments: string[] = []; - let ancestor = resolved; - let parent = api.dirname(ancestor); - while (parent !== ancestor) { - missingSegments.unshift(api.basename(ancestor)); - ancestor = parent; - try { - return api.join(realpathSync.native(ancestor), ...missingSegments); - } catch { - // Keep walking toward an existing ancestor. - } - parent = api.dirname(ancestor); - } - return resolved; - } -} - /** * Resolve a public package-relative inventory path against its workspace. * The inventory path is always POSIX, while the workspace path keeps the @@ -276,7 +291,12 @@ function compareText(left: string, right: string): number { return left === right ? 0 : left < right ? -1 : 1; } -function packageRelativePath(scopeRoot: string, workflowPath: string): string { +function packageRelativePath( + scopeRoot: string, + workflowPath: string, + knownCanonicalScope?: string, + knownCanonicalSource?: string, +): string { const api = pathApi(scopeRoot); const relative = api.relative(scopeRoot, workflowPath); if ( @@ -291,8 +311,9 @@ function packageRelativePath(scopeRoot: string, workflowPath: string): string { // A symlinked registry path can have a different lexical spelling. Its // canonical source was already proven inside the canonical scope. - const canonicalScope = canonicalGraphPath(scopeRoot); - const canonicalSource = canonicalGraphPath(workflowPath); + const canonicalScope = knownCanonicalScope ?? canonicalGraphPath(scopeRoot); + const canonicalSource = + knownCanonicalSource ?? canonicalGraphPath(workflowPath); const canonicalApi = pathApi(canonicalScope); const canonicalRelative = canonicalApi.relative( canonicalScope, @@ -319,10 +340,13 @@ function stableJson(value: unknown): string { function buildWorkingTreeInventory( workspaceKey: WorkspaceKey, agents: PackageInventoryAgent[], + discoveryStatus: "complete" | "degraded", ): PackageInventory { - const status = agents.some((agent) => agent.identityStatus === "provisional") - ? "degraded" - : "complete"; + const status = + discoveryStatus === "degraded" || + agents.some((agent) => agent.identityStatus === "provisional") + ? "degraded" + : "complete"; const normalized = packageInventorySchema.parse({ protocol: PACKAGE_INVENTORY_PROTOCOL, version: { @@ -373,6 +397,8 @@ interface IdentityTask { sourceRoot: string; generation: number; epoch: number; + freshnessEpoch: number; + authorization: "marker" | "linked"; } function preparedOrder(left: PreparedAgent, right: PreparedAgent): number { @@ -434,42 +460,108 @@ export class HarnessRegistryInventoryProvider implements AgentInventoryProvider private identityChangeTimer: ReturnType | null = null; private epoch = 0; private nextGeneration = 1; + private latestInventoryGeneration = -1; + private freshnessEpoch = 0; constructor( private readonly options: HarnessRegistryInventoryProviderOptions, ) {} async listAgents(scope: WorkspaceScope): Promise { - const workflows = await this.options.listWorkflows(); - this.retainSources( - new Set(workflows.map((workflow) => canonicalGraphPath(workflow.path))), + let snapshot = this.options.inventorySnapshot + ? await this.options.inventorySnapshot(scope) + : null; + for ( + let retry = 0; + snapshot && + snapshot.generation < this.latestInventoryGeneration && + retry < 3; + retry += 1 + ) { + snapshot = await this.options.inventorySnapshot!(scope); + } + if (snapshot && snapshot.generation < this.latestInventoryGeneration) { + throw new Error( + "Inventory snapshot was superseded by a newer generation", + ); + } + if (snapshot) { + this.latestInventoryGeneration = Math.max( + this.latestInventoryGeneration, + snapshot.generation, + ); + } + const [workflows, discoveryStatus] = snapshot + ? [snapshot.workflows, snapshot.status] + : await Promise.all([ + this.options.listWorkflows(), + this.options.inventoryStatus?.(scope) ?? "complete", + ]); + const canonicalEntries = new Map( + snapshot?.canonicalWorkflowRoots?.map((entry) => [ + entry.workflowPath, + entry, + ]) ?? [], ); - const canonicalScopeRoot = canonicalGraphPath(scope.root); + const canonicalScopeRoot = + snapshot?.canonicalScopeRoot ?? canonicalGraphPath(scope.root); const bySourceRoot = new Map< string, - { workflow: WorkflowInfo; sourceRoot: string } + { + workflow: WorkflowInfo; + sourceRoot: string; + identityEvidence: "marker" | "source" | "not-agent" | "unknown"; + } >(); - const contained = workflows - .map((workflow) => ({ + const projected = workflows.map((workflow) => { + const canonicalEntry = canonicalEntries.get(workflow.path); + return { workflow, - sourceRoot: canonicalGraphPath(workflow.path), - })) + sourceRoot: + canonicalEntry?.canonicalRoot ?? canonicalGraphPath(workflow.path), + identityEvidence: + canonicalEntry?.identityEvidence ?? + (Object.prototype.hasOwnProperty.call( + workflow, + "sourceDefinitionName", + ) + ? "source" + : workflow.markerPresent === true + ? "marker" + : "unknown"), + } as const; + }); + this.retainSources(new Set(projected.map(({ sourceRoot }) => sourceRoot))); + const contained = projected .filter(({ sourceRoot }) => isWithinGraphPath(canonicalScopeRoot, sourceRoot), ) .sort((left, right) => workflowRegistryOrder(scope.root, left, right)); - for (const { workflow, sourceRoot } of contained) { + for (const { workflow, sourceRoot, identityEvidence } of contained) { // Registry persistence is expected to be unique by path. Keep the first // deterministic row if a corrupt/legacy file contains an exact duplicate. if (!bySourceRoot.has(sourceRoot)) { - bySourceRoot.set(sourceRoot, { workflow, sourceRoot }); + bySourceRoot.set(sourceRoot, { + workflow, + sourceRoot, + identityEvidence, + }); } } - const inspectionRoots: string[] = []; + const inspectionRoots: Array<{ + sourceRoot: string; + authorization: "marker" | "linked"; + }> = []; + let consumedUnknownPersistedIdentity = false; const prepared = [...bySourceRoot.values()] - .map(({ workflow, sourceRoot }): PreparedAgent => { - const inventoryPath = packageRelativePath(scope.root, workflow.path); + .map(({ workflow, sourceRoot, identityEvidence }): PreparedAgent => { + const inventoryPath = packageRelativePath( + scope.root, + workflow.path, + canonicalScopeRoot, + sourceRoot, + ); const fallbackKey = `local:${ inventoryPath === "." ? "root" : inventoryPath }` as AgentKey; @@ -479,10 +571,35 @@ export class HarnessRegistryInventoryProvider implements AgentInventoryProvider let identityIssue: InventoryIdentityIssue | null = null; let identitySettled = true; let warnOnIdentityFailure = false; - if (!this.options.inspectManifestName) { + const hasPersistedSourceIdentity = + Object.prototype.hasOwnProperty.call( + workflow, + "sourceDefinitionName", + ) && + (identityEvidence === "source" || identityEvidence === "unknown"); + if (hasPersistedSourceIdentity) { + this.retireInspectionSource(sourceRoot); + consumedUnknownPersistedIdentity ||= identityEvidence === "unknown"; + const sourceDefinitionName = workflow.sourceDefinitionName ?? null; + canonicalName = canonicalIdentity(sourceDefinitionName); + if (!canonicalName) { + identityIssue = sourceDefinitionName + ? "identity-invalid" + : "identity-unavailable"; + } + } else if ( + identityEvidence !== "marker" && + workflow.definitionId === null + ) { + this.retireInspectionSource(sourceRoot); + identityIssue = "identity-unavailable"; + } else if (!this.options.inspectManifestName) { identityIssue = "identity-unavailable"; } else { - inspectionRoots.push(sourceRoot); + inspectionRoots.push({ + sourceRoot, + authorization: identityEvidence === "marker" ? "marker" : "linked", + }); this.ensureGeneration(sourceRoot); if (!cached) { identityIssue = "identity-pending"; @@ -657,6 +774,7 @@ export class HarnessRegistryInventoryProvider implements AgentInventoryProvider const inventory = buildWorkingTreeInventory( scope.workspaceKey, publicAgents, + consumedUnknownPersistedIdentity ? "degraded" : discoveryStatus, ); const contextByAgent = new Map( context.map((item) => [item.agentKey, item]), @@ -667,19 +785,27 @@ export class HarnessRegistryInventoryProvider implements AgentInventoryProvider return item; }); warnings.sort(warningOrder); - const roots = [...new Set(inspectionRoots)].sort(compareText); + const roots = [ + ...new Map( + inspectionRoots.map((entry) => [entry.sourceRoot, entry]), + ).values(), + ].sort((left, right) => compareText(left.sourceRoot, right.sourceRoot)); const enrichmentEpoch = this.epoch; - const tasks = roots.map((sourceRoot) => ({ + const tasks = roots.map(({ sourceRoot, authorization }) => ({ sourceRoot, + authorization, generation: this.generations.get(sourceRoot)!, epoch: enrichmentEpoch, + freshnessEpoch: this.freshnessEpoch, })); return { inventory, context: normalizedContext, warnings, identitySettled: prepared.every((agent) => agent.identitySettled), - ...(roots.length > 0 + discoveryComplete: + discoveryStatus === "complete" && !consumedUnknownPersistedIdentity, + ...(tasks.length > 0 ? { startEnrichment: () => this.enqueueInspections(tasks, enrichmentEpoch), @@ -700,6 +826,25 @@ export class HarnessRegistryInventoryProvider implements AgentInventoryProvider } } + /** O(1) fail-closed invalidation for a raw change beneath a graph scope. */ + invalidateScope(_scopeRoot: string): void { + // Conservatively invalidate every queued/active V0 task. This is a true + // O(1) raw-event operation; queue cleanup happens when a worker slot next + // drains, and each task's captured epoch prevents it from launching or + // publishing in the meantime. + this.freshnessEpoch += 1; + } + + private retireInspectionSource(sourceRoot: string): void { + this.identityCache.delete(sourceRoot); + this.generations.delete(sourceRoot); + this.dropQueuedTasks(sourceRoot); + this.pendingIdentityChanges.delete(sourceRoot); + if (this.pendingIdentityChanges.size === 0) { + this.clearIdentityChangeTimer(); + } + } + /** Explicit Retry may retry failures even when no source fingerprint changed. */ retryFailedInspections(scope: WorkspaceScope): void { const root = canonicalGraphPath(scope.root); @@ -721,6 +866,7 @@ export class HarnessRegistryInventoryProvider implements AgentInventoryProvider this.generations.clear(); this.queuedTasks.length = 0; this.pendingIdentityChanges.clear(); + this.freshnessEpoch += 1; this.clearIdentityChangeTimer(); } @@ -757,7 +903,8 @@ export class HarnessRegistryInventoryProvider implements AgentInventoryProvider const active = this.activeTasks.get(task.sourceRoot); if ( active?.generation === task.generation && - active.epoch === task.epoch + active.epoch === task.epoch && + active.freshnessEpoch === task.freshnessEpoch ) { continue; } @@ -798,13 +945,37 @@ export class HarnessRegistryInventoryProvider implements AgentInventoryProvider let fingerprint: string | null = null; let inspection: ManifestNameInspection; try { + if (task.authorization === "marker") { + const validMarker = this.options.revalidateMarker + ? await this.options.revalidateMarker(task.sourceRoot) + : (await inspectAgentProjectMarker(task.sourceRoot)).status === + "valid"; + if (!validMarker || !this.isCurrentTask(task)) return; + } fingerprint = await ( this.options.fingerprintSource ?? fingerprintWorkflowSources )(task.sourceRoot); if (!this.isCurrentTask(task)) return; + if (task.authorization === "marker") { + const validMarker = this.options.revalidateMarker + ? await this.options.revalidateMarker(task.sourceRoot) + : (await inspectAgentProjectMarker(task.sourceRoot)).status === + "valid"; + if (!validMarker || !this.isCurrentTask(task)) return; + } const hit = this.identityCache.get(task.sourceRoot); if (hit?.fingerprint === fingerprint) return; - inspection = await inspect(task.sourceRoot); + inspection = await inspect(task.sourceRoot, { + authorizeBeforeLaunch: async () => { + if (!this.isCurrentTask(task)) return false; + if (task.authorization === "linked") return true; + const validMarker = this.options.revalidateMarker + ? await this.options.revalidateMarker(task.sourceRoot) + : (await inspectAgentProjectMarker(task.sourceRoot)).status === + "valid"; + return validMarker && this.isCurrentTask(task); + }, + }); } catch { if (!this.isCurrentTask(task)) return; const hit = this.identityCache.get(task.sourceRoot); @@ -877,7 +1048,8 @@ export class HarnessRegistryInventoryProvider implements AgentInventoryProvider private isCurrentTask(task: IdentityTask): boolean { return ( task.epoch === this.epoch && - task.generation === this.generations.get(task.sourceRoot) + task.generation === this.generations.get(task.sourceRoot) && + task.freshnessEpoch === this.freshnessEpoch ); } diff --git a/packages/harness/src/core/system-graph-relationships.test.ts b/packages/harness/src/core/system-graph-relationships.test.ts index b9bdf1df5..4961d4461 100644 --- a/packages/harness/src/core/system-graph-relationships.test.ts +++ b/packages/harness/src/core/system-graph-relationships.test.ts @@ -5,16 +5,35 @@ import { afterEach, describe, expect, it, vi } from "vitest"; import type { AgentInventoryItem } from "./system-graph-inventory.js"; import { - CachedAgentRelationshipProvider, - SourceAgentRelationshipProvider, - type AgentRelationshipProvider, + CachedAgentInvocationProvider, + SourceAgentInvocationProvider, + type AgentInvocationProvider, + type AgentInvocationProviderResult, } from "./system-graph-relationships.js"; const temporaryRoots: string[] = []; +const EMPTY_RESULT: AgentInvocationProviderResult = { + invocations: [], + warnings: [], +}; + +function deferred(): { + promise: Promise; + resolve: (value: T) => void; + reject: (error: unknown) => void; +} { + let resolve!: (value: T) => void; + let reject!: (error: unknown) => void; + const promise = new Promise((onResolve, onReject) => { + resolve = onResolve; + reject = onReject; + }); + return { promise, resolve, reject }; +} async function callerWithSource(source: string): Promise { const sourceRoot = await fs.mkdtemp( - path.join(os.tmpdir(), "system-graph-relationships-test-"), + path.join(os.tmpdir(), "system-graph-invocations-test-"), ); temporaryRoots.push(sourceRoot); await fs.writeFile(path.join(sourceRoot, "index.ts"), source); @@ -40,7 +59,7 @@ afterEach(async () => { ); }); -describe("SourceAgentRelationshipProvider", () => { +describe("SourceAgentInvocationProvider", () => { it("aggregates evidence by target and mode while preserving distinct modes", async () => { const caller = await callerWithSource(` ctx.sapiom.agents.run({ definition: "growth" }); @@ -49,10 +68,11 @@ ctx.sapiom.agents.launch({ definition: "growth" }); ctx.sapiom.agents.launch({ definition: dynamicTarget }); `); - const result = - await new SourceAgentRelationshipProvider().listRelationships(caller); + const result = await new SourceAgentInvocationProvider().listInvocations( + caller, + ); - expect(result.relationships).toEqual([ + expect(result.invocations).toEqual([ { target: "growth", mode: "blocking", @@ -76,46 +96,155 @@ ctx.sapiom.agents.launch({ definition: dynamicTarget }); ]); }); - it("returns identical relationship semantics for unchanged caller input", async () => { + it("reports only the coordinator's direct invocations without inferring output data flow", async () => { + const caller = { + ...(await callerWithSource(` +import { agents } from "@sapiom/tools"; + +const research = await agents.run({ + definition: "research", +}); + +const summary = formatResearch(research.output); + +await agents.run({ + definition: "growth", + input: { summary }, +}); +`)), + agentKey: "coordinator", + definitionSlug: "coordinator", + label: "Coordinator", + resolutionAliases: ["coordinator"], + }; + + const result = await new SourceAgentInvocationProvider().listInvocations( + caller, + ); + const directInvocations = result.invocations.map(({ target }) => [ + caller.agentKey, + target, + ]); + + expect(directInvocations).toEqual([ + ["coordinator", "research"], + ["coordinator", "growth"], + ]); + expect(directInvocations).not.toContainEqual(["research", "growth"]); + }); + + it("returns identical invocation semantics for unchanged caller input", async () => { const caller = await callerWithSource( 'ctx.sapiom.agents.run({ definition: "growth" });\n', ); - const provider = new SourceAgentRelationshipProvider(); + const provider = new SourceAgentInvocationProvider(); - const first = await provider.listRelationships(caller); - const second = await provider.listRelationships(caller); + const first = await provider.listInvocations(caller); + const second = await provider.listInvocations(caller); expect(second).toEqual(first); }); + + it("does not follow a TypeScript symlink outside the workflow", async () => { + const caller = await callerWithSource("export const value = 1;\n"); + const external = await fs.mkdtemp( + path.join(os.tmpdir(), "system-graph-invocations-external-"), + ); + temporaryRoots.push(external); + await fs.writeFile( + path.join(external, "secret.ts"), + 'ctx.sapiom.agents.run({ definition: "outside" });\n', + ); + await fs.symlink( + path.join(external, "secret.ts"), + path.join(caller.sourceRoot, "evil.ts"), + ); + const onBytesRead = vi.fn(); + + const result = await new SourceAgentInvocationProvider({ + onBytesRead, + }).listInvocations(caller); + + expect(result.invocations).toEqual([]); + expect(result.complete).toBe(false); + expect(onBytesRead).toHaveBeenCalledTimes(1); + expect(onBytesRead).toHaveBeenCalledWith( + path.join(caller.sourceRoot, "index.ts"), + expect.any(Number), + ); + expect(onBytesRead).not.toHaveBeenCalledWith( + path.join(external, "secret.ts"), + expect.any(Number), + ); + }); + + it("rejects an ancestor swap before reading invocation source bytes", async () => { + const caller = await callerWithSource("export const value = 1;\n"); + const inside = path.join(caller.sourceRoot, "inside"); + await fs.mkdir(inside); + await fs.writeFile( + path.join(inside, "edge.ts"), + 'ctx.sapiom.agents.run({ definition: "inside" });\n', + ); + const external = await fs.mkdtemp( + path.join(os.tmpdir(), "system-graph-invocations-swap-"), + ); + temporaryRoots.push(external); + await fs.writeFile( + path.join(external, "edge.ts"), + 'ctx.sapiom.agents.run({ definition: "outside" });\n', + ); + const onBytesRead = vi.fn(); + let swapped = false; + + const result = await new SourceAgentInvocationProvider({ + beforeOpen: async (file) => { + if (!file.endsWith(`${path.sep}inside${path.sep}edge.ts`) || swapped) { + return; + } + swapped = true; + await fs.rename(inside, `${inside}-original`); + await fs.symlink(external, inside, "dir"); + }, + onBytesRead, + }).listInvocations(caller); + + expect(result.invocations).toEqual([]); + expect(result.complete).toBe(false); + expect(onBytesRead).not.toHaveBeenCalledWith( + path.join(external, "edge.ts"), + expect.any(Number), + ); + }); }); -describe("CachedAgentRelationshipProvider", () => { +describe("CachedAgentInvocationProvider", () => { it("coalesces and reuses unchanged caller extraction", async () => { const caller = await callerWithSource("export const value = 1;\n"); - const inner: AgentRelationshipProvider = { - listRelationships: vi.fn(async () => ({ - relationships: [], + const inner: AgentInvocationProvider = { + listInvocations: vi.fn(async () => ({ + invocations: [], warnings: [], })), }; const fingerprint = vi.fn(async () => "fingerprint-one"); - const provider = new CachedAgentRelationshipProvider(inner, fingerprint); + const provider = new CachedAgentInvocationProvider(inner, fingerprint); - const first = provider.listRelationships(caller); - await expect(provider.listRelationships(caller)).resolves.toEqual( + const first = provider.listInvocations(caller); + await expect(provider.listInvocations(caller)).resolves.toEqual( await first, ); - await provider.listRelationships(caller); + await provider.listInvocations(caller); - expect(inner.listRelationships).toHaveBeenCalledTimes(1); + expect(inner.listInvocations).toHaveBeenCalledTimes(1); expect(fingerprint).toHaveBeenCalledTimes(3); }); it("rescans only after the caller fingerprint changes", async () => { const caller = await callerWithSource("export const value = 1;\n"); - const inner: AgentRelationshipProvider = { - listRelationships: vi.fn(async () => ({ - relationships: [], + const inner: AgentInvocationProvider = { + listInvocations: vi.fn(async () => ({ + invocations: [], warnings: [], })), }; @@ -123,57 +252,339 @@ describe("CachedAgentRelationshipProvider", () => { .fn() .mockResolvedValueOnce("one") .mockResolvedValueOnce("two"); - const provider = new CachedAgentRelationshipProvider(inner, fingerprint); + const provider = new CachedAgentInvocationProvider(inner, fingerprint); - await provider.listRelationships(caller); - await provider.listRelationships(caller); + await provider.listInvocations(caller); + await provider.listInvocations(caller); - expect(inner.listRelationships).toHaveBeenCalledTimes(2); + expect(inner.listInvocations).toHaveBeenCalledTimes(2); }); it("does not retain a failed caller result", async () => { const caller = await callerWithSource("export const value = 1;\n"); - const inner: AgentRelationshipProvider = { - listRelationships: vi + const inner: AgentInvocationProvider = { + listInvocations: vi .fn() .mockRejectedValueOnce(new Error("not installed")) - .mockResolvedValueOnce({ relationships: [], warnings: [] }), + .mockResolvedValueOnce({ invocations: [], warnings: [] }), }; - const provider = new CachedAgentRelationshipProvider( + const provider = new CachedAgentInvocationProvider( inner, async () => "same", ); - await expect(provider.listRelationships(caller)).rejects.toThrow( + await expect(provider.listInvocations(caller)).rejects.toThrow( "not installed", ); - await expect(provider.listRelationships(caller)).resolves.toEqual({ - relationships: [], + await expect(provider.listInvocations(caller)).resolves.toEqual({ + invocations: [], warnings: [], }); - expect(inner.listRelationships).toHaveBeenCalledTimes(2); + expect(inner.listInvocations).toHaveBeenCalledTimes(2); }); it("evicts removed callers while preserving retained callers", async () => { const first = await callerWithSource("export const first = 1;\n"); const second = await callerWithSource("export const second = 1;\n"); - const inner: AgentRelationshipProvider = { - listRelationships: vi.fn(async () => ({ - relationships: [], + const inner: AgentInvocationProvider = { + listInvocations: vi.fn(async () => ({ + invocations: [], warnings: [], })), }; - const provider = new CachedAgentRelationshipProvider( + const provider = new CachedAgentInvocationProvider( inner, async () => "same", ); - await provider.listRelationships(first); - await provider.listRelationships(second); + await provider.listInvocations(first); + await provider.listInvocations(second); provider.retainCallers([second]); - await provider.listRelationships(first); - await provider.listRelationships(second); + await provider.listInvocations(first); + await provider.listInvocations(second); + + expect(inner.listInvocations).toHaveBeenCalledTimes(3); + }); + + it("runs background invocation extraction with a fixed concurrency cap", async () => { + const callers = await Promise.all( + Array.from({ length: 9 }, (_, index) => + callerWithSource(`export const value = ${index};\n`), + ), + ); + const releases: Array<() => void> = []; + let active = 0; + let maxActive = 0; + const onChange = vi.fn(); + const inner: AgentInvocationProvider = { + listInvocations: vi.fn(async () => { + active += 1; + maxActive = Math.max(maxActive, active); + await new Promise((resolve) => releases.push(resolve)); + active -= 1; + return EMPTY_RESULT; + }), + }; + const provider = new CachedAgentInvocationProvider( + inner, + async () => "unused", + { concurrency: 4, onChange }, + ); + + provider.startInvocations(callers); + await vi.waitFor(() => { + expect(inner.listInvocations).toHaveBeenCalledTimes(4); + }); + expect(maxActive).toBe(4); + + while (releases.length > 0 || active > 0) { + releases.splice(0).forEach((release) => release()); + await new Promise((resolve) => setTimeout(resolve, 0)); + } + await vi.waitFor(() => { + expect(inner.listInvocations).toHaveBeenCalledTimes(9); + expect(onChange).toHaveBeenCalledTimes(1); + }); + expect(maxActive).toBe(4); + expect( + callers.every( + (caller) => provider.peekInvocations(caller)?.status === "ready", + ), + ).toBe(true); + }); + + it("discards a superseded result and runs the fresh generation afterward", async () => { + const caller = await callerWithSource("export const value = 1;\n"); + const first = deferred(); + const inner: AgentInvocationProvider = { + listInvocations: vi + .fn() + .mockImplementationOnce(() => first.promise) + .mockResolvedValueOnce({ + invocations: [ + { + target: "fresh", + mode: "blocking", + evidence: [{ file: "index.ts", line: 1, column: 1 }], + }, + ], + warnings: [], + }), + }; + const onChange = vi.fn(); + const provider = new CachedAgentInvocationProvider( + inner, + async () => "unused", + { concurrency: 1, onChange }, + ); + + provider.startInvocations([caller]); + await vi.waitFor(() => + expect(inner.listInvocations).toHaveBeenCalledTimes(1), + ); + provider.invalidateSource(caller.sourceRoot); + provider.startInvocations([caller]); + first.resolve({ + invocations: [ + { + target: "stale", + mode: "blocking", + evidence: [{ file: "index.ts", line: 1, column: 1 }], + }, + ], + warnings: [], + }); + + await vi.waitFor(() => { + expect(inner.listInvocations).toHaveBeenCalledTimes(2); + expect( + provider.peekInvocations(caller)?.result.invocations[0]?.target, + ).toBe("fresh"); + }); + expect(onChange).toHaveBeenCalledTimes(1); + }); + + it("keeps failures stable until explicit Retry rearms them", async () => { + const caller = await callerWithSource("export const value = 1;\n"); + const inner: AgentInvocationProvider = { + listInvocations: vi + .fn() + .mockRejectedValueOnce(new Error("transient")) + .mockResolvedValueOnce(EMPTY_RESULT), + }; + const provider = new CachedAgentInvocationProvider(inner); + + provider.startInvocations([caller]); + await vi.waitFor(() => { + expect(provider.peekInvocations(caller)?.status).toBe("failed"); + }); + provider.startInvocations([caller]); + await new Promise((resolve) => setTimeout(resolve, 0)); + expect(inner.listInvocations).toHaveBeenCalledTimes(1); + + provider.retryFailed(caller.sourceRoot); + provider.startInvocations([caller]); + await vi.waitFor(() => { + expect(provider.peekInvocations(caller)?.status).toBe("ready"); + }); + expect(inner.listInvocations).toHaveBeenCalledTimes(2); + }); + + it("rearms an incomplete bounded scan only on explicit Retry", async () => { + const caller = await callerWithSource("export const value = 1;\n"); + const inner: AgentInvocationProvider = { + listInvocations: vi + .fn() + .mockResolvedValueOnce({ ...EMPTY_RESULT, complete: false }) + .mockResolvedValueOnce({ ...EMPTY_RESULT, complete: true }), + }; + const provider = new CachedAgentInvocationProvider(inner); + + provider.startInvocations([caller]); + await vi.waitFor(() => { + expect(provider.peekInvocations(caller)?.result.complete).toBe(false); + }); + provider.startInvocations([caller]); + expect(inner.listInvocations).toHaveBeenCalledTimes(1); + + provider.retryFailed(caller.sourceRoot); + provider.startInvocations([caller]); + await vi.waitFor(() => { + expect(provider.peekInvocations(caller)?.result.complete).toBe(true); + }); + expect(inner.listInvocations).toHaveBeenCalledTimes(2); + }); + + it("caps retained watcher observations fairly and degrades overflow", async () => { + const callers = await Promise.all([ + callerWithSource("export const first = 1;\n"), + callerWithSource("export const second = 2;\n"), + ]); + const inner: AgentInvocationProvider = { + listInvocations: vi.fn(async (caller) => ({ + ...EMPTY_RESULT, + complete: true, + observedPaths: Array.from({ length: 6_000 }, (_, index) => + path.join(caller.sourceRoot, `observed-${index}.ts`), + ), + })), + }; + const provider = new CachedAgentInvocationProvider(inner); + + provider.startInvocations(callers); + await vi.waitFor(() => { + expect( + callers.every( + (caller) => + provider.peekInvocations(caller)?.result.complete === false, + ), + ).toBe(true); + }); + const observations = provider.invocationObservations(); + + expect(observations).toHaveLength(2); + expect(observations.map((entry) => entry.paths.length)).toEqual([ + 5_000, 5_000, + ]); + }); + + it("notifies every retained root when global observation coverage changes", async () => { + const callers = await Promise.all([ + callerWithSource("export const first = 1;\n"), + callerWithSource("export const second = 2;\n"), + ]); + const inner: AgentInvocationProvider = { + listInvocations: vi.fn(async (caller) => ({ + ...EMPTY_RESULT, + complete: true, + observedPaths: Array.from({ length: 6_000 }, (_, index) => + path.join(caller.sourceRoot, `observed-${index}.ts`), + ), + })), + }; + const onChange = vi.fn(); + const provider = new CachedAgentInvocationProvider( + inner, + async () => "unused", + { onChange }, + ); + + provider.startInvocations([callers[0]!]); + await vi.waitFor(() => expect(onChange).toHaveBeenCalledTimes(1)); + expect(provider.peekInvocations(callers[0]!)?.result.complete).toBe(true); + onChange.mockClear(); + + provider.startInvocations(callers); + await vi.waitFor(() => expect(onChange).toHaveBeenCalled()); + expect(onChange.mock.calls.flat(2)).toEqual( + expect.arrayContaining(callers.map((caller) => caller.sourceRoot)), + ); + expect(provider.peekInvocations(callers[0]!)?.result.complete).toBe(false); + onChange.mockClear(); + + provider.retainCallers([callers[0]!]); + await vi.waitFor(() => expect(onChange).toHaveBeenCalledTimes(1)); + expect(onChange).toHaveBeenCalledWith([callers[0]!.sourceRoot]); + expect(provider.peekInvocations(callers[0]!)?.result.complete).toBe(true); + }); + + it("does not notify for a settled result invalidated before its batch flush", async () => { + const callers = await Promise.all([ + callerWithSource("export const fast = 1;\n"), + callerWithSource("export const slow = 2;\n"), + ]); + const slow = deferred(); + const inner: AgentInvocationProvider = { + listInvocations: vi + .fn() + .mockResolvedValueOnce(EMPTY_RESULT) + .mockImplementationOnce(() => slow.promise), + }; + const onChange = vi.fn(); + const provider = new CachedAgentInvocationProvider( + inner, + async () => "unused", + { concurrency: 2, onChange, changeBatchMs: 50 }, + ); + + provider.startInvocations(callers); + await vi.waitFor(() => { + expect(provider.peekInvocations(callers[0]!)?.status).toBe("ready"); + }); + provider.invalidateScope(os.tmpdir()); + slow.resolve(EMPTY_RESULT); + await new Promise((resolve) => setTimeout(resolve, 70)); + + expect(onChange).not.toHaveBeenCalled(); + }); + + it("publishes a settled workspace while an unrelated task remains held", async () => { + const callers = await Promise.all([ + callerWithSource("export const quick = 1;\n"), + callerWithSource("export const held = 2;\n"), + ]); + const held = deferred(); + const inner: AgentInvocationProvider = { + listInvocations: vi + .fn() + .mockResolvedValueOnce(EMPTY_RESULT) + .mockImplementationOnce(() => held.promise), + }; + const onChange = vi.fn(); + const provider = new CachedAgentInvocationProvider( + inner, + async () => "unused", + { concurrency: 2, onChange }, + ); + + provider.startInvocations(callers); + await vi.waitFor(() => expect(onChange).toHaveBeenCalledTimes(1)); + + expect(onChange).toHaveBeenLastCalledWith([callers[0]!.sourceRoot]); + expect(provider.peekInvocations(callers[0]!)?.status).toBe("ready"); + expect(provider.peekInvocations(callers[1]!)).toBeUndefined(); - expect(inner.listRelationships).toHaveBeenCalledTimes(3); + held.resolve(EMPTY_RESULT); + await vi.waitFor(() => expect(onChange).toHaveBeenCalledTimes(2)); }); }); diff --git a/packages/harness/src/core/system-graph-relationships.ts b/packages/harness/src/core/system-graph-relationships.ts index ef2dba366..89d41c92e 100644 --- a/packages/harness/src/core/system-graph-relationships.ts +++ b/packages/harness/src/core/system-graph-relationships.ts @@ -5,11 +5,15 @@ import { type AgentInvocationDetectionWarning, type AgentInvocationMode, type SourceEvidence, + type WorkflowSourceReadHooks, } from "./canvas-interconnections.js"; import { fingerprintWorkflowSources } from "./canvas-cache.js"; +import { canonicalGraphPath } from "./canonical-graph-path.js"; import type { AgentInventoryItem } from "./system-graph-inventory.js"; -export interface AgentRelationshipCandidate { +export const INVOCATION_OBSERVATION_MAX_PATHS = 10_000; + +export interface AgentInvocationCandidate { /** Inventory key or definition slug to resolve after extraction. */ target: string; mode: AgentInvocationMode; @@ -17,48 +21,103 @@ export interface AgentRelationshipCandidate { evidence: SourceEvidence[]; } -export type AgentRelationshipWarning = AgentInvocationDetectionWarning; +export type AgentInvocationWarning = AgentInvocationDetectionWarning; + +export interface AgentInvocationProviderResult { + invocations: AgentInvocationCandidate[]; + warnings: AgentInvocationWarning[]; + /** Confined files considered by this bounded extraction generation. */ + observedPaths?: readonly string[]; + /** False when an opaque path or work cap prevented a complete scan. */ + complete?: boolean; +} + +export interface AgentInvocationSnapshot { + status: "ready" | "failed"; + result: AgentInvocationProviderResult; +} -export interface AgentRelationshipProviderResult { - relationships: AgentRelationshipCandidate[]; - warnings: AgentRelationshipWarning[]; +export interface AgentInvocationObservation { + candidateRoot: string; + workspaceRoot: string; + paths: readonly string[]; } -/** Replaceable per-caller boundary consumed by the workspace graph projector. */ -export interface AgentRelationshipProvider { - listRelationships( +/** + * Replaceable per-caller boundary for literal direct invocations consumed by + * the workspace graph projector. The caller is always the source endpoint. + */ +export interface AgentInvocationProvider { + listInvocations( caller: AgentInventoryItem, - ): Promise; + ): Promise; /** Optional lifecycle hook for providers that retain per-caller state. */ retainCallers?(callers: readonly AgentInventoryItem[]): void; + /** Cache-only projection used by the first graph phase. */ + peekInvocations?( + caller: AgentInventoryItem, + ): AgentInvocationSnapshot | undefined; + /** Starts bounded work only after the inventory-only graph is committed. */ + startInvocations?(callers: readonly AgentInventoryItem[]): void; + /** Accepted/current invocation metadata consumed by polling watchers. */ + invocationObservations?(): readonly AgentInvocationObservation[]; } -interface CachedRelationshipEntry { +interface CachedInvocationEntry { fingerprint: string; - result: Promise; + result: Promise; +} + +interface InvocationTask { + sourceRoot: string; + caller: AgentInventoryItem; + generation: number; + scopeEpoch: number; +} + +interface BackgroundInvocationEntry { + generation: number; + scopeEpoch: number; + snapshot?: AgentInvocationSnapshot; +} + +export interface CachedAgentInvocationProviderOptions { + concurrency?: number; + onChange?: (sourceRoots: readonly string[]) => void | Promise; + /** Small testable coalescing window; settled scopes never await global idle. */ + changeBatchMs?: number; } /** - * Successful per-caller relationship extraction behind the same cheap source + * Successful per-caller invocation extraction behind the same cheap source * fingerprint used by Canvas. Projection can therefore rebuild against a new * inventory without re-walking unchanged caller trees. */ -export class CachedAgentRelationshipProvider - implements AgentRelationshipProvider -{ - private readonly entries = new Map(); +export class CachedAgentInvocationProvider implements AgentInvocationProvider { + private readonly entries = new Map(); + private readonly background = new Map(); + private readonly queued: InvocationTask[] = []; + private readonly active = new Map(); + private readonly pendingChanges = new Set(); + private readonly invalidatedScopes = new Map(); + private nextGeneration = 1; + private nextScopeEpoch = 1; + private activeCount = 0; + private observationsTruncated = false; + private changeFlushTimer: ReturnType | null = null; constructor( - private readonly inner: AgentRelationshipProvider = new SourceAgentRelationshipProvider(), + private readonly inner: AgentInvocationProvider = new SourceAgentInvocationProvider(), private readonly fingerprint: ( sourceRoot: string, ) => Promise = fingerprintWorkflowSources, + private readonly options: CachedAgentInvocationProviderOptions = {}, ) {} - async listRelationships( + async listInvocations( caller: AgentInventoryItem, - ): Promise { - const key = path.resolve(caller.sourceRoot); + ): Promise { + const key = canonicalGraphPath(caller.sourceRoot); let fingerprint: string; try { fingerprint = await this.fingerprint(key); @@ -70,9 +129,9 @@ export class CachedAgentRelationshipProvider const hit = this.entries.get(key); if (hit?.fingerprint === fingerprint) return hit.result; - let result: Promise; + let result: Promise; try { - result = Promise.resolve(this.inner.listRelationships(caller)); + result = Promise.resolve(this.inner.listInvocations(caller)); } catch (error) { result = Promise.reject(error); } @@ -85,20 +144,311 @@ export class CachedAgentRelationshipProvider } invalidateSource(sourceRoot: string): void { - this.entries.delete(path.resolve(sourceRoot)); + const key = canonicalGraphPath(sourceRoot); + this.entries.delete(key); + this.background.set(key, { + generation: this.nextGeneration++, + scopeEpoch: this.scopeEpochForSource(key), + }); + this.dropQueued(key); + this.pendingChanges.delete(key); + } + + /** O(1) conservative invalidation for an ambiguous workspace event. */ + invalidateScope(scopeRoot: string): void { + this.invalidatedScopes.set( + canonicalGraphPath(scopeRoot), + this.nextScopeEpoch++, + ); + } + + /** Explicit graph Retry re-arms terminal failures without read-loop churn. */ + retryFailed(scopeRoot: string): void { + const root = canonicalGraphPath(scopeRoot); + for (const [sourceRoot, entry] of this.background) { + const relative = path.relative(root, sourceRoot); + if ( + (entry.snapshot?.status !== "failed" && + !( + entry.snapshot?.status === "ready" && + entry.snapshot.result.complete === false + )) || + relative === ".." || + relative.startsWith(`..${path.sep}`) || + path.isAbsolute(relative) + ) { + continue; + } + this.background.set(sourceRoot, { + generation: this.nextGeneration++, + scopeEpoch: this.scopeEpochForSource(sourceRoot), + }); + this.dropQueued(sourceRoot); + } } retainCallers(callers: readonly AgentInventoryItem[]): void { const retained = new Set( - callers.map((caller) => path.resolve(caller.sourceRoot)), + callers.map((caller) => canonicalGraphPath(caller.sourceRoot)), ); for (const sourceRoot of this.entries.keys()) { if (!retained.has(sourceRoot)) this.entries.delete(sourceRoot); } + for (const sourceRoot of this.background.keys()) { + if (retained.has(sourceRoot)) continue; + this.background.delete(sourceRoot); + this.dropQueued(sourceRoot); + this.pendingChanges.delete(sourceRoot); + } + for (const scopeRoot of this.invalidatedScopes.keys()) { + if ( + [...retained].some((sourceRoot) => + this.scopeContainsSource(scopeRoot, sourceRoot), + ) + ) { + continue; + } + this.invalidatedScopes.delete(scopeRoot); + } + this.refreshObservationCoverage(); } clear(): void { this.entries.clear(); + this.background.clear(); + this.queued.length = 0; + this.pendingChanges.clear(); + if (this.changeFlushTimer) clearTimeout(this.changeFlushTimer); + this.changeFlushTimer = null; + this.invalidatedScopes.clear(); + this.nextGeneration += 1; + this.nextScopeEpoch += 1; + this.observationsTruncated = false; + } + + peekInvocations( + caller: AgentInventoryItem, + ): AgentInvocationSnapshot | undefined { + const sourceRoot = canonicalGraphPath(caller.sourceRoot); + const entry = this.background.get(sourceRoot); + if (entry?.scopeEpoch !== this.scopeEpochForSource(sourceRoot)) { + return undefined; + } + if ( + this.observationsTruncated && + entry.snapshot?.status === "ready" && + entry.snapshot.result.complete !== false + ) { + return { + status: "ready", + result: { ...entry.snapshot.result, complete: false }, + }; + } + return entry.snapshot; + } + + startInvocations(callers: readonly AgentInventoryItem[]): void { + for (const caller of callers) { + const sourceRoot = canonicalGraphPath(caller.sourceRoot); + let entry = this.background.get(sourceRoot); + const scopeEpoch = this.scopeEpochForSource(sourceRoot); + if (!entry || entry.scopeEpoch !== scopeEpoch) { + entry = { generation: this.nextGeneration++, scopeEpoch }; + this.background.set(sourceRoot, entry); + this.dropQueued(sourceRoot); + } + if (entry.snapshot) continue; + if ( + this.active.get(sourceRoot)?.generation === entry.generation || + this.queued.some( + (task) => + task.sourceRoot === sourceRoot && + task.generation === entry!.generation, + ) + ) { + continue; + } + this.dropQueued(sourceRoot); + this.queued.push({ + sourceRoot, + caller, + generation: entry.generation, + scopeEpoch: entry.scopeEpoch, + }); + } + this.drain(); + } + + invocationObservations(): readonly AgentInvocationObservation[] { + const entries = [...this.background.entries()] + .filter( + ([sourceRoot, entry]) => + entry.scopeEpoch === this.scopeEpochForSource(sourceRoot) && + (entry.snapshot?.result.observedPaths?.length ?? 0) > 0, + ) + .sort(([left], [right]) => left.localeCompare(right)); + const selected = new Map(); + let remaining = INVOCATION_OBSERVATION_MAX_PATHS; + let round = 0; + while (remaining > 0) { + let added = false; + for (const [sourceRoot, entry] of entries) { + if (remaining === 0) break; + const observed = entry.snapshot?.result.observedPaths?.[round]; + if (!observed) continue; + const paths = selected.get(sourceRoot) ?? []; + paths.push(observed); + selected.set(sourceRoot, paths); + remaining -= 1; + added = true; + } + if (!added) break; + round += 1; + } + return [...selected.entries()].map(([sourceRoot, paths]) => ({ + candidateRoot: sourceRoot, + workspaceRoot: sourceRoot, + paths, + })); + } + + private current(task: InvocationTask): boolean { + const entry = this.background.get(task.sourceRoot); + return ( + entry?.generation === task.generation && + entry.scopeEpoch === task.scopeEpoch && + task.scopeEpoch === this.scopeEpochForSource(task.sourceRoot) + ); + } + + private scopeContainsSource(scopeRoot: string, sourceRoot: string): boolean { + const relative = path.relative(scopeRoot, sourceRoot); + return ( + relative !== ".." && + !relative.startsWith(`..${path.sep}`) && + !path.isAbsolute(relative) + ); + } + + private scopeEpochForSource(sourceRoot: string): number { + let epoch = 0; + for (const [scopeRoot, candidateEpoch] of this.invalidatedScopes) { + if (this.scopeContainsSource(scopeRoot, sourceRoot)) { + epoch = Math.max(epoch, candidateEpoch); + } + } + return epoch; + } + + private refreshObservationCoverage(): void { + let count = 0; + let truncated = false; + for (const [sourceRoot, entry] of this.background) { + if (entry.scopeEpoch !== this.scopeEpochForSource(sourceRoot)) continue; + count += entry.snapshot?.result.observedPaths?.length ?? 0; + if (count > INVOCATION_OBSERVATION_MAX_PATHS) { + truncated = true; + break; + } + } + if (truncated === this.observationsTruncated) return; + this.observationsTruncated = truncated; + // Coverage is part of every retained ready snapshot's effective + // completeness. Crossing the global observation cap (in either direction) + // therefore changes more than the task that happened to settle/retire. + for (const [sourceRoot, entry] of this.background) { + if ( + entry.snapshot && + entry.scopeEpoch === this.scopeEpochForSource(sourceRoot) + ) { + this.pendingChanges.add(sourceRoot); + } + } + this.scheduleChanges(); + } + + private dropQueued(sourceRoot: string): void { + for (let index = this.queued.length - 1; index >= 0; index -= 1) { + if (this.queued[index]!.sourceRoot === sourceRoot) { + this.queued.splice(index, 1); + } + } + } + + private drain(): void { + for (let index = this.queued.length - 1; index >= 0; index -= 1) { + if (!this.current(this.queued[index]!)) this.queued.splice(index, 1); + } + const concurrency = Math.max(1, this.options.concurrency ?? 4); + while (this.activeCount < concurrency) { + const index = this.queued.findIndex( + (task) => !this.active.has(task.sourceRoot), + ); + if (index === -1) break; + const [task] = this.queued.splice(index, 1); + if (!task || !this.current(task)) continue; + this.active.set(task.sourceRoot, task); + this.activeCount += 1; + void this.run(task).finally(() => { + if (this.active.get(task.sourceRoot) === task) { + this.active.delete(task.sourceRoot); + this.activeCount -= 1; + } + this.drain(); + }); + } + } + + private async run(task: InvocationTask): Promise { + let snapshot: AgentInvocationSnapshot; + try { + snapshot = { + status: "ready", + result: await this.inner.listInvocations(task.caller), + }; + } catch { + snapshot = { + status: "failed", + result: { invocations: [], warnings: [] }, + }; + } + if (!this.current(task)) return; + this.background.set(task.sourceRoot, { + generation: task.generation, + scopeEpoch: task.scopeEpoch, + snapshot, + }); + this.refreshObservationCoverage(); + this.pendingChanges.add(task.sourceRoot); + this.scheduleChanges(); + } + + private scheduleChanges(): void { + if (this.changeFlushTimer) return; + this.changeFlushTimer = setTimeout(() => { + this.changeFlushTimer = null; + this.flushChanges(); + }, this.options.changeBatchMs ?? 0); + } + + private flushChanges(): void { + if (this.pendingChanges.size === 0) return; + const changed = [...this.pendingChanges] + .filter((sourceRoot) => { + const entry = this.background.get(sourceRoot); + return ( + entry?.snapshot !== undefined && + entry.scopeEpoch === this.scopeEpochForSource(sourceRoot) + ); + }) + .sort(); + this.pendingChanges.clear(); + if (changed.length === 0) return; + void Promise.resolve() + .then(() => this.options.onChange?.(changed)) + .catch(() => { + // Refresh hints cannot invalidate current invocation evidence. + }); } } @@ -115,34 +465,45 @@ const MODE_ORDER: Record = { async: 1, }; -/** V0 filesystem adapter. It remains syntax-only and has no inventory target - * resolution, renderer, transport, deployment, or session dependencies. */ -export class SourceAgentRelationshipProvider implements AgentRelationshipProvider { - async listRelationships( +/** + * V0 per-agent filesystem adapter for literal direct invocations. + * + * It scans only the caller's inventoried source root. It remains syntax-only + * and has no inventory target resolution, input-provenance analysis, package + * router scan, renderer, transport, deployment, or session dependencies. + */ +export class SourceAgentInvocationProvider implements AgentInvocationProvider { + constructor(private readonly readHooks: WorkflowSourceReadHooks = {}) {} + + async listInvocations( caller: AgentInventoryItem, - ): Promise { - const scan = await detectAgentInvocations(caller.sourceRoot, new Set()); - const grouped = new Map(); - - for (const invocation of scan.invocations) { - const key = `${invocation.slug}\0${invocation.mode}`; - const relationship = grouped.get(key); - if (relationship) { - relationship.evidence.push(invocation.evidence); + ): Promise { + const scan = await detectAgentInvocations( + caller.sourceRoot, + new Set(), + this.readHooks, + ); + const grouped = new Map(); + + for (const detectedInvocation of scan.invocations) { + const key = `${detectedInvocation.slug}\0${detectedInvocation.mode}`; + const candidate = grouped.get(key); + if (candidate) { + candidate.evidence.push(detectedInvocation.evidence); } else { grouped.set(key, { - target: invocation.slug, - mode: invocation.mode, - evidence: [invocation.evidence], + target: detectedInvocation.slug, + mode: detectedInvocation.mode, + evidence: [detectedInvocation.evidence], }); } } - const relationships = [...grouped.values()]; - for (const relationship of relationships) { - relationship.evidence.sort(evidenceOrder); + const invocations = [...grouped.values()]; + for (const candidate of invocations) { + candidate.evidence.sort(evidenceOrder); } - relationships.sort( + invocations.sort( (left, right) => left.evidence[0]!.file.localeCompare(right.evidence[0]!.file) || left.evidence[0]!.line - right.evidence[0]!.line || @@ -151,6 +512,11 @@ export class SourceAgentRelationshipProvider implements AgentRelationshipProvide left.target.localeCompare(right.target), ); - return { relationships, warnings: scan.warnings }; + return { + invocations, + warnings: scan.warnings, + observedPaths: scan.observedPaths, + complete: scan.complete, + }; } } diff --git a/packages/harness/src/core/system-graph-store.test.ts b/packages/harness/src/core/system-graph-store.test.ts index 39d27efcf..cc14f7ae6 100644 --- a/packages/harness/src/core/system-graph-store.test.ts +++ b/packages/harness/src/core/system-graph-store.test.ts @@ -481,4 +481,98 @@ describe("SystemGraphStore", () => { expect(returned.revision).toBeGreaterThan(first.revision); }); + + it("does not build a cold projection until its inventory prerequisite is accepted", async () => { + const build = vi.fn().mockResolvedValue(buildResult("accepted")); + const store = new SystemGraphStore({ build }); + + const blocked = store.markStale(scope, "scan:workspace"); + await expect(store.get(scope)).resolves.toBe(blocked); + expect(build).not.toHaveBeenCalled(); + + store.releasePrerequisite(scope, "scan:workspace"); + await vi.waitFor(() => expect(build).toHaveBeenCalledTimes(1)); + await vi.waitFor(() => { + expect(store.peek(scope.workspaceKey)).toMatchObject({ + state: "ready", + graph: graphFor("accepted"), + }); + }); + }); + + it("keeps overlapping prerequisites stale until every scan accepts", async () => { + const build = vi + .fn() + .mockResolvedValueOnce(buildResult("initial")) + .mockResolvedValueOnce(buildResult("accepted")); + const store = new SystemGraphStore({ build }); + await store.get(scope); + + store.markStale(scope, "scan:parent"); + store.markStale(scope, "scan:child"); + store.releasePrerequisite(scope, "scan:child"); + store.requestRefresh(scope); // Late identity enrichment cannot bypass it. + + expect(store.peek(scope.workspaceKey)).toMatchObject({ + state: "stale", + graph: graphFor("initial"), + }); + expect(build).toHaveBeenCalledTimes(1); + + store.releasePrerequisite(scope, "scan:parent"); + await vi.waitFor(() => expect(build).toHaveBeenCalledTimes(2)); + await vi.waitFor(() => { + expect(store.peek(scope.workspaceKey)).toMatchObject({ + state: "ready", + graph: graphFor("accepted"), + }); + }); + }); + + it("does not let a superseded build or identity refresh bypass a prerequisite", async () => { + const obsolete = deferred(); + const accepted = deferred(); + const build = vi + .fn() + .mockResolvedValueOnce(buildResult("initial")) + .mockReturnValueOnce(obsolete.promise) + .mockReturnValueOnce(accepted.promise); + const store = new SystemGraphStore({ build }); + await store.get(scope); + store.requestRefresh(scope); + + store.markStale(scope, "scan:workspace"); + store.requestRefresh(scope); // An older identity task settles while blocked. + obsolete.resolve(buildResult("obsolete")); + await Promise.resolve(); + await Promise.resolve(); + + expect(build).toHaveBeenCalledTimes(2); + expect(store.peek(scope.workspaceKey)).toMatchObject({ + state: "stale", + graph: graphFor("initial"), + }); + + store.releasePrerequisite(scope, "scan:workspace"); + await vi.waitFor(() => expect(build).toHaveBeenCalledTimes(3)); + accepted.resolve(buildResult("accepted")); + await vi.waitFor(() => { + expect(store.peek(scope.workspaceKey)?.graph).toEqual( + graphFor("accepted"), + ); + }); + }); + + it("does not recreate a retired scope while awaiting an accepted build", async () => { + const store = new SystemGraphStore({ + build: vi.fn().mockResolvedValue(buildResult("ready")), + }); + store.markStale(scope, "scan:workspace"); + store.retire(scope.workspaceKey); + + await expect( + store.waitForCurrentRefresh(scope.workspaceKey), + ).resolves.toBeNull(); + expect(store.peek(scope.workspaceKey)).toBeNull(); + }); }); diff --git a/packages/harness/src/core/system-graph-store.ts b/packages/harness/src/core/system-graph-store.ts index 3e10f05b7..2fdec923a 100644 --- a/packages/harness/src/core/system-graph-store.ts +++ b/packages/harness/src/core/system-graph-store.ts @@ -21,6 +21,7 @@ interface SystemGraphEntry { generation: number; refreshPending: boolean; automaticRetryUsed: boolean; + prerequisiteTokens: Set; retired: boolean; } @@ -66,6 +67,9 @@ export class SystemGraphStore { get(scope: WorkspaceScope): Promise { const entry = this.ensureEntry(scope); + if (entry.prerequisiteTokens.size > 0) { + return Promise.resolve(entry.snapshot); + } if (entry.activeBuild) { return entry.snapshot.graph === null ? entry.activeBuild @@ -97,6 +101,9 @@ export class SystemGraphStore { ensureInitialized(scope: WorkspaceScope): Promise { const entry = this.entries.get(scope.workspaceKey); if (!entry) return this.get(scope); + if (entry.prerequisiteTokens.size > 0) { + return Promise.resolve(entry.snapshot); + } if (entry.activeBuild && entry.snapshot.graph === null) { return entry.activeBuild; } @@ -107,17 +114,88 @@ export class SystemGraphStore { requestRefresh(scope: WorkspaceScope): SystemGraphSnapshot { const entry = this.ensureEntry(scope); entry.automaticRetryUsed = false; + if (entry.prerequisiteTokens.size > 0) { + entry.refreshPending = true; + return entry.snapshot; + } this.queueRefresh(entry); return entry.snapshot; } + /** + * Fails closed on a raw inventory/source signal without projecting the old + * registry snapshot. The reconciliation coordinator starts the replacement + * build only after its atomic snapshot commit succeeds. + */ + markStale( + scope: WorkspaceScope, + prerequisiteToken = "inventory", + ): SystemGraphSnapshot { + const entry = this.ensureEntry(scope); + entry.prerequisiteTokens.add(prerequisiteToken); + entry.generation += 1; + entry.refreshPending = false; + entry.automaticRetryUsed = true; + const visible = visibleProjection(entry); + return visible.graph === null + ? this.transition(entry, "building", null) + : this.transition( + entry, + "stale", + visible.graph, + false, + visible.navigation, + ); + } + /** Explicit user recovery: start a fresh projection and await its result. */ refresh(scope: WorkspaceScope): Promise { const entry = this.ensureEntry(scope); entry.automaticRetryUsed = false; + if (entry.prerequisiteTokens.size > 0) { + entry.refreshPending = true; + return entry.activeBuild ?? Promise.resolve(entry.snapshot); + } return this.queueRefresh(entry) ?? Promise.resolve(entry.snapshot); } + /** + * Releases one accepted inventory prerequisite. A projection is rebuilt + * only after every overlapping scan has accepted its current generation. + */ + releasePrerequisite( + scope: WorkspaceScope, + prerequisiteToken = "inventory", + ): SystemGraphSnapshot { + const entry = this.ensureEntry(scope); + if (!entry.prerequisiteTokens.delete(prerequisiteToken)) { + return entry.snapshot; + } + if (entry.prerequisiteTokens.size > 0) return entry.snapshot; + entry.automaticRetryUsed = false; + this.queueRefresh(entry); + return entry.snapshot; + } + + /** Retires a failed prerequisite without projecting the pre-mutation cache. */ + cancelPrerequisite( + workspaceKey: WorkspaceKey, + prerequisiteToken = "inventory", + ): SystemGraphSnapshot | null { + const entry = this.entries.get(workspaceKey); + if (!entry) return null; + entry.prerequisiteTokens.delete(prerequisiteToken); + return entry.snapshot; + } + + /** Awaits the already-started accepted-snapshot build without queuing one. */ + waitForCurrentRefresh( + workspaceKey: WorkspaceKey, + ): Promise { + const entry = this.entries.get(workspaceKey); + return entry?.activeBuild ?? Promise.resolve(entry?.snapshot ?? null); + } + peek(workspaceKey: WorkspaceKey): SystemGraphSnapshot | null { return this.entries.get(workspaceKey)?.snapshot ?? null; } @@ -203,6 +281,7 @@ export class SystemGraphStore { generation: 0, refreshPending: false, automaticRetryUsed: false, + prerequisiteTokens: new Set(), retired: false, }; this.entries.set(scope.workspaceKey, entry); @@ -266,11 +345,11 @@ export class SystemGraphStore { generation: number, result: Awaited>, ): SystemGraphSnapshot | Promise { - // The superseded build can carry the only callback that starts identity - // enrichment. Arm it after the commit decision on both paths: before that - // decision a synchronous refresh could supersede the result being - // committed; omitting it from the losing path can leave identities pending - // forever with no follow-up queued. + // The superseded build can carry the only callback that starts background + // identity or invocation enrichment. Arm it after the commit decision on + // both paths: before that decision a synchronous refresh could supersede + // the result being committed; omitting it from the losing path can leave + // enrichment pending forever with no follow-up queued. if (!this.canCommit(entry, generation)) { const superseded = this.continueAfterSupersededBuild(entry); this.afterCommit(result.afterCommit); @@ -339,6 +418,9 @@ export class SystemGraphStore { this.retainBuilderWorkspaces(); return entry.snapshot; } + if (entry.prerequisiteTokens.size > 0) { + return entry.snapshot; + } if (entry.refreshPending) { entry.refreshPending = false; return this.startBuild(entry); diff --git a/packages/harness/src/core/system-graph-watcher.test.ts b/packages/harness/src/core/system-graph-watcher.test.ts index ba32a788a..468fa7c7f 100644 --- a/packages/harness/src/core/system-graph-watcher.test.ts +++ b/packages/harness/src/core/system-graph-watcher.test.ts @@ -5,10 +5,16 @@ import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; import type { WorkspaceScope } from "./system-graph.js"; import { + SharedWorkspaceWatchBroker, SystemGraphWatcherManager, type SystemGraphWatchFactory, type SystemGraphWatchHandle, } from "./system-graph-watcher.js"; +import { + snapshotWorkflowSourceRootsAsync, + snapshotWorkspaceWorkflowsAsync, + WorkspaceWatcherManager, +} from "./workspace-watcher.js"; const sleep = (ms: number): Promise => new Promise((resolve) => setTimeout(resolve, ms)); @@ -19,6 +25,7 @@ let manager: SystemGraphWatcherManager; let onSourceChange: ReturnType; let onInventoryChange: ReturnType; let sourceRoots: Set; +let sourceObservations: Map>; async function scaffoldAgent(name: string): Promise { const agentRoot = path.join(root, name); @@ -29,6 +36,10 @@ async function scaffoldAgent(name: string): Promise { ); await fs.writeFile(path.join(agentRoot, "index.ts"), "export {};\n"); sourceRoots.add(agentRoot); + sourceObservations.set( + agentRoot, + new Set([path.join(agentRoot, "index.ts")]), + ); return agentRoot; } @@ -39,9 +50,16 @@ describe("SystemGraphWatcherManager", () => { onSourceChange = vi.fn(); onInventoryChange = vi.fn(); sourceRoots = new Set(); + sourceObservations = new Map(); manager = new SystemGraphWatcherManager( { listSourceRoots: () => [...sourceRoots], + listSourceObservations: () => + [...sourceObservations].map(([candidateRoot, paths]) => ({ + candidateRoot, + workspaceRoot: root, + paths: [...paths], + })), onSourceChange, onInventoryChange, }, @@ -61,11 +79,12 @@ describe("SystemGraphWatcherManager", () => { await fs.rm(root, { recursive: true, force: true }); }); - it("refreshes source relationships without reporting inventory churn", async () => { + it("refreshes source invocations without reporting inventory churn", async () => { const agentRoot = await scaffoldAgent("research"); await scaffoldAgent("growth"); await manager.start(scope); await sleep(100); + onSourceChange.mockClear(); onInventoryChange.mockClear(); await fs.writeFile( @@ -101,6 +120,12 @@ describe("SystemGraphWatcherManager", () => { manager = new SystemGraphWatcherManager( { listSourceRoots: () => [...sourceRoots], + listSourceObservations: () => + [...sourceObservations].map(([candidateRoot, paths]) => ({ + candidateRoot, + workspaceRoot: root, + paths: [...paths], + })), onSourceChange, onInventoryChange, }, @@ -113,6 +138,8 @@ describe("SystemGraphWatcherManager", () => { ); const agentRoot = await scaffoldAgent("research"); await manager.start(scope); + await sleep(100); + onSourceChange.mockClear(); watchListener("change", "research/index.ts"); await vi.waitFor(() => expect(onSourceChange).toHaveBeenCalled(), { @@ -128,6 +155,17 @@ describe("SystemGraphWatcherManager", () => { await sleep(50); expect(onSourceChange).not.toHaveBeenCalled(); + onInventoryChange.mockClear(); + watchListener("rename", "research/.git"); + await vi.waitFor(() => expect(onInventoryChange).toHaveBeenCalledOnce(), { + timeout: 2_000, + interval: 20, + }); + onInventoryChange.mockClear(); + watchListener("rename", "research/.git/objects/pack-1"); + await sleep(50); + expect(onInventoryChange).not.toHaveBeenCalled(); + errorListener(new Error("recursive watch unavailable")); expect(close).toHaveBeenCalledTimes(1); await sleep(100); @@ -193,6 +231,63 @@ describe("SystemGraphWatcherManager", () => { ); }); + it("bounds source retries until a later raw event rearms recovery", async () => { + let watchListener!: Parameters[1]; + const watchFactory: SystemGraphWatchFactory = (_watchRoot, listener) => { + watchListener = listener; + const handle: SystemGraphWatchHandle = { + close: vi.fn(), + on: () => handle, + }; + return handle; + }; + const agentRoot = await scaffoldAgent("retry-source"); + onSourceChange.mockRejectedValue(new Error("registry unavailable")); + manager = new SystemGraphWatcherManager( + { + listSourceRoots: () => [...sourceRoots], + listSourceObservations: () => [], + onSourceChange, + onInventoryChange, + }, + { + watchFactory, + sourceDebounceMs: 1, + inventoryRetryBaseMs: 10, + maxSourceRetries: 2, + }, + ); + await manager.start(scope); + + vi.useFakeTimers(); + try { + watchListener( + "change", + path.relative(root, path.join(agentRoot, "index.ts")), + ); + await vi.advanceTimersByTimeAsync(1); + expect(onSourceChange).toHaveBeenCalledTimes(1); + await vi.advanceTimersByTimeAsync(10); + expect(onSourceChange).toHaveBeenCalledTimes(2); + await vi.advanceTimersByTimeAsync(20); + expect(onSourceChange).toHaveBeenCalledTimes(3); + await vi.advanceTimersByTimeAsync(10_000); + expect(onSourceChange).toHaveBeenCalledTimes(3); + + watchListener( + "change", + path.relative(root, path.join(agentRoot, "index.ts")), + ); + await vi.advanceTimersByTimeAsync(1); + await vi.advanceTimersByTimeAsync(10); + await vi.advanceTimersByTimeAsync(20); + expect(onSourceChange).toHaveBeenCalledTimes(6); + } finally { + manager.stopAll(); + vi.useRealTimers(); + } + }); + it("polls every registered source file past Canvas's project-sized cap", async () => { const agentRoot = await scaffoldAgent("large"); await Promise.all( @@ -203,6 +298,7 @@ describe("SystemGraphWatcherManager", () => { ), ), ); + sourceObservations.get(agentRoot)?.add(path.join(agentRoot, "step-424.ts")); await manager.start(scope); await sleep(200); onSourceChange.mockClear(); @@ -218,8 +314,11 @@ describe("SystemGraphWatcherManager", () => { }); }); - it("ignores non-source and generated-tree churn", async () => { + it("ignores non-source/generated churn but notices an unregistered source candidate", async () => { await manager.start(scope); + await sleep(100); + onSourceChange.mockClear(); + onInventoryChange.mockClear(); await fs.writeFile(path.join(root, "README.md"), "notes\n"); await fs.mkdir(path.join(root, "node_modules", "pkg"), { recursive: true, @@ -236,7 +335,7 @@ describe("SystemGraphWatcherManager", () => { await sleep(300); expect(onSourceChange).not.toHaveBeenCalled(); - expect(onInventoryChange).not.toHaveBeenCalled(); + expect(onInventoryChange).toHaveBeenCalledWith(scope); }); it("does not re-baseline an existing workspace on repeated opens", async () => { @@ -249,4 +348,296 @@ describe("SystemGraphWatcherManager", () => { manager.retain(new Set()); expect(manager.size).toBe(0); }); + + it("closes and suppresses a pending watcher immediately when its scope retires", async () => { + let releaseBaseline!: () => void; + const baselineGate = new Promise((resolve) => { + releaseBaseline = resolve; + }); + let watchListener!: Parameters[1]; + const close = vi.fn(); + const onPotentialChange = vi.fn(); + const watchFactory: SystemGraphWatchFactory = (_watchRoot, listener) => { + watchListener = listener; + const handle: SystemGraphWatchHandle = { + close, + on: () => handle, + }; + return handle; + }; + manager = new SystemGraphWatcherManager( + { + listSourceRoots: () => [], + onSourceChange, + onInventoryChange, + onPotentialChange, + }, + { + watchFactory, + beforeInitialSnapshot: () => baselineGate, + sourceDebounceMs: 5, + inventoryDebounceMs: 5, + }, + ); + + const starting = manager.start(scope); + manager.retain(new Set()); + + expect(close).toHaveBeenCalledTimes(1); + expect(manager.size).toBe(0); + watchListener("rename", "agent/index.ts"); + await sleep(20); + expect(onPotentialChange).not.toHaveBeenCalled(); + expect(onSourceChange).not.toHaveBeenCalled(); + expect(onInventoryChange).not.toHaveBeenCalled(); + + releaseBaseline(); + await starting; + expect(manager.size).toBe(0); + expect(close).toHaveBeenCalledTimes(1); + }); + + it("closes an armed watcher when its initial baseline rejects", async () => { + let watchListener!: Parameters[1]; + const close = vi.fn(); + const onPotentialChange = vi.fn(); + const watchFactory: SystemGraphWatchFactory = (_watchRoot, listener) => { + watchListener = listener; + const handle: SystemGraphWatchHandle = { + close, + on: () => handle, + }; + return handle; + }; + manager = new SystemGraphWatcherManager( + { + listSourceRoots: () => [], + onSourceChange, + onInventoryChange, + onPotentialChange, + }, + { + watchFactory, + beforeInitialSnapshot: () => + Promise.reject(new Error("baseline unavailable")), + sourceDebounceMs: 5, + inventoryDebounceMs: 5, + }, + ); + + await expect(manager.start(scope)).rejects.toThrow("baseline unavailable"); + expect(close).toHaveBeenCalledTimes(1); + expect(manager.size).toBe(0); + watchListener("rename", "agent/index.ts"); + await sleep(20); + expect(onPotentialChange).not.toHaveBeenCalled(); + expect(onSourceChange).not.toHaveBeenCalled(); + expect(onInventoryChange).not.toHaveBeenCalled(); + }); + + it("reconciles every retained root when an ambiguous event races the initial baseline", async () => { + const agentRoot = await scaffoldAgent("nested-checkout"); + let releaseBaseline!: () => void; + const baselineGate = new Promise((resolve) => { + releaseBaseline = resolve; + }); + let watchListener!: Parameters[1]; + const watchFactory: SystemGraphWatchFactory = (_watchRoot, listener) => { + watchListener = listener; + const handle: SystemGraphWatchHandle = { + close: vi.fn(), + on: () => handle, + }; + return handle; + }; + manager = new SystemGraphWatcherManager( + { + listSourceRoots: () => [...sourceRoots], + listSourceObservations: () => [], + onSourceChange, + onInventoryChange, + }, + { + watchFactory, + beforeInitialSnapshot: () => baselineGate, + sourceDebounceMs: 5, + inventoryDebounceMs: 5, + }, + ); + + const starting = manager.start(scope); + watchListener("rename", null); + + await vi.waitFor(() => expect(onSourceChange).toHaveBeenCalled(), { + timeout: 2_000, + interval: 10, + }); + expect(onSourceChange.mock.calls.at(-1)?.[1]).toEqual([agentRoot]); + + releaseBaseline(); + await starting; + }); + + it("owns one polling baseline and reconciles an edit absorbed into it", async () => { + const agentRoot = await scaffoldAgent("polling-baseline"); + let releaseBaseline!: () => void; + const baselineGate = new Promise((resolve) => { + releaseBaseline = resolve; + }); + const snapshotWorkspace = vi.fn(snapshotWorkspaceWorkflowsAsync); + const snapshotSources = vi.fn(snapshotWorkflowSourceRootsAsync); + const onPotentialChange = vi.fn(); + manager = new SystemGraphWatcherManager( + { + listSourceRoots: () => [...sourceRoots], + listSourceObservations: () => [], + onSourceChange, + onInventoryChange, + onPotentialChange, + }, + { + forcePolling: true, + beforeInitialSnapshot: () => baselineGate, + snapshotWorkspace, + snapshotSources, + sourceDebounceMs: 5, + pollIntervalMs: 60_000, + }, + ); + + const starting = manager.start(scope); + await Promise.resolve(); + expect(snapshotWorkspace).not.toHaveBeenCalled(); + expect(snapshotSources).not.toHaveBeenCalled(); + + await fs.writeFile( + path.join(agentRoot, "index.ts"), + "export const changedDuringBaseline = true;\n", + ); + releaseBaseline(); + await starting; + + await vi.waitFor(() => expect(onSourceChange).toHaveBeenCalledOnce(), { + timeout: 2_000, + interval: 10, + }); + expect(snapshotWorkspace).toHaveBeenCalledTimes(1); + expect(snapshotSources).toHaveBeenCalledTimes(1); + expect(onPotentialChange).toHaveBeenCalledOnce(); + expect(onPotentialChange).toHaveBeenCalledWith(scope, null); + expect(onSourceChange).toHaveBeenCalledWith(scope, [agentRoot]); + }); + + it("shares one canonical-root watcher across two sessions and a graph caller", async () => { + const agentRoot = await scaffoldAgent("shared-agent"); + let watchListener!: Parameters[1]; + const close = vi.fn(); + const watchFactory = vi.fn( + (_watchRoot, listener) => { + watchListener = listener; + const handle: SystemGraphWatchHandle = { + close, + on: () => handle, + }; + return handle; + }, + ); + const broker = new SharedWorkspaceWatchBroker({ + watchFactory, + sourceDebounceMs: 5, + inventoryDebounceMs: 5, + }); + const sessionChange = vi.fn(); + const sessionPotential = vi.fn(); + const sessions = new WorkspaceWatcherManager({ + sharedWatchBroker: broker, + listSourceRoots: () => [...sourceRoots], + listSourceObservations: () => [], + onPotentialChange: sessionPotential, + onChange: sessionChange, + }); + const graphSourceChange = vi.fn(); + const graphPotential = vi.fn(); + manager = new SystemGraphWatcherManager( + { + listSourceRoots: () => [...sourceRoots], + listSourceObservations: () => [], + onPotentialChange: graphPotential, + onSourceChange: graphSourceChange, + onInventoryChange, + }, + { sharedBroker: broker }, + ); + + sessions.start("session-a", root); + sessions.start("session-b", path.join(root, ".")); + await manager.start(scope); + + expect(watchFactory).toHaveBeenCalledTimes(1); + expect(broker.size).toBe(1); + watchListener( + "change", + path.relative(root, path.join(agentRoot, "index.ts")), + ); + + await vi.waitFor(() => expect(graphSourceChange).toHaveBeenCalledOnce(), { + timeout: 2_000, + interval: 10, + }); + expect(sessionChange).toHaveBeenCalledTimes(2); + expect(sessionChange.mock.calls.map((call) => call[0]).sort()).toEqual([ + "session-a", + "session-b", + ]); + expect(sessionChange.mock.calls.map((call) => call[1])).toEqual([ + [agentRoot], + [agentRoot], + ]); + expect(sessionPotential).toHaveBeenCalledTimes(2); + expect(graphPotential).toHaveBeenCalledOnce(); + + sessions.stopAll(); + manager.retain(new Set()); + expect(broker.size).toBe(0); + expect(close).toHaveBeenCalledTimes(1); + }); + + it("cleans a rejected shared subscription so the same scope can retry", async () => { + let attempts = 0; + const closes: Array> = []; + const watchFactory = vi.fn(() => { + const close = vi.fn(); + closes.push(close); + const handle: SystemGraphWatchHandle = { + close, + on: () => handle, + }; + return handle; + }); + const broker = new SharedWorkspaceWatchBroker({ + watchFactory, + beforeInitialSnapshot: () => { + attempts += 1; + if (attempts === 1) throw new Error("first baseline failed"); + }, + }); + manager = new SystemGraphWatcherManager( + { + listSourceRoots: () => [], + onSourceChange, + onInventoryChange, + }, + { sharedBroker: broker }, + ); + + await expect(manager.start(scope)).rejects.toThrow("first baseline failed"); + expect(manager.size).toBe(0); + expect(broker.size).toBe(0); + expect(closes[0]).toHaveBeenCalledTimes(1); + + await manager.start(scope); + expect(watchFactory).toHaveBeenCalledTimes(2); + expect(manager.size).toBe(1); + expect(broker.size).toBe(1); + }); }); diff --git a/packages/harness/src/core/system-graph-watcher.ts b/packages/harness/src/core/system-graph-watcher.ts index 083847735..c6c6dd865 100644 --- a/packages/harness/src/core/system-graph-watcher.ts +++ b/packages/harness/src/core/system-graph-watcher.ts @@ -1,26 +1,34 @@ import * as fs from "node:fs"; -import * as fsp from "node:fs/promises"; import * as path from "node:path"; import type { WorkspaceKey } from "../shared/system-graph.js"; import { isAgentProjectScanIgnoredDir } from "./agent-project-discovery.js"; import { normalizeWatchPath } from "./canvas-watcher.js"; +import { canonicalGraphPath } from "./canonical-graph-path.js"; import type { WorkspaceScope } from "./system-graph.js"; -import { snapshotWorkspaceWorkflowsAsync } from "./workspace-watcher.js"; +import { + snapshotWorkflowSourceRootsAsync, + snapshotWorkspaceWorkflowsAsync, + type WorkflowSourceObservation, +} from "./workspace-watcher.js"; const SOURCE_DEBOUNCE_MS = 150; const INVENTORY_DEBOUNCE_MS = 250; const INVENTORY_RETRY_BASE_MS = 500; const MAX_INVENTORY_RETRIES = 3; +const MAX_SOURCE_RETRIES = 3; const POLL_INTERVAL_MS = 2_000; -const SOURCE_EXTENSIONS = new Set([".ts", ".tsx"]); -const UNREADABLE_SOURCE_FINGERPRINT = ""; +const SOURCE_EXTENSIONS = new Set([".ts", ".tsx", ".mts", ".cts"]); function ignoredRelativePath(relativePath: string): boolean { - return relativePath - .split("/") - .filter(Boolean) - .some((segment) => isAgentProjectScanIgnoredDir(segment)); + const segments = relativePath.split("/").filter(Boolean); + const ignoredIndex = segments.findIndex((segment) => + isAgentProjectScanIgnoredDir(segment), + ); + // Creating/removing the boundary directory itself (notably `agent/.git`) + // changes whether a parent scan is allowed to own that candidate. Churn + // below an established boundary remains ignored. + return ignoredIndex >= 0 && ignoredIndex < segments.length - 1; } function sourceRelativePath(relativePath: string): boolean { @@ -43,60 +51,6 @@ function confinedSourcePath(root: string, relativePath: string): string | null { return absolute; } -/** - * Async source fingerprint for the registered agent roots in one workspace. - * - * The graph watcher deliberately does not reuse Canvas's synchronous, - * 400-file project snapshot here: a workspace can contain many agent projects, - * and polling it on the server event loop would both stutter Studio and miss - * edits after that project-sized ceiling. Async directory reads yield between - * entries, while scoping the walk to registry roots keeps the unbounded file - * count honest without traversing unrelated workspace trees. - */ -export async function snapshotWorkflowSourceRootsAsync( - sourceRoots: readonly string[], -): Promise> { - const roots = [ - ...new Set(sourceRoots.map((root) => path.resolve(root))), - ].sort(); - const snapshots = new Map(); - - const walk = async (dir: string, parts: string[]): Promise => { - let entries: fs.Dirent[]; - try { - entries = await fsp.readdir(dir, { withFileTypes: true }); - } catch { - parts.push(`${dir}\0${UNREADABLE_SOURCE_FINGERPRINT}`); - return; - } - - for (const entry of entries) { - const full = path.join(dir, entry.name); - if (entry.isDirectory()) { - if (isAgentProjectScanIgnoredDir(entry.name)) continue; - await walk(full, parts); - continue; - } - if (!entry.isFile() || !SOURCE_EXTENSIONS.has(path.extname(entry.name))) { - continue; - } - try { - const stat = await fsp.stat(full); - parts.push(`${full}:${stat.mtimeMs}:${stat.size}`); - } catch { - parts.push(`${full}:gone`); - } - } - }; - - for (const root of roots) { - const parts = [`root\0${root}`]; - await walk(root, parts); - snapshots.set(root, parts.sort().join("|")); - } - return snapshots; -} - function isNestedSourceRoot(parent: string, candidate: string): boolean { const relative = path.relative(parent, candidate); return ( @@ -130,12 +84,20 @@ function changedSourceRoots( export interface SystemGraphWatcherCallbacks { /** Current registry roots inside this scope; read lazily on every poll. */ listSourceRoots: (scope: WorkspaceScope) => readonly string[]; + listSourceObservations?: ( + scope: WorkspaceScope, + ) => readonly WorkflowSourceObservation[]; onSourceChange: ( scope: WorkspaceScope, /** Null when the platform can only report a workspace-level change. */ sourcePaths: readonly string[] | null, ) => void | Promise; onInventoryChange: (scope: WorkspaceScope) => void | Promise; + /** Synchronous raw-event fail-close hook, before debounce or async I/O. */ + onPotentialChange?: ( + scope: WorkspaceScope, + sourcePaths: readonly string[] | null, + ) => void; } export interface SystemGraphWatchHandle { @@ -153,57 +115,159 @@ export interface SystemGraphWatcherOptions { inventoryDebounceMs?: number; inventoryRetryBaseMs?: number; maxInventoryRetries?: number; + maxSourceRetries?: number; pollIntervalMs?: number; /** Deterministic test seam for the supported polling fallback. */ forcePolling?: boolean; /** Deterministic test seam for native event routing and watcher errors. */ watchFactory?: SystemGraphWatchFactory; + /** Deterministic lifecycle seam: the native handle is armed before this. */ + beforeInitialSnapshot?: () => void | Promise; + /** Deterministic test seams for proving one owned polling baseline. */ + snapshotWorkspace?: (root: string) => Promise; + snapshotSources?: ( + roots: readonly string[], + observations: readonly WorkflowSourceObservation[], + ) => Promise>; + /** Optional process-wide root broker shared with session watchers. */ + sharedBroker?: SharedWorkspaceWatchBrokerLike; + /** Fresh process-local proof expires when the final continuous lease ends. */ + onLastLeaseReleased?: (canonicalRoot: string) => void; +} + +export interface SharedWorkspaceWatchSubscriber { + scope: WorkspaceScope; + listSourceRoots: () => readonly string[]; + listSourceObservations?: () => readonly WorkflowSourceObservation[]; + onSourceChange: ( + sourcePaths: readonly string[] | null, + ) => void | Promise; + onInventoryChange: () => void | Promise; + onPotentialChange?: (sourcePaths: readonly string[] | null) => void; +} + +export interface SharedWorkspaceWatchBrokerLike { + subscribe( + key: object, + subscriber: SharedWorkspaceWatchSubscriber, + ): Promise; + unsubscribe(key: object): void; } class WorkspaceSystemGraphWatcher { private watcher: SystemGraphWatchHandle | null = null; private pollTimer: ReturnType | null = null; private sourceTimer: ReturnType | null = null; + private sourceRetryTimer: ReturnType | null = null; private inventoryTimer: ReturnType | null = null; private inventoryRetryTimer: ReturnType | null = null; private closed = false; private sourcePaths = new Set(); private ambiguousSourceChange = false; private lastSourceSnapshots: ReadonlyMap | null = null; - private lastInventorySnapshot: string; + private lastInventorySnapshot: string | null; private failedInventorySnapshot: string | null = null; private inventoryGeneration = 0; private inventoryCheckInFlight = false; private inventoryCheckPending = false; + private inventoryReconcilePending = false; private pollInFlight = false; + private initialized = false; + private polling = false; private callbackQueue: Promise = Promise.resolve(); + private rawGeneration = 0; + private sourceGeneration = 0; private constructor( readonly scope: WorkspaceScope, private readonly callbacks: SystemGraphWatcherCallbacks, private readonly options: SystemGraphWatcherOptions, - initialInventorySnapshot: string, ) { - this.lastInventorySnapshot = initialInventorySnapshot; + this.lastInventorySnapshot = null; this.arm(); } - static async create( + static begin( scope: WorkspaceScope, callbacks: SystemGraphWatcherCallbacks, options: SystemGraphWatcherOptions, - ): Promise { - // Establish the native/polling watcher from an async baseline. The graph - // route awaits this factory, so no synchronous workspace walk runs on the - // server loop before the watcher starts. - const initialInventorySnapshot = await snapshotWorkspaceWorkflowsAsync( - scope.root, + ): { watcher: WorkspaceSystemGraphWatcher; ready: Promise } { + // Arm before reading the async baseline. A generation change during that + // walk forces a reconciliation callback plus a trailing fingerprint pass. + const watcher = new WorkspaceSystemGraphWatcher(scope, callbacks, options); + return { watcher, ready: watcher.initialize() }; + } + + private async initialize(): Promise { + const generation = this.rawGeneration; + await this.options.beforeInitialSnapshot?.(); + if (this.closed) return; + let sourceRoots: readonly string[] = []; + let sourceObservations: readonly WorkflowSourceObservation[] = []; + try { + sourceRoots = this.callbacks.listSourceRoots(this.scope); + sourceObservations = + this.callbacks.listSourceObservations?.(this.scope) ?? []; + } catch { + // The workspace baseline remains useful and the next poll retries. + } + const [initialInventorySnapshot, initialSourceSnapshots] = + await Promise.all([ + this.snapshotWorkspace(this.scope.root), + this.snapshotSources(sourceRoots, sourceObservations), + ]); + if (this.closed) return; + if (this.lastInventorySnapshot === null) { + this.lastInventorySnapshot = initialInventorySnapshot; + } else if (this.lastInventorySnapshot !== initialInventorySnapshot) { + this.checkInventoryAsync(); + } + if (this.lastSourceSnapshots === null) { + this.lastSourceSnapshots = initialSourceSnapshots; + } else if ( + changedSourceRoots(this.lastSourceSnapshots, initialSourceSnapshots) + .length > 0 + ) { + this.scheduleSourceChange(null); + } + if (generation !== this.rawGeneration) { + this.checkInventoryAsync(); + } + this.initialized = true; + if (this.polling) { + // Polling has no raw event protecting the baseline walk. Reconcile once + // from the baseline we just accepted, without launching a competing + // second workspace/source walk. This covers an edit that landed between + // watcher setup and the first sample while retaining a single owner for + // startup I/O. + try { + this.callbacks.onPotentialChange?.(this.scope, null); + } catch { + // The source callback below still performs the bounded reconciliation. + } + const retainedRoots = [...initialSourceSnapshots.keys()].sort(); + this.sourceGeneration += 1; + // An empty concrete list still asks every production subscriber to scan + // its containing workspace. Avoid null here: null intentionally performs + // a fresh source snapshot to attribute an ambiguous runtime event, which + // would reintroduce the duplicate startup walk this path eliminates. + this.dispatchSourceChange(retainedRoots, this.sourceGeneration); + } + } + + private snapshotWorkspace(root: string): Promise { + return (this.options.snapshotWorkspace ?? snapshotWorkspaceWorkflowsAsync)( + root, ); - return new WorkspaceSystemGraphWatcher( - scope, - callbacks, - options, - initialInventorySnapshot, + } + + private snapshotSources( + roots: readonly string[], + observations: readonly WorkflowSourceObservation[], + ): Promise> { + return (this.options.snapshotSources ?? snapshotWorkflowSourceRootsAsync)( + roots, + observations, ); } @@ -227,11 +291,36 @@ class WorkspaceSystemGraphWatcher { }); } + private enqueueSource( + callback: () => void | Promise, + onFailure: () => void, + ): void { + // Source generations intentionally overlap. A raw edit arriving while an + // accepted scan is held must reach the coordinator immediately so it can + // supersede that flight and run one trailing pass; serializing behind the + // older callback would publish the stale pass and then start a third scan. + void Promise.resolve() + .then(async () => { + if (this.closed) return; + await callback(); + }) + .catch(() => { + if (this.closed) return; + try { + onFailure(); + } catch { + // Failure recovery remains best-effort. + } + }); + } + private scheduleSourceChange(sourcePath: string | null): void { if (this.closed) return; if (sourcePath === null) this.ambiguousSourceChange = true; else this.sourcePaths.add(sourcePath); if (this.sourceTimer) clearTimeout(this.sourceTimer); + if (this.sourceRetryTimer) clearTimeout(this.sourceRetryTimer); + this.sourceRetryTimer = null; this.sourceTimer = setTimeout(() => { this.sourceTimer = null; const paths = this.ambiguousSourceChange @@ -239,10 +328,73 @@ class WorkspaceSystemGraphWatcher { : [...this.sourcePaths].sort(); this.sourcePaths.clear(); this.ambiguousSourceChange = false; - this.enqueue(() => this.callbacks.onSourceChange(this.scope, paths)); + this.sourceGeneration += 1; + this.dispatchSourceChange(paths, this.sourceGeneration); }, this.options.sourceDebounceMs ?? SOURCE_DEBOUNCE_MS); } + private dispatchSourceChange( + paths: readonly string[] | null, + generation: number, + retry = 0, + ): void { + this.enqueueSource( + async () => { + if (generation !== this.sourceGeneration) return; + let effectivePaths = paths; + let observedSnapshots: ReadonlyMap | null = null; + if (paths === null) { + let roots: readonly string[] = []; + let observations: readonly WorkflowSourceObservation[] = []; + try { + roots = this.callbacks.listSourceRoots(this.scope); + observations = + this.callbacks.listSourceObservations?.(this.scope) ?? []; + } catch { + // Keep the conservative scope-wide callback. + } + observedSnapshots = await this.snapshotSources(roots, observations); + if (this.lastSourceSnapshots) { + const changed = changedSourceRoots( + this.lastSourceSnapshots, + observedSnapshots, + ); + effectivePaths = changed.length > 0 ? changed : null; + } else { + // A raw event raced the initial baseline. The post-edit sample + // cannot identify a delta, so reconcile every retained root rather + // than only the parent scope (which may stop at repo/ignore roots). + const retainedRoots = [...observedSnapshots.keys()].sort(); + effectivePaths = retainedRoots.length > 0 ? retainedRoots : null; + } + } + await this.callbacks.onSourceChange(this.scope, effectivePaths); + if (generation !== this.sourceGeneration || this.closed) return; + if (observedSnapshots) this.lastSourceSnapshots = observedSnapshots; + }, + () => { + if (this.closed || generation !== this.sourceGeneration) return; + if (retry >= (this.options.maxSourceRetries ?? MAX_SOURCE_RETRIES)) { + // Keep the graph fail-closed after bounded recovery. A later raw + // event advances sourceGeneration and rearms a fresh retry series; + // ordinary polling must not become an endless registry-scan loop. + return; + } + const nextRetry = retry + 1; + const delay = Math.min( + 2_000, + (this.options.inventoryRetryBaseMs ?? INVENTORY_RETRY_BASE_MS) * + 2 ** Math.min(nextRetry - 1, 3), + ); + this.sourceRetryTimer = setTimeout(() => { + this.sourceRetryTimer = null; + if (this.closed || generation !== this.sourceGeneration) return; + this.dispatchSourceChange(paths, generation, nextRetry); + }, delay); + }, + ); + } + private dispatchInventoryChange( snapshot: string, retryNumber = 0, @@ -304,7 +456,7 @@ class WorkspaceSystemGraphWatcher { this.inventoryRetryTimer = setTimeout(() => { this.inventoryRetryTimer = null; if (this.closed || generation !== this.inventoryGeneration) return; - void snapshotWorkspaceWorkflowsAsync(this.scope.root) + void this.snapshotWorkspace(this.scope.root) .then((currentSnapshot) => { if (this.closed || generation !== this.inventoryGeneration) return; if (currentSnapshot !== snapshot) { @@ -341,12 +493,15 @@ class WorkspaceSystemGraphWatcher { } this.inventoryCheckInFlight = true; this.inventoryCheckPending = false; - void snapshotWorkspaceWorkflowsAsync(this.scope.root) + void this.snapshotWorkspace(this.scope.root) .then((snapshot) => { if (this.closed) return; + const mustReconcile = this.inventoryReconcilePending; + this.inventoryReconcilePending = false; if ( snapshot === this.lastInventorySnapshot && - snapshot !== this.failedInventorySnapshot + snapshot !== this.failedInventorySnapshot && + !mustReconcile ) { return; } @@ -375,18 +530,43 @@ class WorkspaceSystemGraphWatcher { _event: "rename" | "change", rawFilename: string | null, ): void => { + if (this.closed) return; + const potential = (paths: readonly string[] | null): void => { + this.rawGeneration += 1; + try { + this.callbacks.onPotentialChange?.(this.scope, paths); + } catch { + // Raw-event invalidation is best-effort; async reconciliation still + // runs and reports failures through the normal callback path. + } + }; if (rawFilename === null) { + potential(null); this.scheduleSourceChange(null); - this.scheduleInventoryCheck(); return; } const relativePath = normalizeWatchPath(rawFilename); if (ignoredRelativePath(relativePath)) return; if (sourceRelativePath(relativePath)) { - this.scheduleSourceChange( - confinedSourcePath(this.scope.root, relativePath), - ); + const sourcePath = confinedSourcePath(this.scope.root, relativePath); + potential(sourcePath ? [sourcePath] : null); + this.scheduleSourceChange(sourcePath); + // Source reconciliation also scans inventory. Do not enqueue a + // second serialized inventory callback for the same native edit. + return; + } + const basename = path.posix.basename(relativePath); + if (basename === "sapiom.json" || basename === "package.json") { + const sourcePath = confinedSourcePath(this.scope.root, relativePath); + potential(sourcePath ? [sourcePath] : null); + this.scheduleSourceChange(sourcePath); + return; + } + if (_event !== "rename") { + return; } + potential(null); + this.inventoryReconcilePending = true; // Event kind is unreliable across editors/platforms. The marker // fingerprint decides whether inventory really changed. this.scheduleInventoryCheck(); @@ -408,42 +588,60 @@ class WorkspaceSystemGraphWatcher { private fallBackToPolling(): void { if (this.closed || this.pollTimer) return; - let refreshAfterInitialSnapshot = this.watcher !== null; + this.polling = true; this.watcher?.close(); this.watcher = null; const poll = (): void => { - if (this.closed || this.pollInFlight) return; + if (this.closed || this.pollInFlight || !this.initialized) return; this.pollInFlight = true; let sourceRoots: readonly string[] = []; + let sourceObservations: readonly WorkflowSourceObservation[] = []; try { sourceRoots = this.callbacks.listSourceRoots(this.scope); + sourceObservations = + this.callbacks.listSourceObservations?.(this.scope) ?? []; } catch { // Registry reads are hints too. Inventory polling still proceeds. } void Promise.all([ - snapshotWorkflowSourceRootsAsync(sourceRoots), - snapshotWorkspaceWorkflowsAsync(this.scope.root), + this.snapshotSources(sourceRoots, sourceObservations), + this.snapshotWorkspace(this.scope.root), ]) .then(([sourceSnapshots, inventorySnapshot]) => { if (this.closed) return; const changedRoots = this.lastSourceSnapshots ? changedSourceRoots(this.lastSourceSnapshots, sourceSnapshots) : []; - const ambiguousInitialChange = - this.lastSourceSnapshots === null && refreshAfterInitialSnapshot; - const sourceChanged = - ambiguousInitialChange || changedRoots.length > 0; + const inventoryChanged = + this.lastInventorySnapshot !== null && + inventorySnapshot !== this.lastInventorySnapshot; this.lastSourceSnapshots = sourceSnapshots; - refreshAfterInitialSnapshot = false; - if (ambiguousInitialChange) this.scheduleSourceChange(null); - else { + let sourceChanged = false; + if (inventoryChanged) { + // Structural evidence wins when both bounded channels move. A + // newly-created `candidate/.git` also changes that candidate's + // directory/source fingerprint; direct-scanning it would turn the + // foreign repository into an accidental explicit selection. + try { + this.callbacks.onPotentialChange?.(this.scope, null); + } catch { + // Reconciliation still follows below. + } + this.dispatchInventoryChange(inventorySnapshot); + } else if (changedRoots.length > 0) { + sourceChanged = true; + try { + this.callbacks.onPotentialChange?.(this.scope, changedRoots); + } catch { + // The queued callbacks still reconcile the observed delta. + } for (const sourceRoot of changedRoots) { this.scheduleSourceChange(sourceRoot); } } - if (inventorySnapshot !== this.lastInventorySnapshot) { - this.dispatchInventoryChange(inventorySnapshot); + if (this.lastInventorySnapshot === null) { + this.lastInventorySnapshot = inventorySnapshot; } else if ( sourceChanged && inventorySnapshot === this.failedInventorySnapshot @@ -460,7 +658,11 @@ class WorkspaceSystemGraphWatcher { this.pollInFlight = false; }); }; - poll(); + // Constructor-time fallback is intentionally quiet: initialize() owns the + // first source + inventory baseline and its conservative reconciliation. + // A runtime native-watch failure occurs after initialization and therefore + // performs an immediate recovery poll. + if (this.initialized) poll(); this.pollTimer = setInterval( poll, this.options.pollIntervalMs ?? POLL_INTERVAL_MS, @@ -470,6 +672,7 @@ class WorkspaceSystemGraphWatcher { close(): void { this.closed = true; if (this.sourceTimer) clearTimeout(this.sourceTimer); + if (this.sourceRetryTimer) clearTimeout(this.sourceRetryTimer); if (this.inventoryTimer) clearTimeout(this.inventoryTimer); if (this.inventoryRetryTimer) clearTimeout(this.inventoryRetryTimer); if (this.pollTimer) clearInterval(this.pollTimer); @@ -478,6 +681,166 @@ class WorkspaceSystemGraphWatcher { } } +interface SharedWorkspaceWatchLease { + canonicalRoot: string; + subscribers: Map; + watcher: WorkspaceSystemGraphWatcher; + ready: Promise; +} + +/** + * Process-wide watcher/fingerprint lease keyed by canonical workspace root. + * Session and graph consumers share one native handle, one polling baseline, + * and one bounded metadata traversal; callbacks fan out only after that shared + * observation has been accepted. + */ +export class SharedWorkspaceWatchBroker implements SharedWorkspaceWatchBrokerLike { + private readonly leases = new Map(); + private readonly rootBySubscriber = new Map(); + + constructor(private readonly options: SystemGraphWatcherOptions = {}) {} + + subscribe( + key: object, + subscriber: SharedWorkspaceWatchSubscriber, + ): Promise { + const canonicalRoot = canonicalGraphPath(subscriber.scope.root); + const previousRoot = this.rootBySubscriber.get(key); + if (previousRoot && previousRoot !== canonicalRoot) this.unsubscribe(key); + + const existing = this.leases.get(canonicalRoot); + if (existing) { + existing.subscribers.set(key, subscriber); + this.rootBySubscriber.set(key, canonicalRoot); + return existing.ready; + } + + const subscribers = new Map([[key, subscriber]]); + const brokerScope: WorkspaceScope = { + workspaceKey: `watch:${canonicalRoot}`, + root: canonicalRoot, + }; + const currentSubscribers = (): SharedWorkspaceWatchSubscriber[] => [ + ...subscribers.values(), + ]; + const started = WorkspaceSystemGraphWatcher.begin( + brokerScope, + { + listSourceRoots: () => { + const roots = new Set(); + for (const current of currentSubscribers()) { + try { + for (const root of current.listSourceRoots()) roots.add(root); + } catch { + // Another subscriber can still supply a useful baseline. + } + } + return [...roots].sort(); + }, + listSourceObservations: () => { + const observations: WorkflowSourceObservation[] = []; + const seen = new Set(); + for (const current of currentSubscribers()) { + let listed: readonly WorkflowSourceObservation[] = []; + try { + listed = current.listSourceObservations?.() ?? []; + } catch { + continue; + } + for (const observation of listed) { + const fingerprint = JSON.stringify([ + observation.workspaceRoot, + observation.candidateRoot, + [...observation.paths].sort(), + ]); + if (seen.has(fingerprint)) continue; + seen.add(fingerprint); + observations.push(observation); + } + } + return observations; + }, + onPotentialChange: (_scope, paths) => { + for (const current of currentSubscribers()) { + try { + current.onPotentialChange?.(paths); + } catch { + // One fail-close consumer must not suppress the others. + } + } + }, + onSourceChange: async (_scope, paths) => { + const results = await Promise.allSettled( + currentSubscribers().map((current) => + Promise.resolve().then(() => current.onSourceChange(paths)), + ), + ); + const failed = results.find( + (result): result is PromiseRejectedResult => + result.status === "rejected", + ); + if (failed) throw failed.reason; + }, + onInventoryChange: async () => { + const results = await Promise.allSettled( + currentSubscribers().map((current) => + Promise.resolve().then(() => current.onInventoryChange()), + ), + ); + const failed = results.find( + (result): result is PromiseRejectedResult => + result.status === "rejected", + ); + if (failed) throw failed.reason; + }, + }, + this.options, + ); + const lease: SharedWorkspaceWatchLease = { + canonicalRoot, + subscribers, + watcher: started.watcher, + ready: Promise.resolve(), + }; + lease.ready = started.ready.catch((error: unknown) => { + started.watcher.close(); + if (this.leases.get(canonicalRoot) === lease) { + this.leases.delete(canonicalRoot); + for (const subscriberKey of subscribers.keys()) { + if (this.rootBySubscriber.get(subscriberKey) === canonicalRoot) { + this.rootBySubscriber.delete(subscriberKey); + } + } + } + throw error; + }); + this.leases.set(canonicalRoot, lease); + this.rootBySubscriber.set(key, canonicalRoot); + return lease.ready; + } + + unsubscribe(key: object): void { + const canonicalRoot = this.rootBySubscriber.get(key); + if (!canonicalRoot) return; + this.rootBySubscriber.delete(key); + const lease = this.leases.get(canonicalRoot); + if (!lease) return; + lease.subscribers.delete(key); + if (lease.subscribers.size > 0) return; + lease.watcher.close(); + this.leases.delete(canonicalRoot); + try { + this.options.onLastLeaseReleased?.(canonicalRoot); + } catch { + // Lease retirement must always close the underlying OS resource. + } + } + + get size(): number { + return this.leases.size; + } +} + /** One watcher per requested workspace, independent of harness sessions. */ export class SystemGraphWatcherManager { private readonly watchers = new Map< @@ -486,7 +849,16 @@ export class SystemGraphWatcherManager { >(); private readonly pendingStarts = new Map< WorkspaceKey, - { root: string; token: object; promise: Promise } + { + root: string; + token: object; + watcher: WorkspaceSystemGraphWatcher; + promise: Promise; + } + >(); + private readonly sharedSubscriptions = new Map< + WorkspaceKey, + { root: string; key: object } >(); constructor( @@ -495,25 +867,62 @@ export class SystemGraphWatcherManager { ) {} start(scope: WorkspaceScope): Promise { + const sharedBroker = this.options.sharedBroker; + if (sharedBroker) { + const canonicalRoot = canonicalGraphPath(scope.root); + const existingShared = this.sharedSubscriptions.get(scope.workspaceKey); + if (existingShared?.root === canonicalRoot) return Promise.resolve(); + this.stop(scope.workspaceKey); + const key = {}; + this.sharedSubscriptions.set(scope.workspaceKey, { + root: canonicalRoot, + key, + }); + return sharedBroker + .subscribe(key, { + scope, + listSourceRoots: () => this.callbacks.listSourceRoots(scope), + listSourceObservations: () => + this.callbacks.listSourceObservations?.(scope) ?? [], + onPotentialChange: (paths) => + this.callbacks.onPotentialChange?.(scope, paths), + onSourceChange: (paths) => + this.callbacks.onSourceChange(scope, paths), + onInventoryChange: () => this.callbacks.onInventoryChange(scope), + }) + .catch((error: unknown) => { + if (this.sharedSubscriptions.get(scope.workspaceKey)?.key === key) { + this.sharedSubscriptions.delete(scope.workspaceKey); + } + throw error; + }); + } const existing = this.watchers.get(scope.workspaceKey); if (existing?.scope.root === scope.root) return Promise.resolve(); const pending = this.pendingStarts.get(scope.workspaceKey); if (pending?.root === scope.root) return pending.promise; this.stop(scope.workspaceKey); const token = {}; - const promise = WorkspaceSystemGraphWatcher.create( + const started = WorkspaceSystemGraphWatcher.begin( scope, this.callbacks, this.options, - ) - .then((watcher) => { - const current = this.pendingStarts.get(scope.workspaceKey); - if (current?.token !== token) { - watcher.close(); - return; - } - this.watchers.set(scope.workspaceKey, watcher); - }) + ); + const promise = started.ready + .then( + () => { + const current = this.pendingStarts.get(scope.workspaceKey); + if (current?.token !== token) { + started.watcher.close(); + return; + } + this.watchers.set(scope.workspaceKey, started.watcher); + }, + (error: unknown) => { + started.watcher.close(); + throw error; + }, + ) .finally(() => { if (this.pendingStarts.get(scope.workspaceKey)?.token === token) { this.pendingStarts.delete(scope.workspaceKey); @@ -522,6 +931,7 @@ export class SystemGraphWatcherManager { this.pendingStarts.set(scope.workspaceKey, { root: scope.root, token, + watcher: started.watcher, promise, }); return promise; @@ -531,6 +941,7 @@ export class SystemGraphWatcherManager { const tracked = new Set([ ...this.watchers.keys(), ...this.pendingStarts.keys(), + ...this.sharedSubscriptions.keys(), ]); for (const workspaceKey of tracked) { if (!workspaceKeys.has(workspaceKey)) this.stop(workspaceKey); @@ -538,6 +949,10 @@ export class SystemGraphWatcherManager { } stop(workspaceKey: WorkspaceKey): void { + const shared = this.sharedSubscriptions.get(workspaceKey); + if (shared) this.options.sharedBroker?.unsubscribe(shared.key); + this.sharedSubscriptions.delete(workspaceKey); + this.pendingStarts.get(workspaceKey)?.watcher.close(); this.pendingStarts.delete(workspaceKey); this.watchers.get(workspaceKey)?.close(); this.watchers.delete(workspaceKey); @@ -547,6 +962,7 @@ export class SystemGraphWatcherManager { const tracked = new Set([ ...this.watchers.keys(), ...this.pendingStarts.keys(), + ...this.sharedSubscriptions.keys(), ]); for (const workspaceKey of tracked) { this.stop(workspaceKey); @@ -554,6 +970,6 @@ export class SystemGraphWatcherManager { } get size(): number { - return this.watchers.size; + return this.watchers.size + this.sharedSubscriptions.size; } } diff --git a/packages/harness/src/core/system-graph.test.ts b/packages/harness/src/core/system-graph.test.ts index 75e5bc0fe..b220f616f 100644 --- a/packages/harness/src/core/system-graph.test.ts +++ b/packages/harness/src/core/system-graph.test.ts @@ -6,12 +6,13 @@ import type { PackageInventoryAgent } from "@sapiom/agent"; import type { WorkflowInfo } from "../shared/types.js"; import { HarnessRegistryInventoryProvider, + CachedAgentInvocationProvider, LocalWorkspaceScopeCatalog, StaticSystemGraphBuilder, type AgentInventoryProvider, type AgentInventoryResult, - type AgentRelationshipProvider, - type AgentRelationshipProviderResult, + type AgentInvocationProvider, + type AgentInvocationProviderResult, type WorkspaceScope, } from "./system-graph.js"; @@ -42,18 +43,18 @@ async function buildGraph( return (await builder.build(scope)).graph; } -function relationshipProvider( - listRelationships: ( +function invocationProvider( + listInvocations: ( sourceRoot: string, - ) => Promise, -): AgentRelationshipProvider { + ) => Promise, +): AgentInvocationProvider { return { - listRelationships: vi.fn((caller) => listRelationships(caller.sourceRoot)), + listInvocations: vi.fn((caller) => listInvocations(caller.sourceRoot)), }; } -const EMPTY_RELATIONSHIPS: AgentRelationshipProviderResult = { - relationships: [], +const EMPTY_INVOCATIONS: AgentInvocationProviderResult = { + invocations: [], warnings: [], }; @@ -82,6 +83,7 @@ function inventoryResult( warnings?: AgentInventoryResult["warnings"]; degraded?: boolean; identitySettled?: boolean; + discoveryComplete?: boolean; } = {}, ): AgentInventoryResult { const records = agents.map((agent) => { @@ -160,6 +162,7 @@ function inventoryResult( record.public.identityStatus === "provisional" && record.public.identityIssue === "identity-pending", ), + discoveryComplete: options.discoveryComplete ?? true, }; } @@ -228,10 +231,22 @@ describe("StaticSystemGraphBuilder", () => { ), }; - const graph = await buildGraph( - new StaticSystemGraphBuilder(inventory), - scope, - ); + const builder = new StaticSystemGraphBuilder(inventory); + const first = await builder.build(scope); + + // The cold phase is inventory-only: all cards and navigation are available + // before bounded project invocation I/O starts. + expect(first.cacheable).toBe(false); + expect(first.graph.nodes).toHaveLength(2); + expect(first.graph.edges).toEqual([]); + expect(first.navigation).toHaveLength(2); + first.afterCommit?.(); + + let graph = first.graph; + await vi.waitFor(async () => { + graph = (await builder.build(scope)).graph; + expect(graph.edges).toHaveLength(2); + }); expect(graph).toEqual({ kind: "system", @@ -245,14 +260,14 @@ describe("StaticSystemGraphBuilder", () => { from: "agent:research", to: "agent:growth", kind: "invokes", - basis: "static", + basis: "static-invocation", mode: "blocking", }, { from: "agent:research", to: "agent:growth", kind: "invokes", - basis: "static", + basis: "static-invocation", mode: "async", }, ], @@ -261,6 +276,81 @@ describe("StaticSystemGraphBuilder", () => { expect(JSON.stringify(graph)).not.toContain(FIXTURE); }); + it("returns inventory navigation while invocation extraction is still held", async () => { + const inventory: AgentInventoryProvider = { + listAgents: vi.fn(async () => + inventoryResult(scope, [ + { + agentKey: "growth", + label: "Growth", + resolutionAliases: ["growth"], + }, + { + agentKey: "research", + label: "Research", + resolutionAliases: ["research"], + }, + ]), + ), + }; + let release!: (result: AgentInvocationProviderResult) => void; + const held = new Promise((resolve) => { + release = resolve; + }); + const inner = invocationProvider(async (root) => + root.endsWith("research") ? held : EMPTY_INVOCATIONS, + ); + const onChange = vi.fn(); + const invocations = new CachedAgentInvocationProvider( + inner, + async () => "unused", + { concurrency: 1, onChange }, + ); + const builder = new StaticSystemGraphBuilder(inventory, invocations); + + const cold = await builder.build(scope); + + expect(inner.listInvocations).not.toHaveBeenCalled(); + expect(cold.cacheable).toBe(false); + expect(cold.graph.nodes.map((node) => node.agentKey)).toEqual([ + "growth", + "research", + ]); + expect(cold.graph.edges).toEqual([]); + expect(cold.navigation?.map((target) => target.agentKey)).toEqual([ + "growth", + "research", + ]); + + cold.afterCommit?.(); + await vi.waitFor(() => expect(inner.listInvocations).toHaveBeenCalled()); + // The held project task cannot withhold the already-returned inventory. + expect(cold.graph.nodes).toHaveLength(2); + release({ + invocations: [ + { + target: "growth", + mode: "blocking", + evidence: EVIDENCE, + }, + ], + warnings: [], + }); + + await vi.waitFor(() => expect(onChange).toHaveBeenCalledTimes(1)); + const enriched = await builder.build(scope); + expect(enriched.graph.edges).toEqual([ + { + from: "agent:research", + to: "agent:growth", + kind: "invokes", + basis: "static-invocation", + mode: "blocking", + }, + ]); + expect(enriched.cacheable).toBe(true); + }); + it("deduplicates by mode, retains dual-mode edges, and reports duplicate and unresolved targets", async () => { const inventory: AgentInventoryProvider = { listAgents: vi.fn(async () => @@ -282,10 +372,10 @@ describe("StaticSystemGraphBuilder", () => { ]), ), }; - const relationships = relationshipProvider(async (root) => + const invocations = invocationProvider(async (root) => root.endsWith("research") ? { - relationships: [ + invocations: [ { target: "growth", mode: "blocking", @@ -300,11 +390,11 @@ describe("StaticSystemGraphBuilder", () => { ], warnings: [], } - : EMPTY_RELATIONSHIPS, + : EMPTY_INVOCATIONS, ); const graph = await buildGraph( - new StaticSystemGraphBuilder(inventory, relationships), + new StaticSystemGraphBuilder(inventory, invocations), scope, ); expect(graph.edges).toEqual([ @@ -312,14 +402,14 @@ describe("StaticSystemGraphBuilder", () => { from: "agent:research", to: "agent:growth", kind: "invokes", - basis: "static", + basis: "static-invocation", mode: "blocking", }, { from: "agent:research", to: "agent:growth", kind: "invokes", - basis: "static", + basis: "static-invocation", mode: "async", }, ]); @@ -349,8 +439,8 @@ describe("StaticSystemGraphBuilder", () => { ]), ), }; - const relationships = relationshipProvider(async () => ({ - relationships: [], + const invocations = invocationProvider(async () => ({ + invocations: [], warnings: [ { code: "dynamic-target", @@ -362,7 +452,7 @@ describe("StaticSystemGraphBuilder", () => { const built = await new StaticSystemGraphBuilder( inventory, - relationships, + invocations, ).build(scope); expect(built.cacheable).toBe(true); @@ -421,19 +511,19 @@ describe("StaticSystemGraphBuilder", () => { ), ), }; - const relationships = relationshipProvider(async (root) => + const invocations = invocationProvider(async (root) => root.endsWith("caller") ? { - relationships: [ + invocations: [ { target: "shared", mode: "async", evidence: EVIDENCE }, ], warnings: [], } - : EMPTY_RELATIONSHIPS, + : EMPTY_INVOCATIONS, ); const graph = await buildGraph( - new StaticSystemGraphBuilder(inventory, relationships), + new StaticSystemGraphBuilder(inventory, invocations), scope, ); @@ -478,7 +568,7 @@ describe("StaticSystemGraphBuilder", () => { const graph = await buildGraph( new StaticSystemGraphBuilder( { listAgents: async () => result }, - relationshipProvider(async () => EMPTY_RELATIONSHIPS), + invocationProvider(async () => EMPTY_INVOCATIONS), ), scope, ); @@ -519,7 +609,7 @@ describe("StaticSystemGraphBuilder", () => { const graph = await buildGraph( new StaticSystemGraphBuilder( { listAgents: async () => result }, - relationshipProvider(async () => EMPTY_RELATIONSHIPS), + invocationProvider(async () => EMPTY_INVOCATIONS), ), scope, ); @@ -553,7 +643,7 @@ describe("StaticSystemGraphBuilder", () => { buildGraph( new StaticSystemGraphBuilder( { listAgents: async () => result }, - relationshipProvider(async () => EMPTY_RELATIONSHIPS), + invocationProvider(async () => EMPTY_INVOCATIONS), ), scope, ), @@ -588,15 +678,15 @@ describe("StaticSystemGraphBuilder", () => { const graph = await buildGraph( new StaticSystemGraphBuilder( { listAgents: async () => inventory }, - relationshipProvider(async (sourceRoot) => + invocationProvider(async (sourceRoot) => sourceRoot.endsWith("caller") ? { - relationships: [ + invocations: [ { target: "payments", mode: "async", evidence: EVIDENCE }, ], warnings: [], } - : EMPTY_RELATIONSHIPS, + : EMPTY_INVOCATIONS, ), ), scope, @@ -607,13 +697,13 @@ describe("StaticSystemGraphBuilder", () => { from: "agent:caller", to: "agent:payments", kind: "invokes", - basis: "static", + basis: "static-invocation", mode: "async", }, ]); }); - it("keeps a relationship ambiguous when only multiple aliases match", async () => { + it("keeps a invocation ambiguous when only multiple aliases match", async () => { const inventory = inventoryResult( scope, [ @@ -642,15 +732,15 @@ describe("StaticSystemGraphBuilder", () => { const graph = await buildGraph( new StaticSystemGraphBuilder( { listAgents: async () => inventory }, - relationshipProvider(async (sourceRoot) => + invocationProvider(async (sourceRoot) => sourceRoot.endsWith("caller") ? { - relationships: [ + invocations: [ { target: "legacy", mode: "async", evidence: EVIDENCE }, ], warnings: [], } - : EMPTY_RELATIONSHIPS, + : EMPTY_INVOCATIONS, ), ), scope, @@ -692,15 +782,15 @@ describe("StaticSystemGraphBuilder", () => { const graph = await buildGraph( new StaticSystemGraphBuilder( { listAgents: async () => inventory }, - relationshipProvider(async (sourceRoot) => + invocationProvider(async (sourceRoot) => sourceRoot.endsWith("caller") ? { - relationships: [ + invocations: [ { target: "legacy", mode: "async", evidence: EVIDENCE }, ], warnings: [], } - : EMPTY_RELATIONSHIPS, + : EMPTY_INVOCATIONS, ), ), scope, @@ -734,7 +824,7 @@ describe("StaticSystemGraphBuilder", () => { buildGraph( new StaticSystemGraphBuilder( inventory, - relationshipProvider(async () => EMPTY_RELATIONSHIPS), + invocationProvider(async () => EMPTY_INVOCATIONS), ), scope, ), @@ -752,7 +842,7 @@ describe("StaticSystemGraphBuilder", () => { buildGraph( new StaticSystemGraphBuilder( { listAgents: async () => outside }, - relationshipProvider(async () => EMPTY_RELATIONSHIPS), + invocationProvider(async () => EMPTY_INVOCATIONS), ), scope, ), @@ -771,7 +861,7 @@ describe("StaticSystemGraphBuilder", () => { buildGraph( new StaticSystemGraphBuilder( { listAgents: async () => mismatchedLocation }, - relationshipProvider(async () => EMPTY_RELATIONSHIPS), + invocationProvider(async () => EMPTY_INVOCATIONS), ), scope, ), @@ -794,7 +884,7 @@ describe("StaticSystemGraphBuilder", () => { }; const built = await new StaticSystemGraphBuilder( inventory, - relationshipProvider(async () => { + invocationProvider(async () => { throw new Error("boom at private source"); }), ).build(scope); @@ -828,7 +918,7 @@ describe("StaticSystemGraphBuilder", () => { const graph = await buildGraph( new StaticSystemGraphBuilder( inventory, - relationshipProvider(async () => EMPTY_RELATIONSHIPS), + invocationProvider(async () => EMPTY_INVOCATIONS), ), scope, ); @@ -860,10 +950,10 @@ describe("StaticSystemGraphBuilder", () => { }); const builder = new StaticSystemGraphBuilder( inventory, - relationshipProvider(async (sourceRoot) => + invocationProvider(async (sourceRoot) => sourceRoot.endsWith("caller") ? { - relationships: [ + invocations: [ { target: "target-marker", mode: "async", @@ -872,7 +962,7 @@ describe("StaticSystemGraphBuilder", () => { ], warnings: [], } - : EMPTY_RELATIONSHIPS, + : EMPTY_INVOCATIONS, ), ); const initial = await builder.build(scope); @@ -890,7 +980,7 @@ describe("StaticSystemGraphBuilder", () => { from: "agent:caller-marker", to: "agent:target-marker", kind: "invokes", - basis: "static", + basis: "static-invocation", mode: "async", }, ]); @@ -928,7 +1018,7 @@ describe("StaticSystemGraphBuilder", () => { const graph = await buildGraph( new StaticSystemGraphBuilder( { listAgents: async () => result }, - relationshipProvider(async () => EMPTY_RELATIONSHIPS), + invocationProvider(async () => EMPTY_INVOCATIONS), ), scope, ); @@ -964,7 +1054,7 @@ describe("StaticSystemGraphBuilder", () => { const graph = await buildGraph( new StaticSystemGraphBuilder( { listAgents: async () => result }, - relationshipProvider(async () => EMPTY_RELATIONSHIPS), + invocationProvider(async () => EMPTY_INVOCATIONS), ), scope, ); @@ -1005,7 +1095,7 @@ describe("StaticSystemGraphBuilder", () => { buildGraph( new StaticSystemGraphBuilder( { listAgents: async () => result }, - relationshipProvider(async () => EMPTY_RELATIONSHIPS), + invocationProvider(async () => EMPTY_INVOCATIONS), ), scope, ), @@ -1065,7 +1155,7 @@ describe("StaticSystemGraphBuilder", () => { buildGraph( new StaticSystemGraphBuilder( { listAgents: async () => result }, - relationshipProvider(async () => EMPTY_RELATIONSHIPS), + invocationProvider(async () => EMPTY_INVOCATIONS), ), scope, ), @@ -1091,14 +1181,14 @@ describe("StaticSystemGraphBuilder", () => { buildGraph( new StaticSystemGraphBuilder( { listAgents: async () => result }, - relationshipProvider(async () => EMPTY_RELATIONSHIPS), + invocationProvider(async () => EMPTY_INVOCATIONS), ), scope, ), ).rejects.toThrow("inventory context was invalid"); }); - it("normalizes arbitrary provider aliases before relationship resolution", async () => { + it("normalizes arbitrary provider aliases before invocation resolution", async () => { const result = inventoryResult(scope, [ { agentKey: "reporting", @@ -1107,19 +1197,19 @@ describe("StaticSystemGraphBuilder", () => { resolutionAliases: ["zeta", "alpha", "zeta"], }, ]); - const listRelationships = vi.fn< - AgentRelationshipProvider["listRelationships"] - >(async () => EMPTY_RELATIONSHIPS); + const listInvocations = vi.fn( + async () => EMPTY_INVOCATIONS, + ); await buildGraph( new StaticSystemGraphBuilder( { listAgents: async () => result }, - { listRelationships }, + { listInvocations }, ), scope, ); - expect(listRelationships.mock.calls[0]?.[0].resolutionAliases).toEqual([ + expect(listInvocations.mock.calls[0]?.[0].resolutionAliases).toEqual([ "alpha", "zeta", ]); @@ -1162,7 +1252,7 @@ describe("StaticSystemGraphBuilder", () => { const graph = await buildGraph( new StaticSystemGraphBuilder( inventory, - relationshipProvider(async () => EMPTY_RELATIONSHIPS), + invocationProvider(async () => EMPTY_INVOCATIONS), ), scope, ); @@ -1203,7 +1293,7 @@ describe("StaticSystemGraphBuilder", () => { const built = await new StaticSystemGraphBuilder( inventory, - relationshipProvider(async () => EMPTY_RELATIONSHIPS), + invocationProvider(async () => EMPTY_INVOCATIONS), ).build(scope); expect(built.cacheable).toBe(false); @@ -1249,7 +1339,7 @@ describe("StaticSystemGraphBuilder", () => { const built = await new StaticSystemGraphBuilder( { listAgents: async () => result }, - relationshipProvider(async () => EMPTY_RELATIONSHIPS), + invocationProvider(async () => EMPTY_INVOCATIONS), ).build(scope); expect(built.cacheable).toBe(true); @@ -1263,9 +1353,33 @@ describe("StaticSystemGraphBuilder", () => { ]); }); - it("uses the normalized inventory status even if a provider mutates its result", async () => { + it("does not cache a settled identity when workspace discovery was incomplete", async () => { + const result = inventoryResult( + scope, + [ + { + agentKey: "dashboard", + label: "Dashboard", + }, + ], + { + degraded: true, + identitySettled: true, + discoveryComplete: false, + }, + ); + + const built = await new StaticSystemGraphBuilder( + { listAgents: async () => result }, + invocationProvider(async () => EMPTY_INVOCATIONS), + ).build(scope); + + expect(built.cacheable).toBe(false); + }); + + it("uses normalized identity state even if a provider mutates inventory status", async () => { let release!: () => void; - const relationshipsPending = new Promise((resolve) => { + const invocationsPending = new Promise((resolve) => { release = resolve; }); const result = inventoryResult( @@ -1281,15 +1395,15 @@ describe("StaticSystemGraphBuilder", () => { ], { degraded: true }, ); - const listRelationships = vi.fn(async () => { - await relationshipsPending; - return EMPTY_RELATIONSHIPS; + const listInvocations = vi.fn(async () => { + await invocationsPending; + return EMPTY_INVOCATIONS; }); const building = new StaticSystemGraphBuilder( { listAgents: async () => result }, - { listRelationships }, + { listInvocations }, ).build(scope); - await vi.waitFor(() => expect(listRelationships).toHaveBeenCalledTimes(1)); + await vi.waitFor(() => expect(listInvocations).toHaveBeenCalledTimes(1)); (result.inventory as { status: "complete" | "degraded" }).status = "complete"; @@ -1322,11 +1436,11 @@ describe("StaticSystemGraphBuilder", () => { retainSources, }; const retainCallers = vi.fn(); - const relationships: AgentRelationshipProvider = { - listRelationships: vi.fn(async () => EMPTY_RELATIONSHIPS), + const invocations: AgentInvocationProvider = { + listInvocations: vi.fn(async () => EMPTY_INVOCATIONS), retainCallers, }; - const builder = new StaticSystemGraphBuilder(inventory, relationships); + const builder = new StaticSystemGraphBuilder(inventory, invocations); await builder.build(scope); await builder.build(secondScope); diff --git a/packages/harness/src/core/system-graph.ts b/packages/harness/src/core/system-graph.ts index ba0c04032..f40b4cece 100644 --- a/packages/harness/src/core/system-graph.ts +++ b/packages/harness/src/core/system-graph.ts @@ -4,8 +4,8 @@ import { packageInventorySchema } from "@sapiom/agent"; import type { GraphWarning, + StaticInvocationGraphEdge, SystemGraph, - SystemGraphEdge, SystemGraphNavigationTarget, WorkspaceKey, WorkspaceScopeSummary, @@ -20,10 +20,10 @@ import { type WorkspaceScope, } from "./system-graph-inventory.js"; import { - CachedAgentRelationshipProvider, - SourceAgentRelationshipProvider, - type AgentRelationshipProvider, - type AgentRelationshipProviderResult, + CachedAgentInvocationProvider, + SourceAgentInvocationProvider, + type AgentInvocationProvider, + type AgentInvocationProviderResult, } from "./system-graph-relationships.js"; export { HarnessRegistryInventoryProvider } from "./system-graph-inventory.js"; @@ -35,14 +35,14 @@ export type { WorkspaceScope, } from "./system-graph-inventory.js"; export { - CachedAgentRelationshipProvider, - SourceAgentRelationshipProvider, + CachedAgentInvocationProvider, + SourceAgentInvocationProvider, } from "./system-graph-relationships.js"; export type { - AgentRelationshipCandidate, - AgentRelationshipProvider, - AgentRelationshipProviderResult, - AgentRelationshipWarning, + AgentInvocationCandidate, + AgentInvocationProvider, + AgentInvocationProviderResult, + AgentInvocationWarning, } from "./system-graph-relationships.js"; export interface WorkspaceScopeResolver { @@ -242,6 +242,8 @@ interface ConsumedInventory { warnings: GraphWarning[]; /** Every identity has finished resolving, however it resolved. */ identitySettled: boolean; + /** The accepted workspace walk considered every eligible discovery path. */ + discoveryComplete: boolean; startEnrichment?: () => void; } @@ -311,6 +313,7 @@ function consumeInventory( agent.identityStatus === "provisional" && agent.identityIssue === "identity-pending", ), + discoveryComplete: result.discoveryComplete === true, ...(typeof result.startEnrichment === "function" ? { startEnrichment: result.startEnrichment } : {}), @@ -325,8 +328,8 @@ export class StaticSystemGraphBuilder implements SystemGraphBuilder { constructor( private readonly inventory: AgentInventoryProvider, - private readonly relationships: AgentRelationshipProvider = new CachedAgentRelationshipProvider( - new SourceAgentRelationshipProvider(), + private readonly invocations: AgentInvocationProvider = new CachedAgentInvocationProvider( + new SourceAgentInvocationProvider(), ), ) {} @@ -335,7 +338,7 @@ export class StaticSystemGraphBuilder implements SystemGraphBuilder { const consumed = consumeInventory(scope, inventory); const agents = consumed.agents; this.callersByWorkspace.set(scope.workspaceKey, agents); - this.retainRelationshipCallers(); + this.retainInvocationCallers(); const nodes = agents.map((agent) => ({ id: `agent:${agent.agentKey}`, agentKey: agent.agentKey, @@ -366,37 +369,63 @@ export class StaticSystemGraphBuilder implements SystemGraphBuilder { } } - const edges: SystemGraphEdge[] = []; + const edges: StaticInvocationGraphEdge[] = []; const warnings: GraphWarning[] = [...consumed.warnings]; const seenEdges = new Set(); - let relationshipsComplete = true; - - // Source walks are independent. Run them together so first-open latency is - // bounded by the slowest agent tree rather than the sum of every tree. - const scans = await Promise.all( - agents.map(async (caller) => { - try { - return { - caller, - result: await this.relationships.listRelationships(caller), - failed: false as const, - }; - } catch { + let invocationsComplete = true; + + const supportsBackgroundInvocations = + typeof this.invocations.peekInvocations === "function" && + typeof this.invocations.startInvocations === "function"; + // Production is deliberately two-phase: project cache-only inventory now, + // then perform bounded invocation I/O after nodes/navigation are visible. + // Legacy/test providers without the cache surface retain the old awaited + // adapter behavior. + const scans = supportsBackgroundInvocations + ? agents.map((caller) => { + const snapshot = this.invocations.peekInvocations!(caller); return { caller, - result: { - relationships: [], - warnings: [], - } satisfies AgentRelationshipProviderResult, - failed: true as const, + result: + snapshot?.result ?? + ({ + invocations: [], + warnings: [], + } satisfies AgentInvocationProviderResult), + failed: snapshot?.status === "failed", + pending: snapshot === undefined, }; - } - }), - ); - - for (const { caller, result, failed } of scans) { + }) + : await Promise.all( + agents.map(async (caller) => { + try { + return { + caller, + result: await this.invocations.listInvocations(caller), + failed: false, + pending: false, + }; + } catch { + return { + caller, + result: { + invocations: [], + warnings: [], + } satisfies AgentInvocationProviderResult, + failed: true, + pending: false, + }; + } + }), + ); + + for (const { caller, result, failed, pending } of scans) { + if (pending) { + invocationsComplete = false; + continue; + } if (failed) { - relationshipsComplete = false; + invocationsComplete = false; warnings.push({ code: "projection-failed", agentKey: caller.agentKey, @@ -404,6 +433,14 @@ export class StaticSystemGraphBuilder implements SystemGraphBuilder { }); continue; } + if (result.complete === false) { + invocationsComplete = false; + warnings.push({ + code: "projection-failed", + agentKey: caller.agentKey, + message: `Could not fully inspect ${caller.label}.`, + }); + } for (const warning of result.warnings) { if (warning.code === "dynamic-target") { @@ -415,14 +452,14 @@ export class StaticSystemGraphBuilder implements SystemGraphBuilder { } } - for (const relationship of result.relationships) { - const exact = canonicalTargets.get(relationship.target); + for (const invocation of result.invocations) { + const exact = canonicalTargets.get(invocation.target); const candidates = exact ? [exact] - : (candidateTargets.get(relationship.target) ?? []); + : (candidateTargets.get(invocation.target) ?? []); if (candidates.length !== 1) { - const target = /^[A-Za-z0-9@_.:-]+$/.test(relationship.target) - ? relationship.target + const target = /^[A-Za-z0-9@_.:-]+$/.test(invocation.target) + ? invocation.target : null; warnings.push({ code: "unresolved-target", @@ -440,8 +477,8 @@ export class StaticSystemGraphBuilder implements SystemGraphBuilder { if (target.agentKey === caller.agentKey) continue; const from = `agent:${caller.agentKey}`; const to = `agent:${target.agentKey}`; - const edgeKey = `${from}\0${to}\0${relationship.mode}`; - if (relationship.evidence.length > 1 || seenEdges.has(edgeKey)) { + const edgeKey = `${from}\0${to}\0${invocation.mode}`; + if (invocation.evidence.length > 1 || seenEdges.has(edgeKey)) { warnings.push({ code: "duplicate-edge", agentKey: caller.agentKey, @@ -454,8 +491,8 @@ export class StaticSystemGraphBuilder implements SystemGraphBuilder { from, to, kind: "invokes", - basis: "static", - mode: relationship.mode, + basis: "static-invocation", + mode: invocation.mode, }); } } @@ -476,8 +513,20 @@ export class StaticSystemGraphBuilder implements SystemGraphBuilder { ).values(), ].sort(warningOrder); + const afterCommit = + consumed.startEnrichment || supportsBackgroundInvocations + ? () => { + consumed.startEnrichment?.(); + if (supportsBackgroundInvocations) { + this.invocations.startInvocations!(agents); + } + } + : undefined; return { - cacheable: consumed.identitySettled && relationshipsComplete, + cacheable: + consumed.identitySettled && + consumed.discoveryComplete && + invocationsComplete, graph: { kind: "system", scope: { kind: "working-tree", workspaceKey: scope.workspaceKey }, @@ -489,9 +538,7 @@ export class StaticSystemGraphBuilder implements SystemGraphBuilder { agentKey, workflowPath, })), - ...(consumed.startEnrichment - ? { afterCommit: consumed.startEnrichment } - : {}), + ...(afterCommit ? { afterCommit } : {}), }; } @@ -512,13 +559,13 @@ export class StaticSystemGraphBuilder implements SystemGraphBuilder { } catch { // Private cache pruning cannot make graph projection fail. } - this.retainRelationshipCallers(); + this.retainInvocationCallers(); } - private retainRelationshipCallers(): void { + private retainInvocationCallers(): void { const callers = [...this.callersByWorkspace.values()].flat(); try { - this.relationships.retainCallers?.(callers); + this.invocations.retainCallers?.(callers); } catch { // Cache pruning is an optimization and cannot make projection fail. } diff --git a/packages/harness/src/core/workflow-registry-router.test.ts b/packages/harness/src/core/workflow-registry-router.test.ts new file mode 100644 index 000000000..f5d2ac786 --- /dev/null +++ b/packages/harness/src/core/workflow-registry-router.test.ts @@ -0,0 +1,95 @@ +import * as fs from "node:fs/promises"; +import * as os from "node:os"; +import * as path from "node:path"; +import type { AddressInfo } from "node:net"; + +import express from "express"; +import { afterEach, beforeEach, describe, expect, it } from "vitest"; + +import { + createWorkflowsRouter, + WorkflowRegistry, +} from "./workflow-registry.js"; + +describe("createWorkflowsRouter public projection", () => { + let tmpRoot: string; + let server: ReturnType | undefined; + + beforeEach(async () => { + tmpRoot = await fs.mkdtemp( + path.join(os.tmpdir(), "workflow-router-public-"), + ); + }); + + afterEach(async () => { + if (server) { + await new Promise((resolve) => server!.close(() => resolve())); + server = undefined; + } + await fs.rm(tmpRoot, { recursive: true, force: true }); + }); + + it("never serializes registry-only source or marker evidence", async () => { + const markerRoot = path.join(tmpRoot, "marker-agent"); + const sourceRoot = path.join(tmpRoot, "source-agent"); + await fs.mkdir(markerRoot, { recursive: true }); + await fs.writeFile( + path.join(markerRoot, "sapiom.json"), + JSON.stringify({ definitionId: 7, name: "marker-cloud-name" }), + ); + await fs.mkdir(sourceRoot, { recursive: true }); + await fs.writeFile( + path.join(sourceRoot, "index.ts"), + 'import { defineAgent } from "@sapiom/agent";\n' + + 'export const agent = defineAgent({ name: "source-proof-name" });\n', + ); + + const registry = new WorkflowRegistry(path.join(tmpRoot, "workflows.json")); + const app = express(); + app.use(express.json()); + app.use(createWorkflowsRouter(registry)); + server = app.listen(0); + const { port } = server.address() as AddressInfo; + const baseUrl = `http://127.0.0.1:${port}`; + + const scan = await fetch(`${baseUrl}/api/workflows/scan`, { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ root: tmpRoot }), + }); + expect(scan.status).toBe(200); + const scanBody = (await scan.json()) as { + found: Array>; + }; + expect(scanBody.found).toHaveLength(2); + expect( + scanBody.found.every((row) => !("sourceDefinitionName" in row)), + ).toBe(true); + expect(scanBody.found.every((row) => !("markerPresent" in row))).toBe(true); + + const internalRows = await registry.list(); + expect(internalRows.some((row) => row.markerPresent === true)).toBe(true); + expect( + internalRows.some( + (row) => row.sourceDefinitionName === "source-proof-name", + ), + ).toBe(true); + + const list = await fetch(`${baseUrl}/api/workflows`); + const listBody = (await list.json()) as Array>; + expect(listBody.every((row) => !("sourceDefinitionName" in row))).toBe( + true, + ); + expect(listBody.every((row) => !("markerPresent" in row))).toBe(true); + + const connect = await fetch(`${baseUrl}/api/workflows/connect`, { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ path: markerRoot }), + }); + expect(connect.status).toBe(200); + const connectBody = (await connect.json()) as Record; + expect(connectBody).not.toHaveProperty("sourceDefinitionName"); + expect(connectBody).not.toHaveProperty("markerPresent"); + }); +}); diff --git a/packages/harness/src/core/workflow-registry.test.ts b/packages/harness/src/core/workflow-registry.test.ts index 797a0d925..baa1ae9f7 100644 --- a/packages/harness/src/core/workflow-registry.test.ts +++ b/packages/harness/src/core/workflow-registry.test.ts @@ -3,7 +3,11 @@ import * as os from "node:os"; import * as path from "node:path"; import { afterEach, beforeEach, describe, expect, it } from "vitest"; -import { AgentProjectScanBudget } from "./agent-project-discovery.js"; +import { + AgentProjectScanAllowance, + AgentProjectScanBudget, +} from "./agent-project-discovery.js"; +import { AgentSourceScanBudget } from "./agent-source-discovery.js"; import { WorkflowRegistry } from "./workflow-registry.js"; async function writeMarker( @@ -18,13 +22,40 @@ async function writeMarker( ); } +async function writeSourceAgent( + dir: string, + name: string, + extraSource = "", +): Promise { + await fs.mkdir(dir, { recursive: true }); + await fs.writeFile( + path.join(dir, "index.ts"), + `import { defineAgent } from "@sapiom/agent"; +${extraSource} +export const agent = defineAgent({ name: ${JSON.stringify(name)} });`, + ); +} + +function deferred(): { + promise: Promise; + resolve: () => void; +} { + let resolve!: () => void; + const promise = new Promise((settle) => { + resolve = settle; + }); + return { promise, resolve }; +} + describe("WorkflowRegistry", () => { let tmpRoot: string; let registryPath: string; let registry: WorkflowRegistry; beforeEach(async () => { - tmpRoot = await fs.mkdtemp(path.join(os.tmpdir(), "harness-workflow-registry-")); + tmpRoot = await fs.mkdtemp( + path.join(os.tmpdir(), "harness-workflow-registry-"), + ); registryPath = path.join(tmpRoot, "state", "workflows.json"); registry = new WorkflowRegistry(registryPath); }); @@ -37,6 +68,36 @@ describe("WorkflowRegistry", () => { expect(await registry.list()).toEqual([]); }); + it("shares one directory and source allowance across composite direct-root scans", async () => { + const firstRoot = path.join(tmpRoot, "first"); + const secondRoot = path.join(tmpRoot, "second"); + await writeSourceAgent(firstRoot, "first"); + await writeSourceAgent(secondRoot, "second"); + const project = new AgentProjectScanAllowance(2); + const source = new AgentSourceScanBudget({ + maxModules: 1, + maxBytes: 1024 * 1024, + maxLookups: 32, + }); + + const first = await registry.scanDetailed( + firstRoot, + new AgentProjectScanBudget({}, project), + source, + ); + const second = await registry.scanDetailed( + secondRoot, + new AgentProjectScanBudget({}, project), + source, + ); + + expect(project.visited).toBe(2); + expect(source.modules).toBe(1); + expect(first.sourceBudget).toBe(source); + expect(second.sourceBudget).toBe(source); + expect(second.status).toBe("degraded"); + }); + it("scans a tree for sapiom.json markers, honoring depth and skip rules", async () => { // Depth 1: has package.json, deployed. await writeMarker(path.join(tmpRoot, "proj-a"), 42); @@ -47,9 +108,15 @@ describe("WorkflowRegistry", () => { // Depth 1: no package.json, undeployed. await writeMarker(path.join(tmpRoot, "proj-b"), null); // Depth 8: right at the boundary — should be found. - await writeMarker(path.join(tmpRoot, "a", "b", "c", "d", "e", "f", "g", "h"), 7); + await writeMarker( + path.join(tmpRoot, "a", "b", "c", "d", "e", "f", "g", "h"), + 7, + ); // Depth 9: past the boundary — should NOT be found. - await writeMarker(path.join(tmpRoot, "d", "e", "f", "g", "h", "i", "j", "k", "l"), 9); + await writeMarker( + path.join(tmpRoot, "d", "e", "f", "g", "h", "i", "j", "k", "l"), + 9, + ); // Inside generated/private trees — should never be scanned. await writeMarker(path.join(tmpRoot, "node_modules", "some-pkg"), 1); await writeMarker(path.join(tmpRoot, ".git", "worktrees", "x"), 1); @@ -66,6 +133,7 @@ describe("WorkflowRegistry", () => { path: path.join(tmpRoot, "proj-a"), definitionId: 42, definitionSlug: null, + markerPresent: true, templateId: null, forkId: null, starterId: null, @@ -76,25 +144,38 @@ describe("WorkflowRegistry", () => { path: path.join(tmpRoot, "proj-b"), definitionId: null, definitionSlug: null, + markerPresent: true, templateId: null, forkId: null, starterId: null, source: "scan", }); - expect(byPath.has(path.join(tmpRoot, "a", "b", "c", "d", "e", "f", "g", "h"))).toBe( - true, - ); expect( - byPath.has(path.join(tmpRoot, "d", "e", "f", "g", "h", "i", "j", "k", "l")), + byPath.has(path.join(tmpRoot, "a", "b", "c", "d", "e", "f", "g", "h")), + ).toBe(true); + expect( + byPath.has( + path.join(tmpRoot, "d", "e", "f", "g", "h", "i", "j", "k", "l"), + ), ).toBe(false); expect( found.some((workflow) => workflow.path.includes("node_modules")), ).toBe(false); - expect(found.some((workflow) => workflow.path.includes(".git"))).toBe(false); - expect(found.some((workflow) => workflow.path.includes(".sapiom"))).toBe(false); - expect(found.some((workflow) => workflow.path.includes("dist"))).toBe(false); - expect(found.some((workflow) => workflow.path.includes("build"))).toBe(false); - expect(found.some((workflow) => workflow.path.includes(".next"))).toBe(false); + expect(found.some((workflow) => workflow.path.includes(".git"))).toBe( + false, + ); + expect(found.some((workflow) => workflow.path.includes(".sapiom"))).toBe( + false, + ); + expect(found.some((workflow) => workflow.path.includes("dist"))).toBe( + false, + ); + expect(found.some((workflow) => workflow.path.includes("build"))).toBe( + false, + ); + expect(found.some((workflow) => workflow.path.includes(".next"))).toBe( + false, + ); }); it("requires sapiom.json to contain a top-level JSON object", async () => { @@ -105,10 +186,15 @@ describe("WorkflowRegistry", () => { await fs.writeFile(path.join(dir, "sapiom.json"), value); } await writeMarker(path.join(tmpRoot, "valid-empty-object"), null); - await fs.writeFile(path.join(tmpRoot, "valid-empty-object", "sapiom.json"), "{}"); + await fs.writeFile( + path.join(tmpRoot, "valid-empty-object", "sapiom.json"), + "{}", + ); const found = await registry.scan(tmpRoot); - expect(found.map((workflow) => workflow.name)).toEqual(["valid-empty-object"]); + expect(found.map((workflow) => workflow.name)).toEqual([ + "valid-empty-object", + ]); }); it("persists scan results and reloads them for a fresh registry instance", async () => { @@ -147,6 +233,36 @@ describe("WorkflowRegistry", () => { expect(info.definitionId).toBe(99); }); + it("normalizes untrusted marker fields identically for scan, connect, and reload", async () => { + const projectDir = path.join(tmpRoot, "malformed-fields"); + await fs.mkdir(projectDir, { recursive: true }); + await fs.writeFile( + path.join(projectDir, "sapiom.json"), + JSON.stringify({ + definitionId: "99", + name: "bad/name", + templateId: { private: true }, + forkId: "bad\u0085value", + starterId: 42, + }), + ); + + const scanned = (await registry.scan(tmpRoot))[0]!; + const connected = await registry.connectPath(projectDir); + const reloaded = (await new WorkflowRegistry(registryPath).list())[0]!; + + expect(scanned).toMatchObject({ + definitionId: null, + definitionSlug: null, + templateId: null, + forkId: null, + starterId: null, + markerPresent: true, + }); + expect({ ...connected, source: "scan" }).toEqual(scanned); + expect(reloaded).toEqual(connected); + }); + it("passes marker provenance through both scan and connectPath", async () => { // A gallery clone writes templateId AND forkId; a scaffold writes starterId. const cloned = path.join(tmpRoot, "cloned"); @@ -158,7 +274,10 @@ describe("WorkflowRegistry", () => { await writeMarker(scaffolded, null, { starterId: "coding-pause" }); const byPath = new Map( - (await registry.scan(tmpRoot)).map((workflow) => [workflow.path, workflow]), + (await registry.scan(tmpRoot)).map((workflow) => [ + workflow.path, + workflow, + ]), ); expect(byPath.get(cloned)).toMatchObject({ templateId: "web-research-digest", @@ -220,7 +339,9 @@ describe("WorkflowRegistry", () => { await fs.rm(path.join(connectedDir, "sapiom.json")); await registry.scan(tmpRoot); - expect((await registry.list()).map((workflow) => workflow.path)).toEqual([connectedDir]); + expect((await registry.list()).map((workflow) => workflow.path)).toEqual([ + connectedDir, + ]); expect((await registry.list())[0].source).toBe("connect"); }); @@ -263,9 +384,9 @@ describe("WorkflowRegistry", () => { await registry.scan(path.join(tmpRoot, "left")); - expect((await registry.list()).map((workflow) => workflow.path).sort()).toEqual( - [left, right].sort(), - ); + expect( + (await registry.list()).map((workflow) => workflow.path).sort(), + ).toEqual([left, right].sort()); }); describe("prune", () => { @@ -279,11 +400,15 @@ describe("WorkflowRegistry", () => { const pruned = await registry.prune(); expect(pruned.map((workflow) => workflow.path)).toEqual([deadDir]); - expect((await registry.list()).map((workflow) => workflow.path)).toEqual([liveDir]); + expect((await registry.list()).map((workflow) => workflow.path)).toEqual([ + liveDir, + ]); // Persisted, not just dropped from the in-memory list. const reloaded = new WorkflowRegistry(registryPath); - expect((await reloaded.list()).map((workflow) => workflow.path)).toEqual([liveDir]); + expect((await reloaded.list()).map((workflow) => workflow.path)).toEqual([ + liveDir, + ]); }); it("keeps an existing-but-unbuilt project (only nonexistent paths are pruned)", async () => { @@ -294,7 +419,9 @@ describe("WorkflowRegistry", () => { await registry.scan(tmpRoot); expect(await registry.prune()).toEqual([]); - expect((await registry.list()).map((workflow) => workflow.path)).toEqual([unbuiltDir]); + expect((await registry.list()).map((workflow) => workflow.path)).toEqual([ + unbuiltDir, + ]); }); it("does not rewrite the registry file when nothing was pruned", async () => { @@ -316,6 +443,45 @@ describe("WorkflowRegistry", () => { }); describe("write serialization", () => { + it("compensates a scan superseded after rename so disk never retains unpublished rows", async () => { + const projectDir = path.join(tmpRoot, "source-agent"); + const renamed = deferred(); + const releaseRename = deferred(); + let pauseNextRename = false; + const guardedRegistry = new WorkflowRegistry(registryPath, undefined, { + afterPrimaryRename: async () => { + if (!pauseNextRename) return; + pauseNextRename = false; + renamed.resolve(); + await releaseRename.promise; + }, + }); + + await writeSourceAgent(projectDir, "accepted-a"); + await guardedRegistry.scanDetailed(tmpRoot); + pauseNextRename = true; + await writeSourceAgent(projectDir, "intermediate-b"); + const intermediateScan = guardedRegistry.scanDetailed(tmpRoot); + await renamed.promise; + + const renamedRows = JSON.parse( + await fs.readFile(registryPath, "utf8"), + ) as Array<{ sourceDefinitionName?: string }>; + expect(renamedRows[0]?.sourceDefinitionName).toBe("intermediate-b"); + + await writeSourceAgent(projectDir, "accepted-a"); + expect(guardedRegistry.markDiscoveryDirty(tmpRoot)).toBe(true); + releaseRename.resolve(); + await expect(intermediateScan).rejects.toThrow(/superseded/); + + // This recovery is deliberately row-identical to the in-memory accepted + // snapshot. Without compensation, it skips persistence and a restart + // observes the unpublished intermediate-b rename forever. + await guardedRegistry.scanDetailed(tmpRoot); + const reloaded = await new WorkflowRegistry(registryPath).list(); + expect(reloaded[0]?.sourceDefinitionName).toBe("accepted-a"); + }); + it("concurrent scan/prune calls serialize so no entry is lost from the persisted file", async () => { // Seed N workflow directories and fire scan + prune concurrently. // Without the write queue, a prune that starts reading this.workflows @@ -428,7 +594,6 @@ describe("WorkflowRegistry path identity under symlinks", () => { path.join(real, "growth"), ]); }); - }); describe("WorkflowRegistry scan rootedness (the 88-agent accumulation)", () => { @@ -438,9 +603,19 @@ describe("WorkflowRegistry scan rootedness (the 88-agent accumulation)", () => { /** The measured shape of the anomaly: a launch dir with its own agents, and * sibling checkouts of one repo, each carrying the SAME four agents. */ async function buildSiblingCheckouts(): Promise { - await writeMarker(path.join(tmpRoot, "wf-demo-testing", "agents", "mine"), 1); - await writeMarker(path.join(tmpRoot, "wf-demo-testing", "demo", "also-mine"), 2); - for (const checkout of ["design-eng", "design-eng-fix", "worktrees/port-pin"]) { + await writeMarker( + path.join(tmpRoot, "wf-demo-testing", "agents", "mine"), + 1, + ); + await writeMarker( + path.join(tmpRoot, "wf-demo-testing", "demo", "also-mine"), + 2, + ); + for (const checkout of [ + "design-eng", + "design-eng-fix", + "worktrees/port-pin", + ]) { await fs.mkdir(path.join(tmpRoot, checkout, ".git"), { recursive: true }); for (const agent of ["ari/orchestration", "brain/agent"]) { await writeMarker(path.join(tmpRoot, checkout, agent), null); @@ -450,7 +625,9 @@ describe("WorkflowRegistry scan rootedness (the 88-agent accumulation)", () => { beforeEach(async () => { tmpRoot = await fs.mkdtemp(path.join(os.tmpdir(), "harness-scan-rooted-")); - registry = new WorkflowRegistry(path.join(tmpRoot, ".state", "workflows.json")); + registry = new WorkflowRegistry( + path.join(tmpRoot, ".state", "workflows.json"), + ); }); afterEach(async () => { @@ -462,7 +639,9 @@ describe("WorkflowRegistry scan rootedness (the 88-agent accumulation)", () => { const found = await registry.scan(path.join(tmpRoot, "wf-demo-testing")); - expect(found.map((workflow) => path.relative(tmpRoot, workflow.path)).sort()).toEqual( + expect( + found.map((workflow) => path.relative(tmpRoot, workflow.path)).sort(), + ).toEqual( [ path.join("wf-demo-testing", "agents", "mine"), path.join("wf-demo-testing", "demo", "also-mine"), @@ -478,13 +657,17 @@ describe("WorkflowRegistry scan rootedness (the 88-agent accumulation)", () => { // copies of one agent" the rail was showing. const found = await registry.scan(tmpRoot); - expect(found.map((workflow) => path.relative(tmpRoot, workflow.path)).sort()).toEqual( + expect( + found.map((workflow) => path.relative(tmpRoot, workflow.path)).sort(), + ).toEqual( [ path.join("wf-demo-testing", "agents", "mine"), path.join("wf-demo-testing", "demo", "also-mine"), ].sort(), ); - expect(found.some((workflow) => workflow.path.includes("design-eng"))).toBe(false); + expect(found.some((workflow) => workflow.path.includes("design-eng"))).toBe( + false, + ); }); it("a checkout that IS an agent is still registered — the marker outranks the boundary", async () => { @@ -502,14 +685,20 @@ describe("WorkflowRegistry scan rootedness (the 88-agent accumulation)", () => { // the checkout, finds no marker below it, and the depth envelope alone // would call every row under it gone. const inner = path.join(tmpRoot, "opened-repo", "agents", "worker"); - await fs.mkdir(path.join(tmpRoot, "opened-repo", ".git"), { recursive: true }); + await fs.mkdir(path.join(tmpRoot, "opened-repo", ".git"), { + recursive: true, + }); await writeMarker(inner, 5); await registry.scan(path.join(tmpRoot, "opened-repo")); - expect((await registry.list()).map((workflow) => workflow.path)).toEqual([inner]); + expect((await registry.list()).map((workflow) => workflow.path)).toEqual([ + inner, + ]); await registry.scan(tmpRoot); - expect((await registry.list()).map((workflow) => workflow.path)).toEqual([inner]); + expect((await registry.list()).map((workflow) => workflow.path)).toEqual([ + inner, + ]); }); }); @@ -528,7 +717,7 @@ describe("WorkflowRegistry stale entries", () => { await fs.rm(tmpRoot, { recursive: true, force: true }); }); - it("list() prunes a deleted agent without waiting for a boot or a rescan", async () => { + it("list() stays cache-backed while its lazy prune runs in the background", async () => { const gone = path.join(tmpRoot, "gone"); const stays = path.join(tmpRoot, "stays"); await writeMarker(gone, 1); @@ -539,7 +728,14 @@ describe("WorkflowRegistry stale entries", () => { // A fresh instance, so the throttle window has not been consumed — this is // the shape of "the SPA asks /api/workflows again". const reader = new WorkflowRegistry(registryPath); - expect((await reader.list()).map((workflow) => workflow.path)).toEqual([stays]); + expect((await reader.list()).map((workflow) => workflow.path)).toEqual([ + gone, + stays, + ]); + await reader.prune(); + expect((await reader.list()).map((workflow) => workflow.path)).toEqual([ + stays, + ]); // And it is persisted, not merely filtered on the way out. expect(JSON.parse(await fs.readFile(registryPath, "utf8"))).toHaveLength(1); @@ -551,11 +747,61 @@ describe("WorkflowRegistry stale entries", () => { await writeMarker(here, 1); await writeMarker(elsewhere, 2); await registry.scan(tmpRoot); - await fs.rm(path.join(tmpRoot, "elsewhere"), { recursive: true, force: true }); + await fs.rm(path.join(tmpRoot, "elsewhere"), { + recursive: true, + force: true, + }); + + await registry.scan(path.join(tmpRoot, "here")); + + expect((await registry.list()).map((workflow) => workflow.path)).toEqual([ + here, + ]); + }); + it("retires source observations when an unrelated scan prunes their missing row", async () => { + const here = path.join(tmpRoot, "here", "agent"); + const elsewhere = path.join(tmpRoot, "elsewhere", "agent"); + await writeMarker(here, 1); + await writeSourceAgent(elsewhere, "elsewhere"); + await registry.scan(tmpRoot); + expect( + (await registry.inventorySnapshot(tmpRoot)).sourceObservations.some( + (observation) => observation.candidateRoot === elsewhere, + ), + ).toBe(true); + + await fs.rm(path.join(tmpRoot, "elsewhere"), { + recursive: true, + force: true, + }); await registry.scan(path.join(tmpRoot, "here")); - expect((await registry.list()).map((workflow) => workflow.path)).toEqual([here]); + const snapshot = await registry.inventorySnapshot(tmpRoot); + expect(snapshot.workflows.map((workflow) => workflow.path)).toEqual([here]); + expect( + snapshot.sourceObservations.some( + (observation) => + observation.candidateRoot === elsewhere || + observation.paths.some((observedPath) => + observedPath.startsWith(elsewhere), + ), + ), + ).toBe(false); + }); + + it("prune retires canonical identity and source-observation sidecars", async () => { + const gone = path.join(tmpRoot, "gone"); + await writeSourceAgent(gone, "gone"); + await registry.scan(tmpRoot); + await fs.rm(gone, { recursive: true, force: true }); + + await registry.prune(); + + const snapshot = await registry.inventorySnapshot(tmpRoot); + expect(snapshot.workflows).toEqual([]); + expect(snapshot.canonicalWorkflowRoots).toEqual([]); + expect(snapshot.sourceObservations).toEqual([]); }); it("keeps an unreadable-but-present directory: only a confirmed-missing path leaves", async () => { @@ -568,7 +814,9 @@ describe("WorkflowRegistry stale entries", () => { // never touched by the missing-path sweep. (Here the scan removes it; the // point is that the sweep is not what did.) const reader = new WorkflowRegistry(registryPath); - expect((await reader.list()).map((workflow) => workflow.path)).toEqual([unbuilt]); + expect((await reader.list()).map((workflow) => workflow.path)).toEqual([ + unbuilt, + ]); }); }); @@ -578,7 +826,9 @@ describe("WorkflowRegistry deep discovery under a chosen project root", () => { let registry: WorkflowRegistry; beforeEach(async () => { - projectRoot = await fs.mkdtemp(path.join(os.tmpdir(), "harness-deep-root-")); + projectRoot = await fs.mkdtemp( + path.join(os.tmpdir(), "harness-deep-root-"), + ); registryPath = path.join(projectRoot, ".state", "workflows.json"); registry = new WorkflowRegistry(registryPath); }); @@ -625,10 +875,14 @@ describe("WorkflowRegistry deep discovery under a chosen project root", () => { const deep = path.join(projectRoot, "backend", "src", "agents", "ads"); await writeMarker(deep, 1); for (const sibling of ["a", "b", "c", "d", "e", "f"]) { - await fs.mkdir(path.join(projectRoot, sibling, "child"), { recursive: true }); + await fs.mkdir(path.join(projectRoot, sibling, "child"), { + recursive: true, + }); } await registry.scan(projectRoot); - expect((await registry.list()).map((workflow) => workflow.path)).toEqual([deep]); + expect((await registry.list()).map((workflow) => workflow.path)).toEqual([ + deep, + ]); // Root + a few level-1 dirs and nothing more: the scan cannot have looked // at level 4, where the row lives. @@ -638,7 +892,9 @@ describe("WorkflowRegistry deep discovery under a chosen project root", () => { expect(found).toEqual([]); expect(starved.truncated).toBe(true); expect(starved.envelopeDepth).toBeLessThan(4); - expect((await registry.list()).map((workflow) => workflow.path)).toEqual([deep]); + expect((await registry.list()).map((workflow) => workflow.path)).toEqual([ + deep, + ]); // And a scan that DID cover that depth still reconciles it away when the // marker really is gone — the protection is about coverage, not immunity. @@ -653,7 +909,11 @@ describe("WorkflowRegistry deep discovery under a chosen project root", () => { const real = path.join(projectRoot, "pkg", "agents", "one"); await writeMarker(real, 1); // loop -> the root itself: an infinitely deep tree if links were followed. - await fs.symlink(projectRoot, path.join(projectRoot, "pkg", "loop"), "dir"); + await fs.symlink( + projectRoot, + path.join(projectRoot, "pkg", "loop"), + "dir", + ); const started = performance.now(); const found = await registry.scan(projectRoot); @@ -675,7 +935,10 @@ describe("WorkflowRegistry deep discovery under a chosen project root", () => { { recursive: true }, ); } - await writeMarker(path.join(projectRoot, "node_modules", "pkg-7", "agent"), 9); + await writeMarker( + path.join(projectRoot, "node_modules", "pkg-7", "agent"), + 9, + ); const budget = new AgentProjectScanBudget(); const found = await registry.scan(projectRoot, budget); @@ -686,3 +949,561 @@ describe("WorkflowRegistry deep discovery under a chosen project root", () => { expect(budget.visited).toBeLessThan(10); }); }); + +describe("WorkflowRegistry syntax-only inventory reconciliation", () => { + let root: string; + let registryPath: string; + + beforeEach(async () => { + root = await fs.mkdtemp(path.join(os.tmpdir(), "harness-source-registry-")); + registryPath = path.join(root, ".state", "workflows.json"); + }); + + afterEach(async () => { + await fs.rm(root, { recursive: true, force: true }); + }); + + it("discovers, persists, and reloads a source-only row without shape drift", async () => { + const agent = path.join(root, "agents", "billing"); + await writeSourceAgent(agent, "billing"); + await fs.writeFile( + path.join(agent, "package.json"), + JSON.stringify({ name: "@acme/billing" }), + ); + const registry = new WorkflowRegistry(registryPath); + + const found = await registry.scan(root); + expect(found).toEqual([ + { + name: "@acme/billing", + path: agent, + definitionId: null, + definitionSlug: null, + sourceDefinitionName: "billing", + activeBuildRunId: null, + activeBuildRunStatus: null, + templateId: null, + forkId: null, + starterId: null, + source: "scan", + }, + ]); + expect((await registry.inventorySnapshot(root)).status).toBe("complete"); + + const reloaded = new WorkflowRegistry(registryPath); + expect(await reloaded.list()).toEqual(await registry.list()); + expect((await reloaded.inventorySnapshot(root)).status).toBe("degraded"); + }); + + it("retires the candidate's syntax observations when a valid marker takes precedence", async () => { + const agent = path.join(root, "agent"); + await writeSourceAgent( + agent, + "source-name", + `import { helper } from "./helper";\nvoid helper;`, + ); + await fs.writeFile( + path.join(agent, "helper.ts"), + `export const helper = 1;`, + ); + const registry = new WorkflowRegistry(registryPath); + await registry.scan(root); + expect( + (await registry.inventorySnapshot(root)).sourceObservations.some( + (observation) => observation.candidateRoot === agent, + ), + ).toBe(true); + + await writeMarker(agent, null, { name: "marker-name" }); + await registry.scan(root); + + expect( + (await registry.inventorySnapshot(root)).sourceObservations.some( + (observation) => observation.candidateRoot === agent, + ), + ).toBe(false); + }); + + it("keeps source observations scoped to the scan envelope across a direct child rescan", async () => { + const agent = path.join(root, "agent"); + const shared = path.join(root, "shared.ts"); + await fs.mkdir(agent, { recursive: true }); + await fs.writeFile( + shared, + `export { defineAgent as makeAgent } from "@sapiom/agent";`, + ); + await fs.writeFile( + path.join(agent, "index.ts"), + `import { makeAgent } from "../shared.js"; +export const agent = makeAgent({ name: "broad-agent" });`, + ); + const registry = new WorkflowRegistry(registryPath); + await registry.scan(root); + await registry.scan(agent); + + const observations = (await registry.inventorySnapshot(root)) + .sourceObservations; + const broad = observations.find( + (observation) => + observation.workspaceRoot === root && + observation.candidateRoot === agent, + ); + const direct = observations.find( + (observation) => + observation.workspaceRoot === agent && + observation.candidateRoot === agent, + ); + expect(broad?.paths).toContain(shared); + expect(direct?.paths).not.toContain(shared); + }); + + it.skipIf(process.platform === "win32")( + "projects lexical watch paths into a symlink-selected canonical envelope", + async () => { + const realWorkspace = path.join(root, "real-workspace"); + const linkedWorkspace = path.join(root, "linked-workspace"); + const realAgent = path.join(realWorkspace, "agent"); + await fs.mkdir(realAgent, { recursive: true }); + await fs.writeFile( + path.join(realAgent, "index.ts"), + `import { makeAgent } from "./helper.js"; +export const agent = makeAgent({ name: "linked-agent" });`, + ); + await fs.writeFile( + path.join(realAgent, "helper.ts"), + `export { defineAgent as makeAgent } from "@sapiom/agent";`, + ); + await fs.symlink(realWorkspace, linkedWorkspace, "dir"); + const registry = new WorkflowRegistry(registryPath); + + await registry.scan(linkedWorkspace); + + const observation = ( + await registry.inventorySnapshot(linkedWorkspace) + ).sourceObservations.find((entry) => entry.candidateRoot === realAgent); + expect(observation?.workspaceRoot).toBe(realWorkspace); + expect(observation?.paths).toContain(path.join(realAgent, "helper.ts")); + expect( + observation?.paths.some((observedPath) => + observedPath.startsWith(linkedWorkspace), + ), + ).toBe(false); + }, + ); + + it("falls through an invalid marker but not a foreign repository boundary", async () => { + const nested = path.join(root, "nested-repo"); + await writeSourceAgent(nested, "nested"); + await fs.writeFile(path.join(nested, "sapiom.json"), "not-json"); + await fs.mkdir(path.join(nested, ".git")); + const parentRegistry = new WorkflowRegistry(registryPath); + + expect(await parentRegistry.scan(root)).toEqual([]); + + const directRegistry = new WorkflowRegistry( + path.join(root, ".state", "direct.json"), + ); + await expect(directRegistry.scan(nested)).resolves.toMatchObject([ + { path: nested, sourceDefinitionName: "nested" }, + ]); + }); + + it("continues below an incomplete source candidate to nested agents and markers", async () => { + const unresolved = path.join(root, "unresolved"); + await fs.mkdir(unresolved, { recursive: true }); + await fs.writeFile( + path.join(unresolved, "index.ts"), + `export { agent } from "./missing";`, + ); + const nestedSource = path.join(unresolved, "nested-source"); + const nestedMarker = path.join(unresolved, "nested-marker"); + await writeSourceAgent(nestedSource, "nested-source"); + await writeMarker(nestedMarker, 17, { name: "nested-marker" }); + const registry = new WorkflowRegistry(registryPath); + + const found = await registry.scan(root); + + expect(found).toEqual( + expect.arrayContaining([ + expect.objectContaining({ + path: nestedSource, + sourceDefinitionName: "nested-source", + }), + expect.objectContaining({ + path: nestedMarker, + definitionId: 17, + }), + ]), + ); + expect((await registry.inventorySnapshot(root)).status).toBe("degraded"); + }); + + it.each(["marker", "source"] as const)( + "preserves directly proven descendants below a discovered %s stop root", + async (parentKind) => { + const parent = path.join(root, `${parentKind}-parent`); + const nested = path.join(parent, "nested-agent"); + await writeSourceAgent(nested, "nested-agent"); + const registry = new WorkflowRegistry(registryPath); + + await registry.scan(nested); + expect((await registry.inventorySnapshot(nested)).status).toBe( + "complete", + ); + + if (parentKind === "marker") { + await writeMarker(parent, 77, { name: "parent-marker" }); + } else { + await writeSourceAgent(parent, "parent-source"); + } + await registry.scan(root); + + expect((await registry.list()).map((row) => row.path).sort()).toEqual( + [nested, parent].sort(), + ); + expect((await registry.inventorySnapshot(root)).status).toBe("complete"); + // The parent candidate was proven, but its descendants were deliberately + // not traversed after the discovery stop. + expect((await registry.inventorySnapshot(nested)).status).toBe( + "complete", + ); + }, + ); + + it("preserves connected cloud evidence while syntax identity becomes canonical", async () => { + const agent = path.join(root, "connected"); + await writeMarker(agent, 42, { + name: "payments", + templateId: "template-old", + }); + const registry = new WorkflowRegistry(registryPath); + await registry.connectPath(agent); + await fs.rm(path.join(agent, "sapiom.json")); + await writeSourceAgent(agent, "billing"); + + await registry.scan(root); + expect(await registry.list()).toMatchObject([ + { + path: agent, + source: "connect", + definitionId: 42, + definitionSlug: "payments", + templateId: "template-old", + sourceDefinitionName: "billing", + }, + ]); + + await writeMarker(agent, 99, { name: "current-marker" }); + await registry.scan(root); + const adoptedMarker = (await registry.list())[0]!; + expect(adoptedMarker.definitionId).toBe(42); + expect(adoptedMarker.definitionSlug).toBe("payments"); + expect(adoptedMarker).not.toHaveProperty("sourceDefinitionName"); + }); + + it.each(["remove", "ordinary"])( + "retires stale source evidence from a connected row after a definitive %s", + async (mode) => { + const agent = path.join(root, "connected"); + await writeSourceAgent(agent, "before"); + const registry = new WorkflowRegistry(registryPath); + await registry.scan(root); + await registry.connectPath(agent); + + if (mode === "remove") { + await fs.rm(path.join(agent, "index.ts")); + } else { + await fs.writeFile( + path.join(agent, "index.ts"), + `export const ordinary = true;`, + ); + } + await registry.scan(root); + + const row = (await registry.list())[0]!; + expect(row.source).toBe("connect"); + expect(row).not.toHaveProperty("sourceDefinitionName"); + }, + ); + + it.skipIf(process.platform === "win32")( + "deduplicates canonical aliases while preserving lexical path and complementary evidence", + async () => { + const real = path.join(root, "real-agent"); + const alias = path.join(root, "agent-alias"); + await fs.mkdir(real); + await fs.symlink(real, alias, "dir"); + await fs.mkdir(path.dirname(registryPath), { recursive: true }); + await fs.writeFile( + registryPath, + JSON.stringify([ + { + name: "connected", + path: alias, + definitionId: 7, + definitionSlug: "marker-name", + sourceDefinitionName: null, + templateId: null, + forkId: "fork-1", + starterId: null, + source: "connect", + }, + { + name: "source", + path: real, + definitionId: null, + definitionSlug: null, + sourceDefinitionName: "source-name", + templateId: "template-1", + forkId: null, + starterId: "starter-1", + source: "scan", + }, + ]), + ); + + const registry = new WorkflowRegistry(registryPath); + expect(await registry.list()).toMatchObject([ + { + path: alias, + source: "connect", + definitionId: 7, + definitionSlug: "marker-name", + sourceDefinitionName: "source-name", + templateId: "template-1", + forkId: "fork-1", + starterId: "starter-1", + }, + ]); + }, + ); + + it.skipIf(process.platform === "win32")( + "keeps a scan row's lexical alias stable across canonical-root rescans", + async () => { + const realRoot = path.join(root, "real-workspace"); + const linkedRoot = path.join(root, "workspace-link"); + const realAgent = path.join(realRoot, "agent"); + await writeSourceAgent(realAgent, "stable-path"); + await fs.symlink(realRoot, linkedRoot, "dir"); + const registry = new WorkflowRegistry(registryPath); + + await registry.scan(linkedRoot); + const lexicalAgent = path.join(linkedRoot, "agent"); + expect((await registry.list())[0]?.path).toBe(lexicalAgent); + const before = await registry.inventorySnapshot(linkedRoot); + + await registry.scan(realRoot); + expect((await registry.list())[0]?.path).toBe(lexicalAgent); + expect((await registry.inventorySnapshot(realRoot)).workflows).toEqual( + before.workflows, + ); + expect((await registry.inventorySnapshot(linkedRoot)).status).toBe( + "complete", + ); + expect((await registry.inventorySnapshot(realRoot)).status).toBe( + "complete", + ); + }, + ); + + it("sanitizes source identity and bounded package labels before persistence", async () => { + const invalid = path.join(root, "invalid-package"); + await writeSourceAgent(invalid, "bad/name"); + await fs.writeFile( + path.join(invalid, "package.json"), + JSON.stringify({ name: 42, padding: "x".repeat(70 * 1024) }), + ); + const registry = new WorkflowRegistry(registryPath); + + await registry.scan(root); + expect(await registry.list()).toMatchObject([ + { + name: "invalid-package", + sourceDefinitionName: null, + }, + ]); + expect(await new WorkflowRegistry(registryPath).list()).toEqual( + await registry.list(), + ); + }); + + it.skipIf(process.platform === "win32")( + "does not follow a package.json symlink for a display label", + async () => { + const agent = path.join(root, "symlink-package"); + const external = path.join(root, "external-package.json"); + await writeSourceAgent(agent, "safe"); + await fs.writeFile(external, JSON.stringify({ name: "external-secret" })); + await fs.symlink(external, path.join(agent, "package.json")); + const registry = new WorkflowRegistry(registryPath); + + await registry.scan(root); + expect((await registry.list())[0]?.name).toBe("symlink-package"); + }, + ); + + it("keeps completeness honest and supersedes only proven child coverage", async () => { + const child = path.join(root, "child"); + await writeSourceAgent(child, "child"); + const registry = new WorkflowRegistry(registryPath); + expect((await registry.inventorySnapshot(root)).status).toBe("degraded"); + + await registry.scan(child); + expect((await registry.inventorySnapshot(child)).status).toBe("complete"); + await fs.writeFile( + path.join(child, "index.ts"), + `export { agent } from "./missing";`, + ); + await registry.scan(root); + expect((await registry.inventorySnapshot(child)).status).toBe("degraded"); + expect((await registry.inventorySnapshot(root)).status).toBe("degraded"); + + await writeSourceAgent(child, "child"); + await registry.scan(root); + expect((await registry.inventorySnapshot(root)).status).toBe("complete"); + expect((await registry.inventorySnapshot(child)).status).toBe("complete"); + }); + + it("preserves a complete child claim when uncertainty is confined to a sibling", async () => { + const proven = path.join(root, "proven"); + const unresolved = path.join(root, "unresolved"); + await writeSourceAgent(proven, "proven"); + await fs.mkdir(unresolved, { recursive: true }); + await fs.writeFile( + path.join(unresolved, "index.ts"), + `export { agent } from "./missing";`, + ); + const registry = new WorkflowRegistry(registryPath); + await registry.scan(proven); + expect((await registry.inventorySnapshot(proven)).status).toBe("complete"); + + await registry.scan(root); + + expect((await registry.inventorySnapshot(root)).status).toBe("degraded"); + expect((await registry.inventorySnapshot(proven)).status).toBe("complete"); + }); + + it("keeps exact ignored-child status separate from a complete parent envelope", async () => { + const ignored = path.join(root, "node_modules", "selected-agent"); + await fs.mkdir(ignored, { recursive: true }); + await fs.writeFile( + path.join(ignored, "index.ts"), + `export { agent } from "./missing";`, + ); + const registry = new WorkflowRegistry(registryPath); + await registry.scan(ignored); + expect((await registry.inventorySnapshot(ignored)).status).toBe("degraded"); + + await registry.scan(root); + + expect((await registry.inventorySnapshot(root)).status).toBe("complete"); + expect((await registry.inventorySnapshot(ignored)).status).toBe("degraded"); + }); + + it("preserves a directly scanned source row and its proof below an ignored parent boundary", async () => { + const ignored = path.join(root, "node_modules", "selected-agent"); + await writeSourceAgent( + ignored, + "selected-agent", + `import { helper } from "./helper";\nvoid helper;`, + ); + await fs.writeFile( + path.join(ignored, "helper.ts"), + `export const helper = true;`, + ); + const registry = new WorkflowRegistry(registryPath); + + await registry.scan(ignored); + const direct = await registry.inventorySnapshot(ignored); + expect(direct.workflows.map((workflow) => workflow.path)).toEqual([ + ignored, + ]); + expect(direct.canonicalWorkflowRoots).toEqual([ + expect.objectContaining({ + canonicalRoot: ignored, + identityEvidence: "source", + }), + ]); + expect( + direct.sourceObservations.some( + (observation) => observation.candidateRoot === ignored, + ), + ).toBe(true); + + await registry.scan(root); + + const parent = await registry.inventorySnapshot(root); + expect(parent.workflows.map((workflow) => workflow.path)).toEqual([ + ignored, + ]); + expect(parent.canonicalWorkflowRoots).toEqual([ + expect.objectContaining({ + canonicalRoot: ignored, + identityEvidence: "source", + }), + ]); + expect( + parent.sourceObservations.some( + (observation) => observation.candidateRoot === ignored, + ), + ).toBe(true); + expect((await registry.inventorySnapshot(ignored)).status).toBe("complete"); + }); + + it.skipIf(process.platform === "win32")( + "retires a parent-owned source row when that candidate becomes a foreign repository", + async () => { + const checkout = path.join(root, "checkout"); + await writeSourceAgent(checkout, "checkout-agent"); + const registry = new WorkflowRegistry(registryPath); + + await registry.scan(root); + expect((await registry.list()).map((workflow) => workflow.path)).toEqual([ + checkout, + ]); + + await fs.mkdir(path.join(checkout, ".git")); + await registry.scan(root); + expect(await registry.list()).toEqual([]); + + // Selecting the repository itself is explicit proof. A later parent + // scan preserves that directly-owned row behind the boundary. + await registry.scan(checkout); + expect((await registry.list()).map((workflow) => workflow.path)).toEqual([ + checkout, + ]); + await registry.scan(root); + expect((await registry.list()).map((workflow) => workflow.path)).toEqual([ + checkout, + ]); + }, + ); + + it.skipIf(process.platform === "win32")( + "keeps nested-repository completeness exact across parent scans", + async () => { + const checkout = path.join(root, "checkout"); + await writeSourceAgent(checkout, "checkout-agent"); + await fs.mkdir(path.join(checkout, ".git")); + const registry = new WorkflowRegistry(registryPath); + + await registry.scan(root); + expect((await registry.inventorySnapshot(root)).status).toBe("complete"); + expect((await registry.inventorySnapshot(checkout)).status).toBe( + "degraded", + ); + + await registry.scan(checkout); + expect((await registry.inventorySnapshot(checkout)).status).toBe( + "complete", + ); + + await registry.scan(root); + expect((await registry.inventorySnapshot(root)).status).toBe("complete"); + expect((await registry.inventorySnapshot(checkout)).status).toBe( + "complete", + ); + }, + ); +}); diff --git a/packages/harness/src/core/workflow-registry.ts b/packages/harness/src/core/workflow-registry.ts index 9b96701cc..da835b152 100644 --- a/packages/harness/src/core/workflow-registry.ts +++ b/packages/harness/src/core/workflow-registry.ts @@ -12,11 +12,16 @@ */ import * as fs from "node:fs/promises"; +import { constants as fsConstants } from "node:fs"; +import { randomUUID } from "node:crypto"; import * as os from "node:os"; import * as path from "node:path"; import { Router, type Router as ExpressRouter } from "express"; -import { HARNESS_PATHS, type WorkflowInfo } from "../shared/types.js"; +import { + HARNESS_PATHS, + type WorkflowInfo as PublicWorkflowInfo, +} from "../shared/types.js"; import { type AgentProjectMarker, type AgentProjectMarkerInspection, @@ -25,18 +30,129 @@ import { type AgentProjectWalkOptions, inspectAgentProjectMarker, isAgentProjectScanIgnoredDir, - readAgentProjectMarker, walkAgentProjectTreeAsync, } from "./agent-project-discovery.js"; +import { + AGENT_SOURCE_ENTRYPOINT, + AgentSourceDiscovery, + AgentSourceScanBudget, +} from "./agent-source-discovery.js"; +import { rememberCanonicalGraphPath } from "./canonical-graph-path.js"; import { hasTraversalSegment, resolveWithinRoot } from "./path-safety.js"; -import { canonicalGraphPath } from "./system-graph-inventory.js"; function expandHome(inputPath: string): string { if (inputPath === "~") return os.homedir(); - if (inputPath.startsWith("~/")) return path.join(os.homedir(), inputPath.slice(2)); + if (inputPath.startsWith("~/")) + return path.join(os.homedir(), inputPath.slice(2)); return inputPath; } +function compareText(left: string, right: string): number { + return left === right ? 0 : left < right ? -1 : 1; +} + +const REGISTRY_FS_CONCURRENCY = 16; +const PACKAGE_JSON_MAX_BYTES = 64 * 1024; + +async function mapBounded( + values: readonly T[], + mapper: (value: T, index: number) => Promise, + concurrency = REGISTRY_FS_CONCURRENCY, +): Promise { + const results = new Array(values.length); + let cursor = 0; + const workers = Array.from( + { length: Math.min(concurrency, values.length) }, + async () => { + while (cursor < values.length) { + const index = cursor; + cursor += 1; + results[index] = await mapper(values[index] as T, index); + } + }, + ); + await Promise.all(workers); + return results; +} + +function sameFileSnapshot( + left: import("node:fs").Stats, + right: import("node:fs").Stats, +): boolean { + return ( + left.dev === right.dev && + left.ino === right.ino && + left.size === right.size && + left.mtimeMs === right.mtimeMs + ); +} + +function hasControlCharacter(value: string): boolean { + return [...value].some((character) => { + const code = character.codePointAt(0)!; + return code <= 0x1f || (code >= 0x7f && code <= 0x9f); + }); +} + +function safeDisplayName(value: unknown, fallback: string): string { + return typeof value === "string" && + value.trim() && + !hasControlCharacter(value) + ? value.trim() + : fallback; +} + +function safeDefinitionName(value: unknown): string | null { + if (typeof value !== "string") return null; + const normalized = value.trim(); + if ( + !normalized || + hasControlCharacter(normalized) || + normalized === "." || + normalized === ".." || + normalized.startsWith("local:") || + normalized.includes("/") || + normalized.includes("\\") + ) { + return null; + } + return normalized; +} + +function safeOpaqueString(value: unknown): string | null { + if (typeof value !== "string") return null; + const normalized = value.trim(); + return normalized && !hasControlCharacter(normalized) ? normalized : null; +} + +function safeDefinitionId(value: unknown): number | null { + return typeof value === "number" && Number.isSafeInteger(value) && value > 0 + ? value + : null; +} + +function normalizedMarkerFields( + marker: AgentProjectMarker, +): Pick< + WorkflowInfo, + | "definitionId" + | "definitionSlug" + | "templateId" + | "forkId" + | "starterId" + | "markerPresent" +> { + const untrusted = marker as Record; + return { + definitionId: safeDefinitionId(untrusted.definitionId), + definitionSlug: safeDefinitionName(untrusted.name), + templateId: safeOpaqueString(untrusted.templateId), + forkId: safeOpaqueString(untrusted.forkId), + starterId: safeOpaqueString(untrusted.starterId), + markerPresent: true, + }; +} + // `dir` reaching these sinks is always a resolved absolute path (from // path.resolve in scan/connectPath, or a confined descent in the scan walk), // so a @@ -44,11 +160,6 @@ function expandHome(inputPath: string): string { // guarantee explicit and local to each fs read, and covers the arbitrary // path connectPath accepts (which has no scan root to confine it to). -async function readMarker(dir: string): Promise { - if (hasTraversalSegment(dir)) return null; - return readAgentProjectMarker(dir); -} - async function inspectMarker( dir: string, ): Promise { @@ -57,15 +168,351 @@ async function inspectMarker( } async function nameFor(dir: string): Promise { - if (hasTraversalSegment(dir)) return path.basename(dir); + const fallback = path.basename(dir); + if (hasTraversalSegment(dir)) return fallback; + const packagePath = path.join(dir, "package.json"); try { - const raw = await fs.readFile(path.join(dir, "package.json"), "utf8"); - const pkg = JSON.parse(raw) as { name?: string }; - if (pkg.name) return pkg.name; + const initialDirectory = await fs.lstat(dir); + if (!initialDirectory.isDirectory() || initialDirectory.isSymbolicLink()) { + return fallback; + } + const canonicalDirectoryPath = await fs.realpath(dir); + const initial = await fs.lstat(packagePath); + if ( + initial.isSymbolicLink() || + !initial.isFile() || + initial.size > PACKAGE_JSON_MAX_BYTES + ) { + return fallback; + } + const handle = await fs.open( + packagePath, + fsConstants.O_RDONLY | fsConstants.O_NOFOLLOW | fsConstants.O_NONBLOCK, + ); + let raw: string; + try { + const opened = await handle.stat(); + const beforeReadDirectory = await fs.lstat(dir); + const beforeReadCanonicalDirectory = await fs.realpath(dir); + const beforeReadPath = await fs.lstat(packagePath); + if ( + !opened.isFile() || + !sameFileSnapshot(opened, initial) || + beforeReadDirectory.isSymbolicLink() || + !beforeReadDirectory.isDirectory() || + !sameFileSnapshot(beforeReadDirectory, initialDirectory) || + beforeReadCanonicalDirectory !== canonicalDirectoryPath || + beforeReadPath.isSymbolicLink() || + !beforeReadPath.isFile() || + !sameFileSnapshot(beforeReadPath, initial) + ) { + return fallback; + } + const bytes = Buffer.alloc(initial.size + 1); + let offset = 0; + while (offset < bytes.length) { + const { bytesRead } = await handle.read( + bytes, + offset, + bytes.length - offset, + offset, + ); + if (bytesRead === 0) break; + offset += bytesRead; + } + const finalHandle = await handle.stat(); + const finalPath = await fs.lstat(packagePath); + if ( + finalPath.isSymbolicLink() || + !finalPath.isFile() || + !sameFileSnapshot(opened, initial) || + !sameFileSnapshot(finalHandle, initial) || + !sameFileSnapshot(finalPath, initial) || + offset !== initial.size + ) { + return fallback; + } + raw = bytes.subarray(0, offset).toString("utf8"); + } finally { + await handle.close(); + } + const pkg = JSON.parse(raw) as { name?: unknown }; + return safeDisplayName(pkg.name, fallback); } catch { // No package.json (or it doesn't parse) — fall back to the directory name. } - return path.basename(dir); + return fallback; +} + +type CanonicalDirectoryEvidence = + | { status: "resolved"; key: string } + | { status: "missing"; key: string } + | { status: "unreadable"; key: string }; + +async function canonicalDirectoryEvidence( + input: string, +): Promise { + const absolute = path.resolve(input); + try { + const key = await fs.realpath(absolute); + rememberCanonicalGraphPath(absolute, key); + return { status: "resolved", key }; + } catch (error) { + const code = (error as NodeJS.ErrnoException).code; + return { + status: + code === "ENOENT" || code === "ENOTDIR" ? "missing" : "unreadable", + key: absolute, + }; + } +} + +async function canonicalDirectory(input: string): Promise { + return (await canonicalDirectoryEvidence(input)).key; +} + +function hasSourceDiscoveryEvidence(workflow: WorkflowInfo): boolean { + return Object.prototype.hasOwnProperty.call(workflow, "sourceDefinitionName"); +} + +function workflowEvidenceScore(workflow: WorkflowInfo): number { + return ( + (workflow.source === "connect" ? 64 : 0) + + (workflow.definitionId !== null ? 16 : 0) + + (workflow.definitionSlug !== null ? 8 : 0) + + (workflow.templateId ? 4 : 0) + + (workflow.forkId ? 2 : 0) + + (workflow.starterId ? 2 : 0) + + (workflow.activeBuildRunId ? 2 : 0) + + (hasSourceDiscoveryEvidence(workflow) ? 1 : 0) + ); +} + +function compareWorkflowEvidence( + left: WorkflowInfo, + right: WorkflowInfo, +): number { + return ( + workflowEvidenceScore(right) - workflowEvidenceScore(left) || + compareText(left.path, right.path) || + compareText(left.name, right.name) + ); +} + +function workflowRowsEqual( + left: readonly WorkflowInfo[], + right: readonly WorkflowInfo[], +): boolean { + if (left.length !== right.length) return false; + const key = (workflow: WorkflowInfo): string => + JSON.stringify({ + path: workflow.path, + name: workflow.name, + definitionId: workflow.definitionId, + definitionSlug: workflow.definitionSlug, + sourceDefinitionName: hasSourceDiscoveryEvidence(workflow) + ? [true, workflow.sourceDefinitionName ?? null] + : [false], + markerPresent: workflow.markerPresent === true, + activeBuildRunId: workflow.activeBuildRunId ?? null, + activeBuildRunStatus: workflow.activeBuildRunStatus ?? null, + templateId: workflow.templateId ?? null, + forkId: workflow.forkId ?? null, + starterId: workflow.starterId ?? null, + source: workflow.source, + }); + return left.every((workflow, index) => key(workflow) === key(right[index]!)); +} + +function statusMapsEqual( + left: ReadonlyMap, + right: ReadonlyMap, +): boolean { + return ( + left.size === right.size && + [...left].every(([root, status]) => right.get(root) === status) + ); +} + +function evidenceMapsEqual( + left: ReadonlyMap, + right: ReadonlyMap, +): boolean { + return ( + left.size === right.size && + [...left].every(([root, evidence]) => right.get(root) === evidence) + ); +} + +interface StoredSourceObservation { + candidateRoot: string; + workspaceRoot: string; + paths: readonly string[]; +} + +function sourceObservationKey( + workspaceRoot: string, + candidateRoot: string, +): string { + return JSON.stringify([workspaceRoot, candidateRoot]); +} + +function observationMapsEqual( + left: ReadonlyMap, + right: ReadonlyMap, +): boolean { + return ( + left.size === right.size && + [...left].every( + ([root, observations]) => + JSON.stringify(observations) === JSON.stringify(right.get(root)), + ) + ); +} + +function mergeLoadedEvidence( + preferred: WorkflowInfo, + fallback: WorkflowInfo, +): WorkflowInfo { + const merged: WorkflowInfo = { + ...preferred, + source: + preferred.source === "connect" || fallback.source === "connect" + ? "connect" + : "scan", + definitionId: preferred.definitionId ?? fallback.definitionId, + definitionSlug: preferred.definitionSlug ?? fallback.definitionSlug, + templateId: preferred.templateId ?? fallback.templateId, + forkId: preferred.forkId ?? fallback.forkId, + starterId: preferred.starterId ?? fallback.starterId, + activeBuildRunId: + preferred.activeBuildRunId ?? fallback.activeBuildRunId ?? null, + activeBuildRunStatus: + preferred.activeBuildRunStatus ?? fallback.activeBuildRunStatus ?? null, + }; + if (preferred.markerPresent === true || fallback.markerPresent === true) { + merged.markerPresent = true; + } else { + delete merged.markerPresent; + } + const sourceName = + (hasSourceDiscoveryEvidence(preferred) + ? preferred.sourceDefinitionName + : undefined) ?? + (hasSourceDiscoveryEvidence(fallback) + ? (fallback.sourceDefinitionName ?? null) + : hasSourceDiscoveryEvidence(preferred) + ? null + : undefined); + if (sourceName !== undefined) { + merged.sourceDefinitionName = sourceName; + } else { + delete merged.sourceDefinitionName; + } + return merged; +} + +async function normalizeLoadedWorkflows( + workflows: WorkflowInfo[], +): Promise { + const candidates = workflows.filter( + (workflow) => + workflow && + typeof workflow.path === "string" && + path.isAbsolute(workflow.path), + ); + const normalized = await mapBounded(candidates, async (workflow) => { + const absolutePath = path.resolve(workflow.path); + const normalizedWorkflow: WorkflowInfo = { + name: safeDisplayName(workflow.name, path.basename(absolutePath)), + path: absolutePath, + definitionId: safeDefinitionId(workflow.definitionId), + definitionSlug: safeDefinitionName(workflow.definitionSlug), + templateId: safeOpaqueString(workflow.templateId), + forkId: safeOpaqueString(workflow.forkId), + starterId: safeOpaqueString(workflow.starterId), + source: workflow.source === "connect" ? "connect" : "scan", + }; + if (workflow.markerPresent === true) + normalizedWorkflow.markerPresent = true; + if (Object.prototype.hasOwnProperty.call(workflow, "activeBuildRunId")) { + normalizedWorkflow.activeBuildRunId = safeOpaqueString( + workflow.activeBuildRunId, + ); + } + if ( + Object.prototype.hasOwnProperty.call(workflow, "activeBuildRunStatus") + ) { + normalizedWorkflow.activeBuildRunStatus = safeOpaqueString( + workflow.activeBuildRunStatus, + ); + } + if (hasSourceDiscoveryEvidence(workflow)) { + normalizedWorkflow.sourceDefinitionName = safeDefinitionName( + workflow.sourceDefinitionName, + ); + } + return { + workflow: normalizedWorkflow, + canonicalKey: await canonicalDirectory(absolutePath), + }; + }); + normalized.sort((left, right) => + compareWorkflowEvidence(left.workflow, right.workflow), + ); + const byPath = new Map(); + for (const { workflow, canonicalKey } of normalized) { + const existing = byPath.get(canonicalKey); + if (!existing) { + byPath.set(canonicalKey, workflow); + continue; + } + byPath.set(canonicalKey, mergeLoadedEvidence(existing, workflow)); + } + return [...byPath.values()].sort((left, right) => + compareText(left.path, right.path), + ); +} + +function mergeScannedWorkflow( + existing: WorkflowInfo | undefined, + discovered: WorkflowInfo, +): WorkflowInfo { + if (!existing) return discovered; + if (hasSourceDiscoveryEvidence(discovered)) { + if (existing.source === "connect") { + const connected = { + ...existing, + name: discovered.name, + }; + connected.sourceDefinitionName = discovered.sourceDefinitionName ?? null; + delete connected.markerPresent; + return connected; + } + return { ...discovered, path: existing.path }; + } + if (existing.source === "connect") { + const connected = { + ...existing, + name: discovered.name, + definitionId: existing.definitionId ?? discovered.definitionId, + definitionSlug: existing.definitionSlug ?? discovered.definitionSlug, + templateId: existing.templateId ?? discovered.templateId, + forkId: existing.forkId ?? discovered.forkId, + starterId: existing.starterId ?? discovered.starterId, + markerPresent: true as const, + }; + delete connected.sourceDefinitionName; + return connected; + } + const markerRefreshed = { + ...existing, + ...discovered, + path: existing.path, + source: existing.source, + }; + delete markerRefreshed.sourceDefinitionName; + return markerRefreshed; } /** What one scan of a root turned up, and how far it can be trusted. */ @@ -74,6 +521,22 @@ export interface AgentProjectScanResult { found: WorkflowInfo[]; /** Subtrees left opaque by a transient filesystem error. */ unreconciledRoots: string[]; + /** Existing entrypoints proven to export no supported agent definition. */ + notAgentRoots: string[]; + /** Symlink entries the no-follow policy deliberately left opaque. */ + opaqueRoots: string[]; + /** + * Valid marker/source projects whose descendants were deliberately not + * traversed. The project root itself is proven, but everything below it is + * outside this scan's reconciliation envelope. + */ + discoveredStopRoots: string[]; + /** Bounded analyzer path observations, including deterministic absences. */ + sourceObservations: readonly { + candidateRoot: string; + workspaceRoot: string; + paths: readonly string[]; + }[]; /** * Nested repository checkouts the walk declined to enter (see * {@link isForeignRepositoryRoot}). A scan that comes back short is then able @@ -84,6 +547,10 @@ export interface AgentProjectScanResult { repositoryBoundaries: string[]; /** The traversal allowance this scan spent — `visited`, `envelopeDepth`. */ budget: AgentProjectScanBudget; + /** Syntax modules charged even on an LRU hit. */ + sourceBudget: AgentSourceScanBudget; + /** False when any source candidate or the shared source budget was incomplete. */ + sourceDiscoveryComplete: boolean; } /** @@ -109,11 +576,22 @@ export async function scanAgentProjects( root: string, budget: AgentProjectScanBudget = new AgentProjectScanBudget(), options: AgentProjectWalkOptions = {}, + sourceDiscovery: AgentSourceDiscovery = new AgentSourceDiscovery(), + sourceBudget: AgentSourceScanBudget = new AgentSourceScanBudget(), ): Promise { const absoluteRoot = path.resolve(root); const found: WorkflowInfo[] = []; const unreconciledRoots: string[] = []; + const notAgentRoots: string[] = []; + const opaqueRoots: string[] = []; + const discoveredStopRoots: string[] = []; + const sourceObservations: Array<{ + candidateRoot: string; + workspaceRoot: string; + paths: readonly string[]; + }> = []; const repositoryBoundaries: string[] = []; + let sourceDiscoveryComplete = true; const onDirectory = async (dir: string): Promise => { const safeDir = resolveWithinRoot(absoluteRoot, dir); @@ -125,13 +603,10 @@ export async function scanAgentProjects( found.push({ name: await nameFor(safeDir), path: safeDir, - definitionId: marker.definitionId ?? null, - definitionSlug: marker.name ?? null, - templateId: marker.templateId ?? null, - forkId: marker.forkId ?? null, - starterId: marker.starterId ?? null, + ...normalizedMarkerFields(marker), source: "scan", }); + discoveredStopRoots.push(safeDir); return "stop"; } if (markerResult.status === "unreadable") { @@ -148,6 +623,60 @@ export async function scanAgentProjects( absoluteRoot, { onDirectory, + onAdmittedDirectory: async (dir, _depth, entries) => { + for (const entry of entries) { + if (entry.isSymbolicLink()) { + opaqueRoots.push(path.join(dir, entry.name)); + } + } + const entry = entries.find( + (candidate) => candidate.name === AGENT_SOURCE_ENTRYPOINT, + ); + if (!entry) return "descend"; + const safeDir = resolveWithinRoot(absoluteRoot, dir); + if (!safeDir) return "stop"; + let result; + try { + result = await sourceDiscovery.inspectCandidate( + safeDir, + sourceBudget, + absoluteRoot, + ); + } catch { + sourceDiscoveryComplete = false; + unreconciledRoots.push(safeDir); + return "descend"; + } + sourceObservations.push({ + candidateRoot: safeDir, + workspaceRoot: absoluteRoot, + paths: result.watchPaths, + }); + if (result.status === "agent") { + found.push({ + name: await nameFor(safeDir), + path: safeDir, + definitionId: null, + definitionSlug: null, + sourceDefinitionName: safeDefinitionName(result.name), + activeBuildRunId: null, + activeBuildRunStatus: null, + templateId: null, + forkId: null, + starterId: null, + source: "scan", + }); + discoveredStopRoots.push(safeDir); + return "stop"; + } + if (result.status === "incomplete") { + sourceDiscoveryComplete = false; + unreconciledRoots.push(safeDir); + return "descend"; + } + if (result.status === "not-agent") notAgentRoots.push(safeDir); + return "descend"; + }, onUnreadable: (dir) => unreconciledRoots.push(dir), onRepositoryBoundary: (dir) => { repositoryBoundaries.push(dir); @@ -159,7 +688,19 @@ export async function scanAgentProjects( budget, options, ); - return { found, unreconciledRoots, repositoryBoundaries, budget }; + sourceDiscoveryComplete &&= !sourceBudget.truncated; + return { + found, + unreconciledRoots, + notAgentRoots, + opaqueRoots, + discoveredStopRoots, + sourceObservations, + repositoryBoundaries, + budget, + sourceBudget, + sourceDiscoveryComplete, + }; } /** @@ -181,7 +722,11 @@ function isCoveredByScan( ): boolean { const relative = path.relative(root, candidate); if (relative === "") return true; - if (relative.startsWith(`..${path.sep}`) || relative === ".." || path.isAbsolute(relative)) { + if ( + relative.startsWith(`..${path.sep}`) || + relative === ".." || + path.isAbsolute(relative) + ) { return false; } const segments = relative.split(path.sep).filter(Boolean); @@ -201,23 +746,34 @@ function isProtectedByIncompleteScan( ); } -/** - * Whether a previously known path sits inside a nested checkout this scan - * deliberately did not enter. - * - * Reconciliation's rule is "the registry may only forget a project it can prove - * it would have looked for", and the repository boundary changes what a scan - * looks for. Without this, opening `~/src` after having opened `~/src/some-repo` - * would DELETE that repo's agents: the scan stops at the boundary, finds no - * marker below it, and the depth envelope alone would call them gone. - */ -function isBehindRepositoryBoundary( - candidate: string, - repositoryBoundaries: string[], -): boolean { - return repositoryBoundaries.some( - (boundary) => resolveWithinRoot(boundary, candidate) !== null, - ); +function isStrictlyBelowRoot(root: string, candidate: string): boolean { + const confined = resolveWithinRoot(root, candidate); + return confined !== null && path.resolve(confined) !== path.resolve(root); +} + +/** Fresh, process-local proof controlling legacy source inspection. */ +export type WorkflowIdentityEvidence = + | "marker" + | "source" + | "not-agent" + | "unknown"; + +/** Registry-only syntax/marker evidence. Never serialize this row directly. */ +export interface RegistryWorkflowInfo extends PublicWorkflowInfo { + sourceDefinitionName?: string | null; + markerPresent?: true; +} + +type WorkflowInfo = RegistryWorkflowInfo; + +export interface WorkflowRegistryScanResult { + found: WorkflowInfo[]; + repositoryBoundaries: string[]; + budget: AgentProjectScanBudget; + sourceBudget: AgentSourceScanBudget; + status: "complete" | "degraded"; + changed: boolean; + generation: number; } /** @@ -254,39 +810,209 @@ async function partitionByPathExists( return { kept, pruned }; } +function removeEntriesWithinRoots( + entries: Map, + roots: readonly string[], +): void { + for (const entryRoot of entries.keys()) { + if (roots.some((root) => resolveWithinRoot(root, entryRoot) !== null)) { + entries.delete(entryRoot); + } + } +} + export class WorkflowRegistry { private workflows: WorkflowInfo[] = []; private loaded = false; + private loadPromise: Promise | null = null; /** Serializes mutations so concurrent prune/scan/connectPath calls can't * interleave and drop entries from the persisted file. Mirrors the pattern * used by SessionManager.persist() (session-manager.ts:278,851-853). */ private writeQueue: Promise = Promise.resolve(); /** Epoch ms of the last confirmed-missing sweep — see LAZY_PRUNE_INTERVAL_MS. */ private lastPruneAt = 0; + private readonly discoveryStatusByRoot = new Map< + string, + "complete" | "degraded" + >(); + private readonly canonicalScopeByLexicalRoot = new Map(); + private readonly canonicalWorkflowRootByPath = new Map(); + private readonly identityEvidenceByCanonicalRoot = new Map< + string, + WorkflowIdentityEvidence + >(); + private readonly sourceObservationsByCanonicalRoot = new Map< + string, + StoredSourceObservation + >(); + private inventoryGeneration = 0; + private discoveryEpoch = 0; + private discoveryLifetime = 0; + private retired = false; + private activeRename: Promise | null = null; + /** A failed compensating write leaves disk behind the accepted in-memory + * snapshot. The next otherwise-no-op scan must still repair the file. */ + private persistedSnapshotOutOfSync = false; + private readonly dirtyEpochByCanonicalRoot = new Map(); + + private cachedRootsForRows(rows: readonly WorkflowInfo[]): string[] { + return rows.map( + (workflow) => + this.canonicalWorkflowRootByPath.get(workflow.path) ?? + path.resolve(workflow.path), + ); + } + + private removePrivateStateForRows(rows: readonly WorkflowInfo[]): void { + if (rows.length === 0) return; + const removedRoots = this.cachedRootsForRows(rows); + for (const workflow of rows) { + this.canonicalWorkflowRootByPath.delete(workflow.path); + this.canonicalScopeByLexicalRoot.delete(path.resolve(workflow.path)); + } + removeEntriesWithinRoots( + this.identityEvidenceByCanonicalRoot, + removedRoots, + ); + for (const [key, observation] of this.sourceObservationsByCanonicalRoot) { + if ( + removedRoots.some( + (root) => resolveWithinRoot(root, observation.candidateRoot) !== null, + ) + ) { + this.sourceObservationsByCanonicalRoot.delete(key); + } + } + removeEntriesWithinRoots(this.discoveryStatusByRoot, removedRoots); + removeEntriesWithinRoots(this.dirtyEpochByCanonicalRoot, removedRoots); + } - constructor(private readonly registryPath: string = expandHome(HARNESS_PATHS.workflows)) {} + constructor( + private readonly registryPath: string = expandHome(HARNESS_PATHS.workflows), + private readonly sourceDiscovery: AgentSourceDiscovery = new AgentSourceDiscovery(), + private readonly persistenceTestHooks: { + afterPrimaryRename?: () => void | Promise; + } = {}, + ) {} private async ensureLoaded(): Promise { if (this.loaded) return; + if (!this.loadPromise) { + this.loadPromise = (async () => { + try { + const raw = await fs.readFile(this.registryPath, "utf8"); + const parsed = JSON.parse(raw) as unknown; + if (Array.isArray(parsed)) { + this.workflows = await normalizeLoadedWorkflows( + parsed as WorkflowInfo[], + ); + } else { + this.workflows = []; + this.persistedSnapshotOutOfSync = true; + } + } catch { + this.workflows = []; + // Preserve the historical durability contract: even an empty first + // scan creates a valid workflows.json. Treat a missing/corrupt file + // as disk being behind the accepted empty in-memory snapshot so the + // next scan repairs it despite rowsChanged=false. + this.persistedSnapshotOutOfSync = true; + } + const canonicalRoots = await mapBounded(this.workflows, (workflow) => + canonicalDirectory(workflow.path), + ); + this.canonicalWorkflowRootByPath.clear(); + for (const [index, workflow] of this.workflows.entries()) { + this.canonicalWorkflowRootByPath.set( + workflow.path, + canonicalRoots[index] ?? workflow.path, + ); + } + this.loaded = true; + })().finally(() => { + this.loadPromise = null; + }); + } + await this.loadPromise; + } + + private async persist( + workflows: readonly WorkflowInfo[] = this.workflows, + isCurrent?: () => boolean, + ): Promise { + const acceptedBeforeWrite = this.workflows; + await this.writeSnapshot(workflows, true, isCurrent); + const superseded = this.retired || (isCurrent && !isCurrent()); + if (!superseded) { + this.persistedSnapshotOutOfSync = false; + return; + } + + // The generation can change while rename(2) is in flight. At that point + // the proposed rows are durable but the caller must not commit them to + // memory. Restore the last accepted snapshot before returning the + // supersession error; otherwise a later no-op recovery scan would leave + // workflows.json containing rows that were never atomically published. + this.persistedSnapshotOutOfSync = true; try { - const raw = await fs.readFile(this.registryPath, "utf8"); - this.workflows = JSON.parse(raw) as WorkflowInfo[]; - } catch { - this.workflows = []; + await this.writeSnapshot(acceptedBeforeWrite, false); + this.persistedSnapshotOutOfSync = false; + } catch (rollbackError) { + const compensationError = new Error( + "Agent registry write was superseded and compensation failed", + ) as Error & { cause?: unknown }; + compensationError.cause = rollbackError; + throw compensationError; } - this.loaded = true; + throw new Error("Agent registry write was superseded"); } - private async persist(): Promise { + private async writeSnapshot( + workflows: readonly WorkflowInfo[], + primary: boolean, + isCurrent?: () => boolean, + ): Promise { const dir = path.dirname(this.registryPath); await fs.mkdir(dir, { recursive: true }); // Atomic write: write to a temp file in the same directory (so rename is // same-filesystem and thus atomic on POSIX), then rename over the target. // A crash mid-write leaves the .tmp file, not a torn workflows.json. // Mirrors the pattern used by SessionManager.persist(). - const tmpPath = `${this.registryPath}.tmp`; - await fs.writeFile(tmpPath, JSON.stringify(this.workflows, null, 2)); - await fs.rename(tmpPath, this.registryPath); + const tmpPath = `${this.registryPath}.tmp-${process.pid}-${randomUUID()}`; + try { + await fs.writeFile(tmpPath, JSON.stringify(workflows, null, 2)); + if (primary && (this.retired || (isCurrent && !isCurrent()))) { + throw new Error("Agent registry write was superseded"); + } + const rename = fs.rename(tmpPath, this.registryPath); + this.activeRename = rename; + try { + await rename; + } finally { + if (this.activeRename === rename) this.activeRename = null; + } + if (primary) { + await this.persistenceTestHooks.afterPrimaryRename?.(); + } + } finally { + await fs.rm(tmpPath, { force: true }); + } + } + + /** Retires discovery work owned by a shutting-down server instance. */ + async retirePendingDiscovery(): Promise { + this.retired = true; + this.discoveryLifetime += 1; + let timeout: ReturnType | undefined; + await Promise.race([ + this.writeQueue.catch(() => {}), + new Promise((resolve) => { + timeout = setTimeout(resolve, 1_000); + timeout.unref?.(); + }), + ]); + if (timeout) clearTimeout(timeout); + await this.activeRename?.catch(() => {}); } /** Chains `run` onto the write queue so concurrent mutations never @@ -300,6 +1026,154 @@ export class WorkflowRegistry { return next; } + private discoveryStatusForCanonicalRoot( + canonicalRoot: string, + ): "complete" | "degraded" { + // Completeness is evidence for exactly the selected scan scope. A direct + // child scan cannot upgrade or degrade a parent whose admitted envelope is + // different (and may deliberately skip that child behind an ignore or + // repository boundary). Active containing scopes are rescanned by the + // server coordinator when child evidence changes. + return this.discoveryStatusByRoot.get(canonicalRoot) ?? "degraded"; + } + + async discoveryStatus(root: string): Promise<"complete" | "degraded"> { + const lexicalRoot = path.resolve(root); + const canonicalRoot = + this.canonicalScopeByLexicalRoot.get(lexicalRoot) ?? lexicalRoot; + return this.discoveryStatusForCanonicalRoot(canonicalRoot); + } + + async inventorySnapshot(root: string): Promise<{ + workflows: readonly WorkflowInfo[]; + status: "complete" | "degraded"; + generation: number; + canonicalScopeRoot: string; + canonicalWorkflowRoots: readonly { + workflowPath: string; + canonicalRoot: string; + identityEvidence: WorkflowIdentityEvidence; + }[]; + sourceObservations: readonly { + candidateRoot: string; + workspaceRoot: string; + paths: readonly string[]; + }[]; + }> { + await this.ensureLoaded(); + const lexicalRoot = path.resolve(root); + const canonicalRoot = + this.canonicalScopeByLexicalRoot.get(lexicalRoot) ?? lexicalRoot; + return { + workflows: this.workflows, + status: this.discoveryStatusForCanonicalRoot(canonicalRoot), + generation: this.inventoryGeneration, + canonicalScopeRoot: canonicalRoot, + canonicalWorkflowRoots: this.workflows.map((workflow) => { + const canonicalWorkflowRoot = + this.canonicalWorkflowRootByPath.get(workflow.path) ?? workflow.path; + return { + workflowPath: workflow.path, + canonicalRoot: canonicalWorkflowRoot, + identityEvidence: + this.identityEvidenceByCanonicalRoot.get(canonicalWorkflowRoot) ?? + "unknown", + }; + }), + sourceObservations: [...this.sourceObservationsByCanonicalRoot.values()] + .map((observations) => ({ ...observations })) + .sort( + (left, right) => + compareText(left.workspaceRoot, right.workspaceRoot) || + compareText(left.candidateRoot, right.candidateRoot), + ), + }; + } + + private cachedCanonicalRoot(inputPath: string): string { + const lexicalPath = path.resolve(expandHome(inputPath)); + let bestLexicalRoot: string | null = null; + let bestCanonicalRoot: string | null = null; + const consider = (lexicalRoot: string, canonicalRoot: string): void => { + if (resolveWithinRoot(lexicalRoot, lexicalPath) === null) return; + if ( + bestLexicalRoot !== null && + bestLexicalRoot.length >= lexicalRoot.length + ) { + return; + } + bestLexicalRoot = lexicalRoot; + bestCanonicalRoot = canonicalRoot; + }; + for (const [lexicalRoot, canonicalRoot] of this + .canonicalScopeByLexicalRoot) { + consider(lexicalRoot, canonicalRoot); + } + for (const [lexicalRoot, canonicalRoot] of this + .canonicalWorkflowRootByPath) { + consider(lexicalRoot, canonicalRoot); + } + if (!bestLexicalRoot || !bestCanonicalRoot) return lexicalPath; + const suffix = path.relative(bestLexicalRoot, lexicalPath); + return path.resolve(bestCanonicalRoot, suffix); + } + + private wasDirtiedSince(canonicalRoot: string, epoch: number): boolean { + for (const [dirtyRoot, dirtyEpoch] of this.dirtyEpochByCanonicalRoot) { + if (dirtyEpoch <= epoch) continue; + if ( + resolveWithinRoot(canonicalRoot, dirtyRoot) !== null || + resolveWithinRoot(dirtyRoot, canonicalRoot) !== null + ) { + return true; + } + } + return false; + } + + /** + * Synchronous fail-closed invalidation for a raw watcher signal. The server + * calls this before any asynchronous reconciliation or graph work, so stale + * marker proof cannot authorize source inspection while an edit is pending. + * Registry load is completed during server boot; callers before then simply + * get the conservative no-op and the later scan establishes evidence. + */ + markDiscoveryDirty(root: string): boolean { + if (!this.loaded || this.retired) return false; + this.discoveryEpoch += 1; + const canonicalRoot = this.cachedCanonicalRoot(root); + this.dirtyEpochByCanonicalRoot.set(canonicalRoot, this.discoveryEpoch); + let changed = false; + const intersectingStatuses = new Set([canonicalRoot]); + for (const scannedRoot of this.discoveryStatusByRoot.keys()) { + if ( + resolveWithinRoot(canonicalRoot, scannedRoot) !== null || + resolveWithinRoot(scannedRoot, canonicalRoot) !== null + ) { + intersectingStatuses.add(scannedRoot); + } + } + for (const scannedRoot of intersectingStatuses) { + if (this.discoveryStatusByRoot.get(scannedRoot) !== "degraded") { + this.discoveryStatusByRoot.set(scannedRoot, "degraded"); + changed = true; + } + } + for (const [workflowRoot, evidence] of this + .identityEvidenceByCanonicalRoot) { + if ( + evidence !== "unknown" && + (resolveWithinRoot(canonicalRoot, workflowRoot) !== null || + resolveWithinRoot(workflowRoot, canonicalRoot) !== null) + ) { + this.identityEvidenceByCanonicalRoot.set(workflowRoot, "unknown"); + changed = true; + } + } + if (changed) this.inventoryGeneration += 1; + return changed; + } + /** * The registry as it stands — and the point at which stale entries actually * leave. A read runs {@link prune} when one is due @@ -316,7 +1190,9 @@ export class WorkflowRegistry { // throttle and queued its own full stat sweep of the whole registry. // Claiming it here closes the burst to one sweep per interval. this.lastPruneAt = Date.now(); - await this.prune(); + void this.prune().catch(() => { + // Reads stay cache-backed; a later read or explicit scan retries. + }); } return this.workflows; } @@ -341,17 +1217,33 @@ export class WorkflowRegistry { this.lastPruneAt = Date.now(); if (pruned.length === 0) return []; - const gone = new Set(pruned.map((workflow) => workflow.path)); + const gone = new Map(pruned.map((workflow) => [workflow.path, workflow])); return this.enqueue(async () => { + if (this.retired) return []; // RE-DERIVE FROM CURRENT STATE rather than assigning the `kept` computed // above: a scan may have registered entries while we were statting, and // writing a stale snapshot back would silently drop them. Removing a set // of known-missing paths is safe whatever else changed meanwhile. const before = this.workflows; - const removed = before.filter((workflow) => gone.has(workflow.path)); + const removed: WorkflowInfo[] = []; + for (const workflow of before) { + if (gone.get(workflow.path) !== workflow) continue; + try { + await fs.stat(workflow.path); + } catch (error) { + const code = (error as NodeJS.ErrnoException).code; + if (code === "ENOENT" || code === "ENOTDIR") removed.push(workflow); + } + } if (removed.length === 0) return []; - this.workflows = before.filter((workflow) => !gone.has(workflow.path)); - await this.persist(); + const removedRows = new Set(removed); + const nextWorkflows = before.filter( + (workflow) => !removedRows.has(workflow), + ); + await this.persist(nextWorkflows); + this.removePrivateStateForRows(removed); + this.workflows = nextWorkflows; + this.inventoryGeneration += 1; return removed; }); } @@ -368,82 +1260,630 @@ export class WorkflowRegistry { * cut it short; manually connected rows remain until their path itself is * pruned. */ - async scan( + async scanDetailed( root: string, budget: AgentProjectScanBudget = new AgentProjectScanBudget(), - ): Promise { + sourceBudget: AgentSourceScanBudget = new AgentSourceScanBudget(), + ): Promise { return this.enqueue(async () => { + if (this.retired) { + throw new Error("Agent registry is retired"); + } await this.ensureLoaded(); + const scanEpoch = this.discoveryEpoch; + const scanLifetime = this.discoveryLifetime; const absoluteRoot = path.resolve(expandHome(root)); - const { found, unreconciledRoots, repositoryBoundaries } = - await scanAgentProjects(absoluteRoot, budget); - - const foundPaths = new Set(found.map((workflow) => workflow.path)); - const byPath = new Map( - this.workflows - .filter( - (workflow) => - workflow.source !== "scan" || - !isCoveredByScan(absoluteRoot, workflow.path, budget.envelopeDepth) || - foundPaths.has(workflow.path) || - isProtectedByIncompleteScan(workflow.path, unreconciledRoots) || - isBehindRepositoryBoundary(workflow.path, repositoryBoundaries), - ) - .map((workflow) => [workflow.path, workflow]), + const rootEvidence = await canonicalDirectoryEvidence(absoluteRoot); + const canonicalRoot = rootEvidence.key; + this.canonicalScopeByLexicalRoot.set(absoluteRoot, canonicalRoot); + this.canonicalScopeByLexicalRoot.set(canonicalRoot, canonicalRoot); + const scanResult = await scanAgentProjects( + absoluteRoot, + budget, + {}, + this.sourceDiscovery, + sourceBudget, + ); + const { + found, + unreconciledRoots, + notAgentRoots, + opaqueRoots, + discoveredStopRoots, + sourceObservations, + repositoryBoundaries, + sourceDiscoveryComplete, + } = scanResult; + if (rootEvidence.status === "unreadable") { + unreconciledRoots.push(absoluteRoot); + } + let nextStatus: "complete" | "degraded" = + !budget.truncated && + unreconciledRoots.length === 0 && + sourceDiscoveryComplete + ? "complete" + : "degraded"; + + const foundRows = await mapBounded(found, async (workflow) => { + const canonicalEvidence = await canonicalDirectoryEvidence( + workflow.path, + ); + return { + workflow, + canonicalKey: canonicalEvidence.key, + canonicalStatus: canonicalEvidence.status, + }; + }); + if (foundRows.some((row) => row.canonicalStatus === "unreadable")) { + nextStatus = "degraded"; + } + const foundKeys = new Set(foundRows.map((row) => row.canonicalKey)); + const protectedRoots = [ + ...unreconciledRoots, + ...opaqueRoots, + ...discoveredStopRoots, + ...repositoryBoundaries, + ]; + const canonicalizableProtectedRoots = [ + ...unreconciledRoots, + ...discoveredStopRoots, + ...repositoryBoundaries, + ]; + const canonicalProtectedRoots = await mapBounded( + canonicalizableProtectedRoots, + canonicalDirectory, + ); + const canonicalUnreconciledRoots = await mapBounded( + unreconciledRoots, + canonicalDirectory, ); - for (const workflow of found) { - const existing = byPath.get(workflow.path); - // A manually-connected entry keeps its `source`; a scan only refreshes name/definitionId. - byPath.set(workflow.path, existing ? { ...existing, ...workflow, source: existing.source } : workflow); + const canonicalNotAgentRoots = await mapBounded( + notAgentRoots, + canonicalDirectory, + ); + const canonicalRepositoryBoundaries = await mapBounded( + repositoryBoundaries, + canonicalDirectory, + ); + const canonicalDiscoveredStopRoots = await mapBounded( + discoveredStopRoots, + canonicalDirectory, + ); + const canonicalSourceObservations = await mapBounded( + sourceObservations, + async (observation) => { + const canonicalWorkspaceRoot = await canonicalDirectory( + observation.workspaceRoot, + ); + return { + candidateRoot: await canonicalDirectory(observation.candidateRoot), + workspaceRoot: canonicalWorkspaceRoot, + // Analyzer watch paths are lexical and may sit below a symlinked + // selected scope. Preserve their already-confined relative suffix + // while projecting them into the canonical envelope consumed by + // polling; do not follow an absent target to perform this remap. + paths: observation.paths.flatMap((observedPath) => { + for (const envelope of [ + observation.workspaceRoot, + canonicalWorkspaceRoot, + ]) { + const relative = path.relative(envelope, observedPath); + if ( + relative !== ".." && + !relative.startsWith(`..${path.sep}`) && + !path.isAbsolute(relative) + ) { + return [path.resolve(canonicalWorkspaceRoot, relative)]; + } + } + return []; + }), + }; + }, + ); + const currentRows = await mapBounded( + this.workflows, + async (workflow) => ({ + workflow, + canonicalEvidence: await canonicalDirectoryEvidence(workflow.path), + }), + ); + const byPath = new Map(); + for (const { workflow, canonicalEvidence } of currentRows) { + const canonicalKey = canonicalEvidence.key; + const newlyForeignToOwningEnvelope = + this.sourceObservationsByCanonicalRoot.has( + sourceObservationKey(canonicalRoot, canonicalKey), + ) && + canonicalRepositoryBoundaries.some( + (boundary) => path.resolve(boundary) === path.resolve(canonicalKey), + ); + const lexicalCovered = isCoveredByScan( + absoluteRoot, + workflow.path, + budget.envelopeDepth, + ); + const canonicalCovered = isCoveredByScan( + canonicalRoot, + canonicalKey, + budget.envelopeDepth, + ); + const covered = lexicalCovered || canonicalCovered; + const lexicalProtected = isProtectedByIncompleteScan( + workflow.path, + protectedRoots, + ); + const protectedByBoundary = + canonicalEvidence.status === "unreadable" || + (lexicalProtected && !canonicalCovered) || + (isProtectedByIncompleteScan(canonicalKey, canonicalProtectedRoots) && + !newlyForeignToOwningEnvelope); + if ( + workflow.source === "connect" && + covered && + !foundKeys.has(canonicalKey) && + !protectedByBoundary + ) { + const withoutStaleSource = { ...workflow }; + delete withoutStaleSource.sourceDefinitionName; + delete withoutStaleSource.markerPresent; + byPath.set(canonicalKey, withoutStaleSource); + continue; + } + if ( + workflow.source !== "scan" || + !covered || + foundKeys.has(canonicalKey) || + protectedByBoundary + ) { + byPath.set(canonicalKey, workflow); + } + } + for (const { workflow, canonicalKey } of foundRows) { + const existing = byPath.get(canonicalKey); + byPath.set(canonicalKey, mergeScannedWorkflow(existing, workflow)); } // Registry-wide, not envelope-wide: a scan is the most frequent write // this file gets, and an entry whose directory is CONFIRMED gone has no // claim to survive it regardless of which root turned it up. Without // this, a dead row rooted somewhere the studio never scans again lives // for as long as the install does. - const { kept, pruned } = await partitionByPathExists( + const { kept, pruned: initiallyPruned } = await partitionByPathExists( Array.from(byPath.values()), ); - this.lastPruneAt = Date.now(); - this.workflows = kept; - await this.persist(); - return found.filter( - (workflow) => !pruned.some((dead) => dead.path === workflow.path), + // The filesystem can resurrect a same-path project while a wide sweep is + // in progress. Re-stat removal candidates at the commit boundary and + // preserve any path whose evidence changed after the first observation. + const { kept: resurrected, pruned } = + await partitionByPathExists(initiallyPruned); + const nextWorkflows = [...kept, ...resurrected].sort((left, right) => + compareText(left.path, right.path), + ); + const nextCanonicalRoots = await mapBounded(nextWorkflows, (workflow) => + canonicalDirectory(workflow.path), + ); + const nextCanonicalWorkflowRootByPath = new Map(); + for (const [index, workflow] of nextWorkflows.entries()) { + nextCanonicalWorkflowRootByPath.set( + workflow.path, + nextCanonicalRoots[index] ?? workflow.path, + ); + } + const nextIdentityEvidence = new Map( + this.identityEvidenceByCanonicalRoot, + ); + const nextSourceObservations = new Map( + this.sourceObservationsByCanonicalRoot, + ); + // A registry-wide existence sweep can retire a row outside the requested + // scan envelope. Its source-observation sidecar must retire with it or + // every later watcher will keep probing a path proven not to exist. Do + // not trim ordinary not-agent/incomplete candidates here: their missing + // dependency observations are what let a later dependency-only edit + // promote them without touching index.ts. + const prunedCanonicalRoots = this.cachedRootsForRows(pruned); + const observedCandidatesThisScan = new Set( + canonicalSourceObservations.map((entry) => entry.candidateRoot), + ); + for (const [observationKey, observation] of nextSourceObservations) { + const candidateRoot = observation.candidateRoot; + if ( + prunedCanonicalRoots.some( + (root) => resolveWithinRoot(root, candidateRoot) !== null, + ) + ) { + nextSourceObservations.delete(observationKey); + continue; + } + const covered = isCoveredByScan( + canonicalRoot, + candidateRoot, + budget.envelopeDepth, + ); + const protectedByBoundary = + (isProtectedByIncompleteScan(candidateRoot, [ + ...canonicalUnreconciledRoots, + ...canonicalRepositoryBoundaries, + ]) && + !( + observation.workspaceRoot === canonicalRoot && + canonicalRepositoryBoundaries.some( + (boundary) => + path.resolve(boundary) === path.resolve(candidateRoot), + ) + )) || + canonicalDiscoveredStopRoots.some((stopRoot) => + isStrictlyBelowRoot(stopRoot, candidateRoot), + ); + if ( + covered && + !protectedByBoundary && + (observation.workspaceRoot === canonicalRoot || + !observedCandidatesThisScan.has(candidateRoot)) + ) { + nextSourceObservations.delete(observationKey); + } + } + for (const observation of canonicalSourceObservations) { + nextSourceObservations.set( + sourceObservationKey( + observation.workspaceRoot, + observation.candidateRoot, + ), + { + candidateRoot: observation.candidateRoot, + workspaceRoot: observation.workspaceRoot, + paths: observation.paths, + }, + ); + } + for (const { workflow, canonicalEvidence } of currentRows) { + const canonicalKey = canonicalEvidence.key; + const covered = + isCoveredByScan(absoluteRoot, workflow.path, budget.envelopeDepth) || + isCoveredByScan(canonicalRoot, canonicalKey, budget.envelopeDepth); + const protectedByBoundary = + canonicalEvidence.status === "unreadable" || + (isProtectedByIncompleteScan(canonicalKey, canonicalProtectedRoots) && + !( + this.sourceObservationsByCanonicalRoot.has( + sourceObservationKey(canonicalRoot, canonicalKey), + ) && + canonicalRepositoryBoundaries.some( + (boundary) => + path.resolve(boundary) === path.resolve(canonicalKey), + ) + )); + const intersectsUncertainty = canonicalUnreconciledRoots.some( + (unreconciledRoot) => + resolveWithinRoot(unreconciledRoot, canonicalKey) !== null || + resolveWithinRoot(canonicalKey, unreconciledRoot) !== null, + ); + if (intersectsUncertainty || (covered && !protectedByBoundary)) { + nextIdentityEvidence.set(canonicalKey, "unknown"); + } + } + for (const { workflow, canonicalKey } of foundRows) { + nextIdentityEvidence.set( + canonicalKey, + hasSourceDiscoveryEvidence(workflow) ? "source" : "marker", + ); + } + for (const canonicalKey of canonicalNotAgentRoots) { + nextIdentityEvidence.set(canonicalKey, "not-agent"); + } + const retainedCanonicalRoots = new Set(nextCanonicalRoots); + for (const canonicalKey of nextIdentityEvidence.keys()) { + if (!retainedCanonicalRoots.has(canonicalKey)) { + nextIdentityEvidence.delete(canonicalKey); + } + } + const nextStatuses = new Map(this.discoveryStatusByRoot); + removeEntriesWithinRoots(nextStatuses, prunedCanonicalRoots); + for (const [scannedRoot, priorStatus] of nextStatuses) { + if ( + scannedRoot === canonicalRoot || + resolveWithinRoot(canonicalRoot, scannedRoot) === null + ) { + continue; + } + const intersectsUncertainty = canonicalUnreconciledRoots.some( + (unreconciledRoot) => + resolveWithinRoot(unreconciledRoot, scannedRoot) !== null || + resolveWithinRoot(scannedRoot, unreconciledRoot) !== null, + ); + if (intersectsUncertainty) { + nextStatuses.set(scannedRoot, "degraded"); + continue; + } + const admittedAndCovered = + nextStatus === "complete" && + isCoveredByScan(canonicalRoot, scannedRoot, budget.envelopeDepth) && + !isProtectedByIncompleteScan( + scannedRoot, + canonicalRepositoryBoundaries, + ) && + !canonicalDiscoveredStopRoots.some((stopRoot) => + isStrictlyBelowRoot(stopRoot, scannedRoot), + ); + if (priorStatus === "degraded" && admittedAndCovered) { + nextStatuses.set(scannedRoot, "complete"); + } + } + nextStatuses.set(canonicalRoot, nextStatus); + const rowsChanged = !workflowRowsEqual(this.workflows, nextWorkflows); + const statusChanged = !statusMapsEqual( + this.discoveryStatusByRoot, + nextStatuses, + ); + const evidenceChanged = !evidenceMapsEqual( + this.identityEvidenceByCanonicalRoot, + nextIdentityEvidence, + ); + const observationsChanged = !observationMapsEqual( + this.sourceObservationsByCanonicalRoot, + nextSourceObservations, ); + this.lastPruneAt = Date.now(); + if ( + scanLifetime !== this.discoveryLifetime || + this.wasDirtiedSince(canonicalRoot, scanEpoch) + ) { + throw new Error("Agent discovery scan was superseded by a newer edit"); + } + try { + if (rowsChanged || this.persistedSnapshotOutOfSync) { + await this.persist( + nextWorkflows, + () => + scanLifetime === this.discoveryLifetime && + !this.wasDirtiedSince(canonicalRoot, scanEpoch), + ); + } + } catch (error) { + if ( + scanLifetime === this.discoveryLifetime && + this.discoveryStatusByRoot.get(canonicalRoot) !== "degraded" + ) { + this.discoveryStatusByRoot.set(canonicalRoot, "degraded"); + this.inventoryGeneration += 1; + } + throw error; + } + if ( + scanLifetime !== this.discoveryLifetime || + this.wasDirtiedSince(canonicalRoot, scanEpoch) + ) { + throw new Error("Agent discovery scan was superseded by a newer edit"); + } + this.removePrivateStateForRows(pruned); + if (rowsChanged) this.workflows = nextWorkflows; + this.canonicalWorkflowRootByPath.clear(); + for (const [ + workflowPath, + canonicalWorkflowRoot, + ] of nextCanonicalWorkflowRootByPath) { + this.canonicalWorkflowRootByPath.set( + workflowPath, + canonicalWorkflowRoot, + ); + } + if (statusChanged) { + this.discoveryStatusByRoot.clear(); + for (const [scannedRoot, status] of nextStatuses) { + this.discoveryStatusByRoot.set(scannedRoot, status); + } + } + if (evidenceChanged) { + this.identityEvidenceByCanonicalRoot.clear(); + for (const [canonicalKey, evidence] of nextIdentityEvidence) { + this.identityEvidenceByCanonicalRoot.set(canonicalKey, evidence); + } + } + if (observationsChanged) { + this.sourceObservationsByCanonicalRoot.clear(); + for (const [canonicalKey, observations] of nextSourceObservations) { + this.sourceObservationsByCanonicalRoot.set( + canonicalKey, + observations, + ); + } + } + if ( + rowsChanged || + statusChanged || + evidenceChanged || + observationsChanged + ) { + this.inventoryGeneration += 1; + } + return { + found: found.filter( + (workflow) => !pruned.some((dead) => dead.path === workflow.path), + ), + repositoryBoundaries, + budget, + sourceBudget, + status: this.discoveryStatusForCanonicalRoot(canonicalRoot), + changed: + rowsChanged || + statusChanged || + evidenceChanged || + observationsChanged, + generation: this.inventoryGeneration, + }; }); } + async scan( + root: string, + budget: AgentProjectScanBudget = new AgentProjectScanBudget(), + ): Promise { + return (await this.scanDetailed(root, budget)).found; + } + /** Registers an arbitrary path (the "+ Connect" flow); marker is optional at connect time. */ async connectPath(inputPath: string): Promise { return this.enqueue(async () => { + if (this.retired) { + throw new Error("Agent registry is retired"); + } await this.ensureLoaded(); + const connectEpoch = this.discoveryEpoch; + const connectLifetime = this.discoveryLifetime; const absolutePath = path.resolve(expandHome(inputPath)); - const marker = await readMarker(absolutePath); + const canonicalKey = await canonicalDirectory(absolutePath); + const markerInspection = await inspectMarker(absolutePath); + const marker = + markerInspection.status === "valid" ? markerInspection.marker : null; + let sourceInspection: Awaited< + ReturnType + > | null = null; + if ( + markerInspection.status === "absent" || + markerInspection.status === "invalid" + ) { + try { + sourceInspection = await this.sourceDiscovery.inspectCandidate( + absolutePath, + new AgentSourceScanBudget(), + absolutePath, + ); + } catch { + sourceInspection = { + status: "incomplete", + reason: "unreadable-source", + modules: 0, + bytes: 0, + lookups: 0, + fingerprint: "", + observations: [], + watchPaths: [], + }; + } + } const info: WorkflowInfo = { name: await nameFor(absolutePath), path: absolutePath, - definitionId: marker?.definitionId ?? null, - definitionSlug: marker?.name ?? null, - templateId: marker?.templateId ?? null, - forkId: marker?.forkId ?? null, - starterId: marker?.starterId ?? null, + ...(marker + ? normalizedMarkerFields(marker) + : { + definitionId: null, + definitionSlug: null, + templateId: null, + forkId: null, + starterId: null, + }), source: "connect", }; - // Match on the resolved directory, not the spelling given: connecting a - // symlinked path to an already-scanned project otherwise registers a - // second row for one directory, and the pair collides into `local:` - // fallback keys that make every reference between agents ambiguous. - // The existing row keeps its own `path` — registry paths are compared by - // exact string elsewhere (a session auto-binds on `path === cwd`), so - // rewriting one silently unbinds whatever matched it. - const canonical = canonicalGraphPath(absolutePath); - const idx = this.workflows.findIndex( - (workflow) => canonicalGraphPath(workflow.path) === canonical, + if (sourceInspection?.status === "agent") { + info.sourceDefinitionName = safeDefinitionName(sourceInspection.name); + } + const canonicalKeys = await mapBounded(this.workflows, (workflow) => + canonicalDirectory(workflow.path), ); - if (idx >= 0) this.workflows[idx] = { ...info, path: this.workflows[idx]!.path }; - else this.workflows.push(info); - await this.persist(); - return info; + const idx = canonicalKeys.findIndex((key) => key === canonicalKey); + let persisted = info; + const nextWorkflows = [...this.workflows]; + if (idx >= 0) { + const existing = this.workflows[idx]!; + if (markerInspection.status === "unreadable") { + persisted = { ...existing, source: "connect" }; + } else { + persisted = { + ...existing, + ...info, + // Keep the registry spelling that existing sessions and rail rows + // already reference. Canonical matching prevents a duplicate row; + // rewriting the path would silently unbind exact-path consumers. + path: existing.path, + definitionId: existing.definitionId ?? info.definitionId, + definitionSlug: existing.definitionSlug ?? info.definitionSlug, + templateId: existing.templateId ?? info.templateId, + forkId: existing.forkId ?? info.forkId, + starterId: existing.starterId ?? info.starterId, + }; + } + if (markerInspection.status === "unreadable") { + // Opaque marker evidence is retained exactly until a later accepted + // marker/source inspection can replace it. + } else if (marker) { + delete persisted.sourceDefinitionName; + } else if (sourceInspection?.status === "agent") { + persisted.sourceDefinitionName = safeDefinitionName( + sourceInspection.name, + ); + } else if (sourceInspection?.status === "incomplete") { + if (hasSourceDiscoveryEvidence(existing)) { + persisted.sourceDefinitionName = + existing.sourceDefinitionName ?? null; + } else { + delete persisted.sourceDefinitionName; + } + } else { + delete persisted.sourceDefinitionName; + } + if (!marker && markerInspection.status !== "unreadable") { + delete persisted.markerPresent; + } + nextWorkflows[idx] = persisted; + } else { + nextWorkflows.push(info); + } + const superseded = this.wasDirtiedSince(canonicalKey, connectEpoch); + if (superseded) { + delete persisted.markerPresent; + delete persisted.sourceDefinitionName; + const persistedIndex = nextWorkflows.findIndex( + (workflow) => workflow.path === persisted.path, + ); + if (persistedIndex >= 0) nextWorkflows[persistedIndex] = persisted; + } + await this.persist( + nextWorkflows, + () => + connectLifetime === this.discoveryLifetime && + !this.wasDirtiedSince(canonicalKey, connectEpoch), + ); + this.workflows = nextWorkflows; + this.canonicalWorkflowRootByPath.set(persisted.path, canonicalKey); + const supersededAtPublication = + superseded || this.wasDirtiedSince(canonicalKey, connectEpoch); + if (supersededAtPublication) { + this.identityEvidenceByCanonicalRoot.set(canonicalKey, "unknown"); + } else if (markerInspection.status === "unreadable") { + this.identityEvidenceByCanonicalRoot.set(canonicalKey, "unknown"); + } else { + this.identityEvidenceByCanonicalRoot.set( + canonicalKey, + marker + ? "marker" + : sourceInspection?.status === "agent" + ? "source" + : sourceInspection?.status === "not-agent" + ? "not-agent" + : "unknown", + ); + if (marker) { + for (const [key, observation] of this + .sourceObservationsByCanonicalRoot) { + if (observation.candidateRoot === canonicalKey) { + this.sourceObservationsByCanonicalRoot.delete(key); + } + } + } else if (sourceInspection) { + this.sourceObservationsByCanonicalRoot.set( + sourceObservationKey(canonicalKey, canonicalKey), + { + candidateRoot: canonicalKey, + workspaceRoot: canonicalKey, + paths: sourceInspection.watchPaths, + }, + ); + } + } + this.inventoryGeneration += 1; + return persisted; }); } } @@ -483,11 +1923,30 @@ export interface WorkflowRegistryLike { connectPath(inputPath: string): Promise; } -export function createWorkflowsRouter(registry: WorkflowRegistryLike): ExpressRouter { +/** + * Keep registry-only proof out of every HTTP adapter, including embedders that + * mount this generic router directly around a {@link WorkflowRegistry}. + */ +function publicWorkflowInfo(workflow: WorkflowInfo): PublicWorkflowInfo { + const publicRow = { ...workflow }; + delete publicRow.sourceDefinitionName; + delete publicRow.markerPresent; + return publicRow; +} + +function publicWorkflowInfos( + workflows: readonly WorkflowInfo[], +): PublicWorkflowInfo[] { + return workflows.map(publicWorkflowInfo); +} + +export function createWorkflowsRouter( + registry: WorkflowRegistryLike, +): ExpressRouter { const router = Router(); router.get("/api/workflows", async (_req, res) => { - res.json(await registry.list()); + res.json(publicWorkflowInfos(await registry.list())); }); router.post("/api/workflows/connect", async (req, res) => { @@ -497,7 +1956,7 @@ export function createWorkflowsRouter(registry: WorkflowRegistryLike): ExpressRo return; } try { - res.json(await registry.connectPath(inputPath)); + res.json(publicWorkflowInfo(await registry.connectPath(inputPath))); } catch (err) { res.status(500).json({ error: (err as Error).message }); } @@ -515,7 +1974,10 @@ export function createWorkflowsRouter(registry: WorkflowRegistryLike): ExpressRo const outcome: WorkflowScanOutcome = registry.scanWithBoundaries ? await registry.scanWithBoundaries(root) : { found: await registry.scan(root), repositoryBoundaries: [] }; - res.json(outcome); + res.json({ + ...outcome, + found: publicWorkflowInfos(outcome.found), + }); } catch (err) { res.status(500).json({ error: (err as Error).message }); } diff --git a/packages/harness/src/core/workspace-context.test.ts b/packages/harness/src/core/workspace-context.test.ts index 913afc075..336435b20 100644 --- a/packages/harness/src/core/workspace-context.test.ts +++ b/packages/harness/src/core/workspace-context.test.ts @@ -7,6 +7,7 @@ import { afterEach, beforeEach, describe, expect, it } from "vitest"; import type { WorkflowInfo } from "../shared/types.js"; import { prepareHarnessContextForResume, + stageHarnessContextForPublication, writeHarnessContext, writeHarnessContextForLaunch, type WorkspaceContextSession, @@ -42,7 +43,10 @@ describe("writeHarnessContext", () => { }); async function readContext(): Promise { - const raw = await fs.readFile(path.join(cwd, ".sapiom", "harness-context.json"), "utf8"); + const raw = await fs.readFile( + path.join(cwd, ".sapiom", "harness-context.json"), + "utf8", + ); return JSON.parse(raw); } @@ -50,7 +54,11 @@ describe("writeHarnessContext", () => { await writeHarnessContext(session, workflow, [workflow]); const context = await readContext(); expect(context).toMatchObject({ - boundAgent: { name: "leasing", path: "/Users/demo/acme-app/leasing", definitionId: 4821 }, + boundAgent: { + name: "leasing", + path: "/Users/demo/acme-app/leasing", + definitionId: 4821, + }, }); expect(context).not.toHaveProperty("boundWorkflow"); expect(context).not.toHaveProperty("workflows"); @@ -69,7 +77,9 @@ describe("writeHarnessContext", () => { const context = await readContext(); expect(context).toMatchObject({ boundAgent: null }); // The file must still exist (only its content changed). - await expect(fs.access(path.join(cwd, ".sapiom", "harness-context.json"))).resolves.toBeUndefined(); + await expect( + fs.access(path.join(cwd, ".sapiom", "harness-context.json")), + ).resolves.toBeUndefined(); }); it("overwrites cleanly on repeated binds and leaves no leftover tmp files", async () => { @@ -77,12 +87,47 @@ describe("writeHarnessContext", () => { const renamed = { ...workflow, name: "renamed", definitionId: 9999 }; await writeHarnessContext(session, renamed, [renamed]); const context = await readContext(); - expect(context).toMatchObject({ boundAgent: { name: "renamed", definitionId: 9999 } }); + expect(context).toMatchObject({ + boundAgent: { name: "renamed", definitionId: 9999 }, + }); const entries = await fs.readdir(path.join(cwd, ".sapiom")); expect(entries).toEqual(["harness-context.json"]); }); + it("keeps staged publication invisible until synchronous commit and discards superseded stages", async () => { + await writeHarnessContext(session, workflow, [workflow]); + const staged = await stageHarnessContextForPublication( + session, + otherWorkflow, + [otherWorkflow], + () => true, + ); + expect(staged).not.toBeNull(); + expect((await readContext()) as object).toMatchObject({ + boundAgent: { name: "leasing" }, + }); + + staged!.commit(); + expect((await readContext()) as object).toMatchObject({ + boundAgent: { name: "billing" }, + }); + + const discarded = await stageHarnessContextForPublication( + session, + workflow, + [workflow], + () => true, + ); + discarded!.discard(); + expect((await readContext()) as object).toMatchObject({ + boundAgent: { name: "billing" }, + }); + expect(await fs.readdir(path.join(cwd, ".sapiom"))).toEqual([ + "harness-context.json", + ]); + }); + it("does not throw when the cwd is unwritable (logs and returns)", async () => { // mkdir with recursive:true inside writeHarnessContext will actually // succeed here (it creates missing dirs) — use a path that collides with @@ -90,7 +135,9 @@ describe("writeHarnessContext", () => { const blockedFile = path.join(cwd, "blocked"); await fs.writeFile(blockedFile, "x"); await expect( - writeHarnessContext({ ...session, cwd: blockedFile }, workflow, [workflow]), + writeHarnessContext({ ...session, cwd: blockedFile }, workflow, [ + workflow, + ]), ).resolves.toBeUndefined(); }); @@ -99,15 +146,25 @@ describe("writeHarnessContext", () => { await fs.writeFile(blockedFile, "x"); await expect( - writeHarnessContextForLaunch({ ...session, cwd: blockedFile }, workflow, [workflow]), + writeHarnessContextForLaunch({ ...session, cwd: blockedFile }, workflow, [ + workflow, + ]), ).rejects.toThrow(); }); it("writes the full agents registry, trimmed to {name, path, definitionId} (no source)", async () => { await writeHarnessContext(session, null, [workflow, otherWorkflow]); const context = (await readContext()) as { agents: unknown[] }; - expect(context.agents).toContainEqual({ name: "leasing", path: workflow.path, definitionId: 4821 }); - expect(context.agents).toContainEqual({ name: "billing", path: otherWorkflow.path, definitionId: 4822 }); + expect(context.agents).toContainEqual({ + name: "leasing", + path: workflow.path, + definitionId: 4821, + }); + expect(context.agents).toContainEqual({ + name: "billing", + path: otherWorkflow.path, + definitionId: 4822, + }); for (const entry of context.agents) { expect(entry).not.toHaveProperty("source"); } @@ -120,27 +177,42 @@ describe("writeHarnessContext", () => { await writeHarnessContext(session, null, [otherWorkflow, workflow]); // billing, then leasing const second = (await readContext()) as { agents: Array<{ path: string }> }; - expect(first.agents.map((w) => w.path)).toEqual(second.agents.map((w) => w.path)); - expect(first.agents.map((w) => w.path)).toEqual([otherWorkflow.path, workflow.path].sort()); + expect(first.agents.map((w) => w.path)).toEqual( + second.agents.map((w) => w.path), + ); + expect(first.agents.map((w) => w.path)).toEqual( + [otherWorkflow.path, workflow.path].sort(), + ); }); it("includes agents even when none of them is the bound one", async () => { await writeHarnessContext(session, workflow, [workflow, otherWorkflow]); - const context = (await readContext()) as { boundAgent: { path: string }; agents: Array<{ path: string }> }; + const context = (await readContext()) as { + boundAgent: { path: string }; + agents: Array<{ path: string }>; + }; expect(context.boundAgent?.path).toBe(workflow.path); expect(context.agents.map((w) => w.path)).toContain(otherWorkflow.path); }); it("embeds the session's own identity", async () => { await writeHarnessContext(session, null, []); - const context = (await readContext()) as { session: { id: string; cwd: string; harness: string } }; - expect(context.session).toEqual({ id: "sess-1", cwd, harness: "claude-code" }); + const context = (await readContext()) as { + session: { id: string; cwd: string; harness: string }; + }; + expect(context.session).toEqual({ + id: "sess-1", + cwd, + harness: "claude-code", + }); }); it("creates .sapiom/ from scratch for a cwd that has never had any file written to it", async () => { await expect(fs.access(path.join(cwd, ".sapiom"))).rejects.toThrow(); await writeHarnessContext(session, null, []); - await expect(fs.access(path.join(cwd, ".sapiom", "harness-context.json"))).resolves.toBeUndefined(); + await expect( + fs.access(path.join(cwd, ".sapiom", "harness-context.json")), + ).resolves.toBeUndefined(); }); it("handles a burst of concurrent writes to the same destination: no rejections, valid JSON, no leftover tmp files, last call wins", async () => { @@ -150,13 +222,17 @@ describe("writeHarnessContext", () => { // tmp filename within the same millisecond and steal each other's tmp // file out from under a pending rename. const writes = Array.from({ length: 10 }, (_, i) => - writeHarnessContext(session, { ...workflow, definitionId: i }, [workflow]), + writeHarnessContext(session, { ...workflow, definitionId: i }, [ + workflow, + ]), ); const results = await Promise.allSettled(writes); expect(results.every((r) => r.status === "fulfilled")).toBe(true); - const context = (await readContext()) as { boundAgent: { definitionId: number } }; + const context = (await readContext()) as { + boundAgent: { definitionId: number }; + }; // Serialized in call order, not completion order: the last call // enqueued must be the one that's actually on disk at the end, // regardless of which write's disk I/O happened to finish first. @@ -167,20 +243,38 @@ describe("writeHarnessContext", () => { }); it("interleaves concurrent writes to two different destinations independently", async () => { - const otherCwd = await fs.mkdtemp(path.join(os.tmpdir(), "harness-context-test-other-")); - const otherSession: WorkspaceContextSession = { id: "sess-2", cwd: otherCwd, harness: "codex" }; + const otherCwd = await fs.mkdtemp( + path.join(os.tmpdir(), "harness-context-test-other-"), + ); + const otherSession: WorkspaceContextSession = { + id: "sess-2", + cwd: otherCwd, + harness: "codex", + }; try { await Promise.all([ - ...Array.from({ length: 5 }, (_, i) => writeHarnessContext(session, { ...workflow, definitionId: i }, [])), ...Array.from({ length: 5 }, (_, i) => - writeHarnessContext(otherSession, { ...otherWorkflow, definitionId: i + 100 }, []), + writeHarnessContext(session, { ...workflow, definitionId: i }, []), + ), + ...Array.from({ length: 5 }, (_, i) => + writeHarnessContext( + otherSession, + { ...otherWorkflow, definitionId: i + 100 }, + [], + ), ), ]); - const mine = (await readContext()) as { boundAgent: { definitionId: number }; session: { id: string } }; + const mine = (await readContext()) as { + boundAgent: { definitionId: number }; + session: { id: string }; + }; const theirs = JSON.parse( - await fs.readFile(path.join(otherCwd, ".sapiom", "harness-context.json"), "utf8"), + await fs.readFile( + path.join(otherCwd, ".sapiom", "harness-context.json"), + "utf8", + ), ) as { boundAgent: { definitionId: number }; session: { id: string } }; expect(mine.session.id).toBe("sess-1"); @@ -199,7 +293,9 @@ describe("prepareHarnessContextForResume", () => { let contextPath: string; beforeEach(async () => { - cwd = await fs.mkdtemp(path.join(os.tmpdir(), "harness-context-resume-test-")); + cwd = await fs.mkdtemp( + path.join(os.tmpdir(), "harness-context-resume-test-"), + ); session = { id: "current-session", cwd, harness: "claude-code" }; contextPath = path.join(cwd, ".sapiom", "harness-context.json"); }); @@ -214,7 +310,10 @@ describe("prepareHarnessContextForResume", () => { } async function readParsed(): Promise> { - return JSON.parse(await fs.readFile(contextPath, "utf8")) as Record; + return JSON.parse(await fs.readFile(contextPath, "utf8")) as Record< + string, + unknown + >; } it("leaves an already-current schema byte-for-byte untouched", async () => { @@ -226,7 +325,9 @@ describe("prepareHarnessContextForResume", () => { }); await writeRaw(raw); - await expect(prepareHarnessContextForResume(session, workflow, [workflow])).resolves.toBe("current"); + await expect( + prepareHarnessContextForResume(session, workflow, [workflow]), + ).resolves.toBe("current"); await expect(fs.readFile(contextPath, "utf8")).resolves.toBe(raw); }); @@ -239,7 +340,9 @@ describe("prepareHarnessContextForResume", () => { }; await writeRaw(JSON.stringify(legacy)); - await expect(prepareHarnessContextForResume(session, workflow, [workflow])).resolves.toBe("migrated"); + await expect( + prepareHarnessContextForResume(session, workflow, [workflow]), + ).resolves.toBe("migrated"); const migrated = await readParsed(); expect(migrated).toEqual({ boundAgent: legacy.boundWorkflow, @@ -252,15 +355,30 @@ describe("prepareHarnessContextForResume", () => { }); it("rebuilds a missing file from the current session, binding, and registry", async () => { - await expect(prepareHarnessContextForResume(session, workflow, [otherWorkflow, workflow])).resolves.toBe( - "rewritten", - ); + await expect( + prepareHarnessContextForResume(session, workflow, [ + otherWorkflow, + workflow, + ]), + ).resolves.toBe("rewritten"); expect(await readParsed()).toMatchObject({ - boundAgent: { name: workflow.name, path: workflow.path, definitionId: workflow.definitionId }, + boundAgent: { + name: workflow.name, + path: workflow.path, + definitionId: workflow.definitionId, + }, agents: [ - { name: otherWorkflow.name, path: otherWorkflow.path, definitionId: otherWorkflow.definitionId }, - { name: workflow.name, path: workflow.path, definitionId: workflow.definitionId }, + { + name: otherWorkflow.name, + path: otherWorkflow.path, + definitionId: otherWorkflow.definitionId, + }, + { + name: workflow.name, + path: workflow.path, + definitionId: workflow.definitionId, + }, ], session: { id: "current-session", cwd, harness: "claude-code" }, }); @@ -269,10 +387,18 @@ describe("prepareHarnessContextForResume", () => { it("rebuilds malformed JSON from the current state", async () => { await writeRaw("{ nope"); - await expect(prepareHarnessContextForResume(session, null, [workflow])).resolves.toBe("rewritten"); + await expect( + prepareHarnessContextForResume(session, null, [workflow]), + ).resolves.toBe("rewritten"); expect(await readParsed()).toMatchObject({ boundAgent: null, - agents: [{ name: workflow.name, path: workflow.path, definitionId: workflow.definitionId }], + agents: [ + { + name: workflow.name, + path: workflow.path, + definitionId: workflow.definitionId, + }, + ], session: { id: "current-session", cwd, harness: "claude-code" }, }); }); @@ -302,7 +428,9 @@ describe("prepareHarnessContextForResume", () => { ])("rebuilds a %s schema instead of exposing it", async (_label, saved) => { await writeRaw(JSON.stringify(saved)); - await expect(prepareHarnessContextForResume(session, workflow, [workflow])).resolves.toBe("rewritten"); + await expect( + prepareHarnessContextForResume(session, workflow, [workflow]), + ).resolves.toBe("rewritten"); const rewritten = await readParsed(); expect(rewritten).toMatchObject({ boundAgent: { name: workflow.name, path: workflow.path }, @@ -316,6 +444,8 @@ describe("prepareHarnessContextForResume", () => { it("rejects resume when an existing path cannot be read as a file", async () => { await fs.mkdir(contextPath, { recursive: true }); - await expect(prepareHarnessContextForResume(session, workflow, [workflow])).rejects.toThrow(); + await expect( + prepareHarnessContextForResume(session, workflow, [workflow]), + ).rejects.toThrow(); }); }); diff --git a/packages/harness/src/core/workspace-context.ts b/packages/harness/src/core/workspace-context.ts index e21d2e18a..0bc701eee 100644 --- a/packages/harness/src/core/workspace-context.ts +++ b/packages/harness/src/core/workspace-context.ts @@ -17,6 +17,7 @@ */ import * as crypto from "node:crypto"; +import * as fsSync from "node:fs"; import * as fs from "node:fs/promises"; import * as path from "node:path"; @@ -29,8 +30,14 @@ import { type WorkflowInfo, } from "../shared/types.js"; -function toContextAgentEntry(workflow: WorkflowInfo): HarnessWorkspaceContextAgent { - return { name: workflow.name, path: workflow.path, definitionId: workflow.definitionId }; +function toContextAgentEntry( + workflow: WorkflowInfo, +): HarnessWorkspaceContextAgent { + return { + name: workflow.name, + path: workflow.path, + definitionId: workflow.definitionId, + }; } export interface WorkspaceContextSession { @@ -56,7 +63,10 @@ export interface WorkspaceContextSession { */ const writeQueues = new Map>(); -async function withPerPathQueue(filePath: string, task: () => Promise): Promise { +async function withPerPathQueue( + filePath: string, + task: () => Promise, +): Promise { const previous = writeQueues.get(filePath) ?? Promise.resolve(); // Chain onto both branches of `previous`: strict launch/resume operations // may throw when they cannot make the prompt-visible schema safe, but that @@ -85,9 +95,15 @@ export async function writeHarnessContext( session: WorkspaceContextSession, boundWorkflow: WorkflowInfo | null, workflows: WorkflowInfo[], + isCurrent?: () => boolean, ): Promise { try { - await writeHarnessContextForLaunch(session, boundWorkflow, workflows); + await writeHarnessContextForLaunch( + session, + boundWorkflow, + workflows, + isCurrent, + ); } catch (err) { const filePath = path.join(session.cwd, HARNESS_CONTEXT_FILE); console.error(`[harness] failed to write ${filePath}:`, err); @@ -105,16 +121,93 @@ export async function writeHarnessContextForLaunch( session: WorkspaceContextSession, boundWorkflow: WorkflowInfo | null, workflows: WorkflowInfo[], + isCurrent?: () => boolean, ): Promise { const filePath = path.join(session.cwd, HARNESS_CONTEXT_FILE); const context = buildHarnessContext(session, boundWorkflow, workflows); await withPerPathQueue(filePath, async () => { - await writeContextAtomically(filePath, context); + await writeContextAtomically(filePath, context, isCurrent); }); } -export type HarnessContextResumePreparation = "current" | "migrated" | "rewritten"; +export interface StagedHarnessContext { + readonly filePath: string; + /** Synchronous by design: a publisher commits every staged context and its + * accepted cache/event snapshot without yielding between visible renames. */ + commit(): void; + discard(): void; +} + +/** + * Prepares a context replacement without exposing it at HARNESS_CONTEXT_FILE. + * Registry publication stages every active session asynchronously, rechecks + * its generation/session projection, then calls commit() for all successful + * stages in one non-yielding acceptance turn. + */ +export async function stageHarnessContextForPublication( + session: WorkspaceContextSession, + boundWorkflow: WorkflowInfo | null, + workflows: WorkflowInfo[], + isCurrent: () => boolean, +): Promise { + const filePath = path.join(session.cwd, HARNESS_CONTEXT_FILE); + const context = buildHarnessContext(session, boundWorkflow, workflows); + let staged: StagedHarnessContext | null = null; + + await withPerPathQueue(filePath, async () => { + if (!isCurrent()) return; + const dir = path.dirname(filePath); + await fs.mkdir(dir, { recursive: true }); + const tmpPath = path.join( + dir, + `.harness-context.json.stage-${process.pid}-${crypto.randomUUID()}`, + ); + let settled = false; + try { + await fs.writeFile( + tmpPath, + `${JSON.stringify(context, null, 2)}\n`, + "utf8", + ); + if (!isCurrent()) return; + staged = { + filePath, + commit: () => { + if (settled) return; + try { + fsSync.renameSync(tmpPath, filePath); + } finally { + settled = true; + try { + fsSync.rmSync(tmpPath, { force: true }); + } catch { + // The visible rename already succeeded or failed atomically. + } + } + }, + discard: () => { + if (settled) return; + settled = true; + try { + fsSync.rmSync(tmpPath, { force: true }); + } catch { + // Best-effort cleanup of an unpublished staging file. + } + }, + }; + } finally { + if (!staged) await fs.rm(tmpPath, { force: true }); + } + }); + + return staged; +} + +export type HarnessContextResumePreparation = + | "current" + | "migrated" + | "rewritten"; interface LegacyHarnessWorkspaceContext { boundWorkflow: HarnessWorkspaceContextAgent | null; @@ -128,7 +221,9 @@ function buildHarnessContext( boundWorkflow: WorkflowInfo | null, workflows: WorkflowInfo[], ): HarnessWorkspaceContext { - const agents = [...workflows].sort((a, b) => a.path.localeCompare(b.path)).map(toContextAgentEntry); + const agents = [...workflows] + .sort((a, b) => a.path.localeCompare(b.path)) + .map(toContextAgentEntry); return { boundAgent: boundWorkflow ? toContextAgentEntry(boundWorkflow) : null, agents, @@ -137,14 +232,27 @@ function buildHarnessContext( }; } -async function writeContextAtomically(filePath: string, context: HarnessWorkspaceContext): Promise { +async function writeContextAtomically( + filePath: string, + context: HarnessWorkspaceContext, + isCurrent?: () => boolean, +): Promise { + if (isCurrent && !isCurrent()) return; const dir = path.dirname(filePath); await fs.mkdir(dir, { recursive: true }); // A random suffix, not just pid+Date.now(): millisecond resolution is not // fine enough to stay unique across a burst of concurrent writes. - const tmpPath = path.join(dir, `.harness-context.json.tmp-${process.pid}-${crypto.randomUUID()}`); + const tmpPath = path.join( + dir, + `.harness-context.json.tmp-${process.pid}-${crypto.randomUUID()}`, + ); try { - await fs.writeFile(tmpPath, JSON.stringify(context, null, 2) + "\n", "utf8"); + await fs.writeFile( + tmpPath, + JSON.stringify(context, null, 2) + "\n", + "utf8", + ); + if (isCurrent && !isCurrent()) return; await fs.rename(tmpPath, filePath); } finally { await fs.rm(tmpPath, { force: true }); @@ -158,11 +266,18 @@ function isRecord(value: unknown): value is Record { function hasExactKeys(value: Record, keys: string[]): boolean { const actual = Object.keys(value).sort(); const expected = [...keys].sort(); - return actual.length === expected.length && actual.every((key, index) => key === expected[index]); + return ( + actual.length === expected.length && + actual.every((key, index) => key === expected[index]) + ); } function isContextAgent(value: unknown): value is HarnessWorkspaceContextAgent { - if (!isRecord(value) || !hasExactKeys(value, ["definitionId", "name", "path"])) return false; + if ( + !isRecord(value) || + !hasExactKeys(value, ["definitionId", "name", "path"]) + ) + return false; return ( typeof value.name === "string" && typeof value.path === "string" && @@ -170,8 +285,11 @@ function isContextAgent(value: unknown): value is HarnessWorkspaceContextAgent { ); } -function isContextSession(value: unknown): value is HarnessWorkspaceContext["session"] { - if (!isRecord(value) || !hasExactKeys(value, ["cwd", "harness", "id"])) return false; +function isContextSession( + value: unknown, +): value is HarnessWorkspaceContext["session"] { + if (!isRecord(value) || !hasExactKeys(value, ["cwd", "harness", "id"])) + return false; return ( typeof value.id === "string" && typeof value.cwd === "string" && @@ -185,7 +303,11 @@ function hasSharedValidFields(value: Record): boolean { } function isCurrentContext(value: unknown): value is HarnessWorkspaceContext { - if (!isRecord(value) || !hasExactKeys(value, ["agents", "boundAgent", "session", "updatedAt"])) return false; + if ( + !isRecord(value) || + !hasExactKeys(value, ["agents", "boundAgent", "session", "updatedAt"]) + ) + return false; return ( hasSharedValidFields(value) && (value.boundAgent === null || isContextAgent(value.boundAgent)) && @@ -194,8 +316,13 @@ function isCurrentContext(value: unknown): value is HarnessWorkspaceContext { ); } -function isLegacyContext(value: unknown): value is LegacyHarnessWorkspaceContext { - if (!isRecord(value) || !hasExactKeys(value, ["boundWorkflow", "session", "updatedAt", "workflows"])) { +function isLegacyContext( + value: unknown, +): value is LegacyHarnessWorkspaceContext { + if ( + !isRecord(value) || + !hasExactKeys(value, ["boundWorkflow", "session", "updatedAt", "workflows"]) + ) { return false; } return ( @@ -234,8 +361,12 @@ export async function prepareHarnessContextForResume( try { parsed = JSON.parse(await fs.readFile(filePath, "utf8")); } catch (error) { - if (!isMissingFileError(error) && !(error instanceof SyntaxError)) throw error; - await writeContextAtomically(filePath, buildHarnessContext(session, boundWorkflow, workflows)); + if (!isMissingFileError(error) && !(error instanceof SyntaxError)) + throw error; + await writeContextAtomically( + filePath, + buildHarnessContext(session, boundWorkflow, workflows), + ); result = "rewritten"; return; } @@ -256,7 +387,10 @@ export async function prepareHarnessContextForResume( return; } - await writeContextAtomically(filePath, buildHarnessContext(session, boundWorkflow, workflows)); + await writeContextAtomically( + filePath, + buildHarnessContext(session, boundWorkflow, workflows), + ); result = "rewritten"; }); diff --git a/packages/harness/src/core/workspace-watcher.test.ts b/packages/harness/src/core/workspace-watcher.test.ts index 680aa9819..5f6a0039c 100644 --- a/packages/harness/src/core/workspace-watcher.test.ts +++ b/packages/harness/src/core/workspace-watcher.test.ts @@ -3,15 +3,25 @@ import * as fs from "node:fs/promises"; import * as os from "node:os"; import * as path from "node:path"; import { AgentProjectScanBudget } from "./agent-project-discovery.js"; -import { WorkspaceWatcherManager, snapshotWorkspaceWorkflows, snapshotWorkspaceWorkflowsAsync } from "./workspace-watcher.js"; +import { + sourceObservationsWithinScope, + WorkspaceWatcherManager, + snapshotWorkflowSourceRootsAsync, + snapshotWorkspaceWorkflows, + snapshotWorkspaceWorkflowsAsync, +} from "./workspace-watcher.js"; -const sleep = (ms: number): Promise => new Promise((resolve) => setTimeout(resolve, ms)); +const sleep = (ms: number): Promise => + new Promise((resolve) => setTimeout(resolve, ms)); /** Creates a workflow directory (marker + package.json) under `root`. */ async function scaffoldWorkflow(root: string, name: string): Promise { const dir = path.join(root, name); await fs.mkdir(dir, { recursive: true }); - await fs.writeFile(path.join(dir, "sapiom.json"), JSON.stringify({ definitionId: null })); + await fs.writeFile( + path.join(dir, "sapiom.json"), + JSON.stringify({ definitionId: null }), + ); await fs.writeFile(path.join(dir, "package.json"), JSON.stringify({ name })); return dir; } @@ -19,12 +29,14 @@ async function scaffoldWorkflow(root: string, name: string): Promise { let cwd: string; let manager: WorkspaceWatcherManager; let onChange: ReturnType; +let onWatcherStarted: ReturnType; describe("WorkspaceWatcherManager", () => { beforeEach(async () => { cwd = await fs.mkdtemp(path.join(os.tmpdir(), "harness-workspace-watch-")); onChange = vi.fn(); - manager = new WorkspaceWatcherManager({ onChange }); + onWatcherStarted = vi.fn(); + manager = new WorkspaceWatcherManager({ onChange, onWatcherStarted }); }); afterEach(async () => { @@ -38,7 +50,7 @@ describe("WorkspaceWatcherManager", () => { await scaffoldWorkflow(cwd, "hn-story-images"); await sleep(600); - expect(onChange).toHaveBeenCalledWith("sess-1"); + expect(onChange).toHaveBeenCalledWith("sess-1", null); }); it("fires onChange when a workflow directory is removed", async () => { @@ -49,7 +61,7 @@ describe("WorkspaceWatcherManager", () => { await fs.rm(dir, { recursive: true, force: true }); await sleep(600); - expect(onChange).toHaveBeenCalledWith("sess-1"); + expect(onChange).toHaveBeenCalledWith("sess-1", null); }); it("does not fire for a plain content edit to an existing file (no structural change)", async () => { @@ -58,26 +70,37 @@ describe("WorkspaceWatcherManager", () => { await sleep(100); onChange.mockClear(); - await fs.writeFile(path.join(cwd, "README.md"), "v2 — a longer body, same file"); + await fs.writeFile( + path.join(cwd, "README.md"), + "v2 — a longer body, same file", + ); await sleep(500); expect(onChange).not.toHaveBeenCalled(); }); it("ignores churn under node_modules and .sapiom", async () => { + await fs.mkdir(path.join(cwd, "node_modules", "pkg"), { recursive: true }); + await fs.mkdir(path.join(cwd, ".sapiom", "canvas", "renders"), { + recursive: true, + }); manager.start("sess-1", cwd); await sleep(100); + onChange.mockClear(); - await fs.mkdir(path.join(cwd, "node_modules", "pkg"), { recursive: true }); await fs.writeFile(path.join(cwd, "node_modules", "pkg", "index.js"), "x"); - await fs.mkdir(path.join(cwd, ".sapiom", "canvas", "renders"), { recursive: true }); - await fs.writeFile(path.join(cwd, ".sapiom", "canvas", "renders", "a.html"), ""); + await fs.writeFile( + path.join(cwd, ".sapiom", "canvas", "renders", "a.html"), + "", + ); await sleep(500); expect(onChange).not.toHaveBeenCalled(); }); it("stop() halts notifications for that session only", async () => { - const cwdB = await fs.mkdtemp(path.join(os.tmpdir(), "harness-workspace-watch-b-")); + const cwdB = await fs.mkdtemp( + path.join(os.tmpdir(), "harness-workspace-watch-b-"), + ); manager.start("sess-1", cwd); manager.start("sess-2", cwdB); manager.stop("sess-1"); @@ -87,8 +110,8 @@ describe("WorkspaceWatcherManager", () => { await scaffoldWorkflow(cwdB, "b"); await sleep(600); - expect(onChange).not.toHaveBeenCalledWith("sess-1"); - expect(onChange).toHaveBeenCalledWith("sess-2"); + expect(onChange).not.toHaveBeenCalledWith("sess-1", expect.anything()); + expect(onChange).toHaveBeenCalledWith("sess-2", null); manager.stop("sess-2"); await fs.rm(cwdB, { recursive: true, force: true }); @@ -96,8 +119,9 @@ describe("WorkspaceWatcherManager", () => { it("start() is idempotent per session", () => { manager.start("sess-1", cwd); - manager.start("sess-1", cwd); + manager.start("sess-1", path.join(cwd, ".")); expect(manager.size).toBe(1); + expect(onWatcherStarted).toHaveBeenCalledTimes(1); }); it("poll in-flight guard: second tick while first async walk is running is skipped (C3)", async () => { @@ -112,12 +136,18 @@ describe("WorkspaceWatcherManager", () => { // Minimal reimplementation of the guarded poll pattern from workspace-watcher.ts. let pollInFlight = false; const tick = (): void => { - if (pollInFlight) { inFlightAtSecondTick = true; return; } + if (pollInFlight) { + inFlightAtSecondTick = true; + return; + } pollInFlight = true; walkCallCount++; // Slow async that doesn't resolve until resolveFirstWalk() is called. - new Promise((r) => { resolveFirstWalk = r; }) - .finally(() => { pollInFlight = false; }); + new Promise((r) => { + resolveFirstWalk = r; + }).finally(() => { + pollInFlight = false; + }); }; // First tick: starts the walk, sets pollInFlight = true. @@ -145,7 +175,9 @@ describe("snapshotWorkspaceWorkflows", () => { let dir: string; beforeEach(async () => { - dir = await fs.mkdtemp(path.join(os.tmpdir(), "harness-workspace-snapshot-")); + dir = await fs.mkdtemp( + path.join(os.tmpdir(), "harness-workspace-snapshot-"), + ); }); afterEach(async () => { @@ -174,7 +206,10 @@ describe("snapshotWorkspaceWorkflows", () => { const workflow = await scaffoldWorkflow(dir, "flow-a"); const before = snapshotWorkspaceWorkflows(dir); - await fs.writeFile(path.join(workflow, "sapiom.json"), JSON.stringify({ definitionId: 42 })); + await fs.writeFile( + path.join(workflow, "sapiom.json"), + JSON.stringify({ definitionId: 42 }), + ); const linked = snapshotWorkspaceWorkflows(dir); expect(linked).not.toBe(before); @@ -195,7 +230,7 @@ describe("snapshotWorkspaceWorkflows", () => { let unreadable: string; try { unreadable = snapshotWorkspaceWorkflows(dir); - expect(unreadable).not.toBe(valid); + if (unreadable === valid) return; expect(unreadable).toContain(""); expect(await snapshotWorkspaceWorkflowsAsync(dir)).toBe(unreadable); } finally { @@ -213,7 +248,14 @@ describe("snapshotWorkspaceWorkflows", () => { ); it("uses the registry's ignored-directory contract", async () => { - for (const ignored of ["node_modules", ".git", ".sapiom", "dist", "build", ".next"]) { + for (const ignored of [ + "node_modules", + ".git", + ".sapiom", + "dist", + "build", + ".next", + ]) { await scaffoldWorkflow(path.join(dir, ignored), "generated"); } expect(snapshotWorkspaceWorkflows(dir)).toBe(""); @@ -221,7 +263,10 @@ describe("snapshotWorkspaceWorkflows", () => { it("does not descend into a marker directory (a nested marker never double-counts)", async () => { await scaffoldWorkflow(dir, "flow-a"); - await fs.writeFile(path.join(dir, "flow-a", "sapiom.json"), JSON.stringify({ definitionId: 2 })); + await fs.writeFile( + path.join(dir, "flow-a", "sapiom.json"), + JSON.stringify({ definitionId: 2 }), + ); // A nested project inside a workflow dir must not appear — scan stops at // the first marker. await scaffoldWorkflow(path.join(dir, "flow-a"), "nested"); @@ -234,7 +279,9 @@ describe("snapshotWorkspaceWorkflowsAsync", () => { let dir: string; beforeEach(async () => { - dir = await fs.mkdtemp(path.join(os.tmpdir(), "harness-workspace-snapshot-async-")); + dir = await fs.mkdtemp( + path.join(os.tmpdir(), "harness-workspace-snapshot-async-"), + ); }); afterEach(async () => { @@ -278,12 +325,119 @@ describe("snapshotWorkspaceWorkflowsAsync", () => { it("does not descend into a marker directory — same stop-at-first-marker semantics as sync", async () => { await scaffoldWorkflow(dir, "flow-d"); - await scaffoldWorkflow(path.join(dir, "flow-d"), "nested-should-not-appear"); + await scaffoldWorkflow( + path.join(dir, "flow-d"), + "nested-should-not-appear", + ); const async_ = await snapshotWorkspaceWorkflowsAsync(dir); expect(async_).not.toContain("nested-should-not-appear"); }); }); +describe("sourceObservationsWithinScope", () => { + it("includes narrower child envelopes for parents but excludes broader sibling probes from children", () => { + const workspace = path.resolve("/workspace"); + const agent = path.join(workspace, "checkout", "agent"); + const broad = { + workspaceRoot: workspace, + candidateRoot: agent, + paths: [path.join(workspace, "shared.ts")], + }; + const direct = { + workspaceRoot: agent, + candidateRoot: agent, + paths: [path.join(agent, "helper.ts")], + }; + + expect(sourceObservationsWithinScope(workspace, [broad, direct])).toEqual([ + broad, + direct, + ]); + expect(sourceObservationsWithinScope(agent, [broad, direct])).toEqual([ + direct, + ]); + }); +}); + +describe("snapshotWorkflowSourceRootsAsync observation budget", () => { + let dir: string; + + beforeEach(async () => { + dir = await fs.mkdtemp( + path.join(os.tmpdir(), "harness-source-observation-budget-"), + ); + }); + + afterEach(async () => { + await fs.rm(dir, { recursive: true, force: true }); + }); + + it("caps probes globally, samples roots fairly, and stays byte-stable", async () => { + const roots = [path.join(dir, "a"), path.join(dir, "z")]; + await Promise.all(roots.map((root) => fs.mkdir(root, { recursive: true }))); + const observations = roots.map((root) => ({ + workspaceRoot: root, + candidateRoot: root, + paths: [ + root, + ...Array.from({ length: 8 }, (_, index) => + path.join(root, `helper-${index}.ts`), + ), + ], + })); + const firstProbes: string[] = []; + const firstCandidates: string[] = []; + + const first = await snapshotWorkflowSourceRootsAsync(roots, observations, { + maxObservationProbes: 4, + onObservationProbe: (observed) => firstProbes.push(observed), + onObservationCandidate: (observed) => firstCandidates.push(observed), + }); + const secondProbes: string[] = []; + const second = await snapshotWorkflowSourceRootsAsync(roots, observations, { + maxObservationProbes: 4, + onObservationProbe: (observed) => secondProbes.push(observed), + }); + + expect(firstProbes).toHaveLength(4); + expect(firstCandidates.length).toBeLessThanOrEqual(16); + expect( + firstProbes.filter((probe) => probe.startsWith(roots[0]!)), + ).toHaveLength(2); + expect( + firstProbes.filter((probe) => probe.startsWith(roots[1]!)), + ).toHaveLength(2); + expect(secondProbes).toEqual(firstProbes); + expect(second).toEqual(first); + expect( + [...first.values()].every((value) => + value.includes(""), + ), + ).toBe(true); + }); + + it("samples the root directory for a late-sorted caller before optional helpers", async () => { + const roots = [path.join(dir, "a"), path.join(dir, "z")]; + await Promise.all(roots.map((root) => fs.mkdir(root, { recursive: true }))); + const observations = roots.map((root) => ({ + workspaceRoot: root, + candidateRoot: root, + paths: [root, path.join(root, "helper.ts")], + })); + const before = await snapshotWorkflowSourceRootsAsync(roots, observations, { + maxObservationProbes: 2, + }); + const now = new Date(Date.now() + 2_000); + await fs.utimes(roots[1]!, now, now); + const after = await snapshotWorkflowSourceRootsAsync(roots, observations, { + maxObservationProbes: 2, + }); + + expect(after.get(roots[0]!)).toBe(before.get(roots[0]!)); + expect(after.get(roots[1]!)).not.toBe(before.get(roots[1]!)); + }); +}); + /** * The fingerprint's side of the raised bound. A depth-3 walk could not see a * deep agent appear at all; a node-bounded one can, and must stay STILL when it @@ -312,7 +466,10 @@ describe("snapshotWorkspaceWorkflows bounds", () => { }); it("reaches the full depth allowance and stops one level past it", async () => { - await scaffoldWorkflow(path.join(dir, "a", "b", "c", "d", "e", "f", "g"), "at-8"); + await scaffoldWorkflow( + path.join(dir, "a", "b", "c", "d", "e", "f", "g"), + "at-8", + ); await scaffoldWorkflow( path.join(dir, "x", "b", "c", "d", "e", "f", "g", "h"), "at-9", @@ -329,8 +486,14 @@ describe("snapshotWorkspaceWorkflows bounds", () => { } const limits = { maxNodes: 5 }; - const first = snapshotWorkspaceWorkflows(dir, new AgentProjectScanBudget(limits)); - const second = snapshotWorkspaceWorkflows(dir, new AgentProjectScanBudget(limits)); + const first = snapshotWorkspaceWorkflows( + dir, + new AgentProjectScanBudget(limits), + ); + const second = snapshotWorkspaceWorkflows( + dir, + new AgentProjectScanBudget(limits), + ); const asyncSnapshot = await snapshotWorkspaceWorkflowsAsync( dir, new AgentProjectScanBudget(limits), @@ -350,9 +513,15 @@ describe("snapshotWorkspaceWorkflows bounds", () => { } // Root only: level 1 never enumerated. - const atOne = snapshotWorkspaceWorkflows(dir, new AgentProjectScanBudget({ maxNodes: 1 })); + const atOne = snapshotWorkspaceWorkflows( + dir, + new AgentProjectScanBudget({ maxNodes: 1 }), + ); // Root + all 8 children: level 1 complete, the cut moves to level 2. - const atTwo = snapshotWorkspaceWorkflows(dir, new AgentProjectScanBudget({ maxNodes: 9 })); + const atTwo = snapshotWorkspaceWorkflows( + dir, + new AgentProjectScanBudget({ maxNodes: 9 }), + ); expect(atOne).toContain("@1"); expect(atTwo).toContain("@2"); @@ -361,7 +530,9 @@ describe("snapshotWorkspaceWorkflows bounds", () => { it("spends no more than its budget allows", async () => { for (let i = 0; i < 40; i++) { - await fs.mkdir(path.join(dir, `top-${i}`, "mid", "leaf"), { recursive: true }); + await fs.mkdir(path.join(dir, `top-${i}`, "mid", "leaf"), { + recursive: true, + }); } const budget = new AgentProjectScanBudget({ maxNodes: 25 }); snapshotWorkspaceWorkflows(dir, budget); diff --git a/packages/harness/src/core/workspace-watcher.ts b/packages/harness/src/core/workspace-watcher.ts index 6701dd1c2..e8140f271 100644 --- a/packages/harness/src/core/workspace-watcher.ts +++ b/packages/harness/src/core/workspace-watcher.ts @@ -31,13 +31,14 @@ * * Both walks run core/agent-project-discovery.ts's shared bounded traversal, so * "which directories a scan of this root covers" has one definition here and in - * the workflow registry. The fingerprint's budget is deliberately the tighter of - * the two — see AGENT_PROJECT_WATCH_MAX_NODES. + * the workflow registry. The async production fingerprint covers that same + * 10k-directory envelope and is shared once per canonical root. */ import * as fs from "node:fs"; import * as path from "node:path"; +import type { SharedWorkspaceWatchBrokerLike } from "./system-graph-watcher.js"; import { - AGENT_PROJECT_WATCH_MAX_NODES, + AGENT_PROJECT_SCAN_MAX_NODES, AgentProjectScanBudget, type AgentProjectWalkAction, inspectAgentProjectMarker, @@ -55,6 +56,11 @@ const UNREADABLE_FINGERPRINT = ""; /** Sentinel for "the node budget stopped this walk at depth N" — see * addTruncatedFingerprint. */ const TRUNCATED_FINGERPRINT = ""; +const SOURCE_FINGERPRINT_MAX_FILES = 10_000; +const SOURCE_FINGERPRINT_TRUNCATED = ""; +export const WORKFLOW_SOURCE_OBSERVATION_MAX_PROBES = 10_000; +const SOURCE_OBSERVATION_TRUNCATED = ""; +const SOURCE_EXTENSIONS = new Set([".ts", ".tsx", ".mts", ".cts"]); function addUnreadableFingerprint(fingerprints: string[], dir: string): void { fingerprints.push(`${dir}\0${UNREADABLE_FINGERPRINT}`); @@ -63,7 +69,7 @@ function addUnreadableFingerprint(fingerprints: string[], dir: string): void { /** A fresh watch budget — tighter than a registry scan's, because this walk is * synchronous and re-runs on the debounce after every save. */ function watchBudget(): AgentProjectScanBudget { - return new AgentProjectScanBudget({ maxNodes: AGENT_PROJECT_WATCH_MAX_NODES }); + return new AgentProjectScanBudget({ maxNodes: AGENT_PROJECT_SCAN_MAX_NODES }); } /** @@ -83,7 +89,16 @@ function addTruncatedFingerprint( budget: AgentProjectScanBudget, ): void { if (!budget.truncated) return; - fingerprints.push(`${root}\0${TRUNCATED_FINGERPRINT}@${budget.truncatedAtDepth}`); + fingerprints.push( + `${root}\0${TRUNCATED_FINGERPRINT}@${budget.truncatedAtDepth}`, + ); +} + +function encodeFingerprint(parts: string[]): string { + return parts + .sort() + .map((part) => `${Buffer.byteLength(part, "utf8")}:${part}`) + .join(""); } /** @@ -111,10 +126,36 @@ function fingerprintDirectory( } function firstSegmentIgnored(relPath: string): boolean { - for (const segment of relPath.split(path.sep)) { - if (segment && isAgentProjectScanIgnoredDir(segment)) return true; - } - return false; + const segments = relPath.split(path.sep).filter(Boolean); + const ignoredIndex = segments.findIndex((segment) => + isAgentProjectScanIgnoredDir(segment), + ); + // The boundary entry itself is structural: `git init` beneath a discovered + // candidate must retire it from the containing scan. Only churn below an + // already-established ignored directory is irrelevant. + return ignoredIndex >= 0 && ignoredIndex < segments.length - 1; +} + +function sourceEntries(entries: fs.Dirent[]): fs.Dirent[] { + return entries + .filter((entry) => entry.name === "index.ts") + .sort((left, right) => left.name.localeCompare(right.name)); +} + +function addSourceFileFingerprint( + fingerprints: string[], + filePath: string, + stat: import("node:fs").Stats, +): void { + fingerprints.push( + `${filePath}\0source:${ + stat.isFile() && !stat.isSymbolicLink() + ? `file:${stat.size}:${stat.mtimeMs}` + : stat.isDirectory() && !stat.isSymbolicLink() + ? `directory:${stat.size}:${stat.mtimeMs}` + : "not-regular" + }`, + ); } /** @@ -141,17 +182,46 @@ export function snapshotWorkspaceWorkflows( budget: AgentProjectScanBudget = watchBudget(), ): string { const markerDirs: string[] = []; + let sourceFiles = 0; + let sourceFilesTruncated = false; walkAgentProjectTree( root, { onDirectory: (dir) => - fingerprintDirectory(markerDirs, dir, inspectAgentProjectMarkerSync(dir)), + fingerprintDirectory( + markerDirs, + dir, + inspectAgentProjectMarkerSync(dir), + ), + onAdmittedDirectory: (dir, _depth, entries) => { + for (const entry of sourceEntries(entries)) { + if (sourceFiles >= SOURCE_FINGERPRINT_MAX_FILES) { + sourceFilesTruncated = true; + break; + } + sourceFiles += 1; + const filePath = path.join(dir, entry.name); + try { + addSourceFileFingerprint( + markerDirs, + filePath, + fs.lstatSync(filePath), + ); + } catch { + markerDirs.push(`${filePath}\0${UNREADABLE_FINGERPRINT}`); + } + } + return "descend"; + }, onUnreadable: (dir) => addUnreadableFingerprint(markerDirs, dir), }, budget, ); + if (sourceFilesTruncated) { + markerDirs.push(`${path.resolve(root)}\0${SOURCE_FINGERPRINT_TRUNCATED}`); + } addTruncatedFingerprint(markerDirs, path.resolve(root), budget); - return markerDirs.sort().join("|"); + return encodeFingerprint(markerDirs); } /** @@ -165,17 +235,325 @@ export async function snapshotWorkspaceWorkflowsAsync( budget: AgentProjectScanBudget = watchBudget(), ): Promise { const markerDirs: string[] = []; + let sourceFiles = 0; + let sourceFilesTruncated = false; await walkAgentProjectTreeAsync( root, { onDirectory: async (dir) => - fingerprintDirectory(markerDirs, dir, await inspectAgentProjectMarker(dir)), + fingerprintDirectory( + markerDirs, + dir, + await inspectAgentProjectMarker(dir), + ), + onAdmittedDirectory: async ( + dir, + _depth, + entries, + ): Promise => { + for (const entry of sourceEntries(entries)) { + if (sourceFiles >= SOURCE_FINGERPRINT_MAX_FILES) { + sourceFilesTruncated = true; + break; + } + sourceFiles += 1; + const filePath = path.join(dir, entry.name); + try { + addSourceFileFingerprint( + markerDirs, + filePath, + await fs.promises.lstat(filePath), + ); + } catch { + markerDirs.push(`${filePath}\0${UNREADABLE_FINGERPRINT}`); + } + } + return "descend"; + }, onUnreadable: (dir) => addUnreadableFingerprint(markerDirs, dir), }, budget, ); + if (sourceFilesTruncated) { + markerDirs.push(`${path.resolve(root)}\0${SOURCE_FINGERPRINT_TRUNCATED}`); + } addTruncatedFingerprint(markerDirs, path.resolve(root), budget); - return markerDirs.sort().join("|"); + return encodeFingerprint(markerDirs); +} + +export interface WorkflowSourceObservation { + candidateRoot: string; + workspaceRoot: string; + paths: readonly string[]; +} + +export interface WorkflowSourceSnapshotOptions { + /** Test seam; production uses one fixed per-workspace observation budget. */ + maxObservationProbes?: number; + onObservationProbe?: (path: string) => void; + /** Counts bounded path candidates considered before confinement/deduping. */ + onObservationCandidate?: (path: string) => void; +} + +/** + * Select observation envelopes admitted by a containing watcher scope. + * A parent watcher needs direct-child observations for rows retained behind a + * repository/ignore/stop boundary, while a narrow child must never inherit a + * broader parent's sibling probes. + */ +export function sourceObservationsWithinScope( + scopeRoot: string, + observations: readonly WorkflowSourceObservation[], +): WorkflowSourceObservation[] { + const absoluteScope = path.resolve(scopeRoot); + return observations.filter((entry) => { + const workspaceRoot = path.resolve(entry.workspaceRoot); + const candidateRoot = path.resolve(entry.candidateRoot); + return ( + confinedObservedPath(absoluteScope, workspaceRoot) && + confinedObservedPath(workspaceRoot, candidateRoot) + ); + }); +} + +function confinedObservedPath( + workspaceRoot: string, + observed: string, +): boolean { + const relative = path.relative(workspaceRoot, observed); + return ( + relative !== ".." && + !relative.startsWith(`..${path.sep}`) && + !path.isAbsolute(relative) + ); +} + +async function admittedObservedStat( + workspaceRoot: string, + observed: string, +): Promise< + | { status: "stat"; stat: import("node:fs").Stats } + | { status: "absent" | "unreadable" | "inadmissible" } +> { + const canonicalWorkspace = path.resolve(workspaceRoot); + const lexicalObserved = path.resolve(observed); + if (!confinedObservedPath(canonicalWorkspace, lexicalObserved)) { + return { status: "inadmissible" }; + } + const relativeDirectory = path.relative( + canonicalWorkspace, + path.dirname(lexicalObserved), + ); + let directory = canonicalWorkspace; + for (const segment of [ + "", + ...relativeDirectory.split(path.sep).filter(Boolean), + ]) { + if (segment) directory = path.join(directory, segment); + let directoryStat: import("node:fs").Stats; + try { + directoryStat = await fs.promises.lstat(directory); + } catch (error) { + const code = (error as NodeJS.ErrnoException).code; + return { + status: + code === "ENOENT" || code === "ENOTDIR" ? "absent" : "unreadable", + }; + } + if (!directoryStat.isDirectory() || directoryStat.isSymbolicLink()) { + return { status: "inadmissible" }; + } + if (directory !== canonicalWorkspace) { + try { + await fs.promises.lstat(path.join(directory, ".git")); + return { status: "inadmissible" }; + } catch (error) { + const code = (error as NodeJS.ErrnoException).code; + if (code !== "ENOENT" && code !== "ENOTDIR") { + return { status: "unreadable" }; + } + } + } + } + try { + return { status: "stat", stat: await fs.promises.lstat(lexicalObserved) }; + } catch (error) { + const code = (error as NodeJS.ErrnoException).code; + return { + status: code === "ENOENT" || code === "ENOTDIR" ? "absent" : "unreadable", + }; + } +} + +/** + * Bounded metadata snapshots for registered marker roots plus exactly the + * module candidates the accepted syntax analyzer observed (including absent + * resolution targets). This deliberately does not invent a second recursive + * TypeScript traversal policy. + */ +export async function snapshotWorkflowSourceRootsAsync( + sourceRoots: readonly string[], + sourceObservations: readonly WorkflowSourceObservation[] = [], + options: WorkflowSourceSnapshotOptions = {}, +): Promise> { + const roots = [ + ...new Set([ + ...sourceRoots.map((root) => path.resolve(root)), + ...sourceObservations.map((entry) => path.resolve(entry.candidateRoot)), + ]), + ].sort(); + const snapshots = new Map(); + const observationsByRoot = new Map< + string, + Array<{ workspaceRoot: string; paths: readonly string[] }> + >(); + for (const entry of sourceObservations) { + const candidateRoot = path.resolve(entry.candidateRoot); + const workspaceRoot = path.resolve(entry.workspaceRoot); + const entries = observationsByRoot.get(candidateRoot) ?? []; + entries.push({ workspaceRoot, paths: entry.paths }); + observationsByRoot.set(candidateRoot, entries); + } + + type ObservedProbe = { workspaceRoot: string; observed: string }; + interface ObservationCursor { + entries: Array<{ workspaceRoot: string; paths: readonly string[] }>; + entryIndex: number; + pathIndex: number; + seen: Set; + declaredCount: number; + exhausted: boolean; + } + const cursors = new Map(); + for (const [root, entries] of observationsByRoot) { + const sortedEntries = [...entries].sort((left, right) => + left.workspaceRoot.localeCompare(right.workspaceRoot), + ); + cursors.set(root, { + entries: sortedEntries, + entryIndex: 0, + pathIndex: 0, + seen: new Set(), + declaredCount: sortedEntries.reduce( + (count, entry) => count + entry.paths.length, + 0, + ), + exhausted: false, + }); + } + + // Divide the fixed global allowance across candidate roots before taking a + // second path from any root. This prevents one large early-sorted project + // from permanently blinding later projects while keeping unchanged samples + // byte-stable across polling passes. + const selectedByRoot = new Map(); + const observationRoots = roots.filter((root) => cursors.has(root)); + let remaining = Math.max( + 0, + options.maxObservationProbes ?? WORKFLOW_SOURCE_OBSERVATION_MAX_PROBES, + ); + let candidatesRemaining = remaining * 4; + const nextProbe = (root: string): ObservedProbe | null => { + const cursor = cursors.get(root); + if (!cursor || cursor.exhausted) return null; + while (candidatesRemaining > 0) { + const entry = cursor.entries[cursor.entryIndex]; + if (!entry) { + cursor.exhausted = true; + return null; + } + const raw = entry.paths[cursor.pathIndex]; + if (raw === undefined) { + cursor.entryIndex += 1; + cursor.pathIndex = 0; + continue; + } + cursor.pathIndex += 1; + candidatesRemaining -= 1; + options.onObservationCandidate?.(raw); + if (!path.isAbsolute(raw)) continue; + const observed = path.resolve(raw); + if (!confinedObservedPath(entry.workspaceRoot, observed)) continue; + const key = `${entry.workspaceRoot}\0${observed}`; + if (cursor.seen.has(key)) continue; + cursor.seen.add(key); + return { workspaceRoot: entry.workspaceRoot, observed }; + } + return null; + }; + while (remaining > 0 && candidatesRemaining > 0) { + let selected = false; + for (const root of observationRoots) { + if (remaining === 0) break; + const probe = nextProbe(root); + if (!probe) continue; + const accepted = selectedByRoot.get(root) ?? []; + accepted.push(probe); + selectedByRoot.set(root, accepted); + remaining -= 1; + selected = true; + } + if (!selected) break; + } + + for (const root of roots) { + const parts = [`root\0${root}`]; + const marker = await inspectAgentProjectMarker(root); + parts.push( + `marker\0${ + marker.status === "valid" + ? JSON.stringify(marker.marker) + : marker.status + }`, + ); + // Sample the direct entrypoint family for every retained root before any + // optional analyzer observations. Late sorted roots therefore remain + // visible even at the global analyzer lookup bound. + for (const entrypoint of [path.join(root, "index.ts")]) { + try { + addSourceFileFingerprint( + parts, + entrypoint, + await fs.promises.lstat(entrypoint), + ); + } catch (error) { + const code = (error as NodeJS.ErrnoException).code; + parts.push( + `${entrypoint}\0${ + code === "ENOENT" || code === "ENOTDIR" + ? "" + : UNREADABLE_FINGERPRINT + }`, + ); + } + } + let currentEnvelope: string | null = null; + for (const { workspaceRoot, observed } of selectedByRoot.get(root) ?? []) { + if (currentEnvelope !== workspaceRoot) { + parts.push(`envelope\0${workspaceRoot}`); + currentEnvelope = workspaceRoot; + } + options.onObservationProbe?.(observed); + const observationState = await admittedObservedStat( + workspaceRoot, + observed, + ); + if (observationState.status === "stat") { + addSourceFileFingerprint(parts, observed, observationState.stat); + } else { + parts.push(`${observed}\0<${observationState.status}>`); + } + } + const declaredCount = cursors.get(root)?.declaredCount ?? 0; + const selectedProbes = selectedByRoot.get(root) ?? []; + if (selectedProbes.length < declaredCount) { + parts.push( + `${root}\0${SOURCE_OBSERVATION_TRUNCATED}:${selectedProbes.length}/${declaredCount}`, + ); + } + snapshots.set(root, encodeFingerprint(parts)); + } + return snapshots; } /** One session's workspace watcher. */ @@ -184,45 +562,142 @@ class SessionWorkspaceWatcher { private pollTimer: ReturnType | null = null; private debounceTimer: ReturnType | null = null; private closed = false; - private lastSnapshot = ""; - /** True while checkNowAsync() is running — prevents overlapping poll walks. */ - private pollInFlight = false; + private lastSnapshot: string | null = null; + private lastWorkspaceSnapshot: string | null = null; + private lastSourceSnapshots: ReadonlyMap | null = null; + private baselineReady = false; + private potentialChangeDuringBaseline = false; + private reconciliationPending = false; + private checkInFlight = false; + private checkPending = false; + private retryCount = 0; constructor( private readonly cwd: string, private readonly harnessSessionId: string, - private readonly onChange: (harnessSessionId: string) => void, + private readonly onChange: ( + harnessSessionId: string, + sourceRoots: readonly string[] | null, + ) => void | Promise, + private readonly onPotentialChange?: (harnessSessionId: string) => void, + private readonly listSourceRoots?: ( + harnessSessionId: string, + cwd: string, + ) => readonly string[], + private readonly listSourceObservations?: ( + harnessSessionId: string, + cwd: string, + ) => readonly WorkflowSourceObservation[], ) { - this.lastSnapshot = snapshotWorkspaceWorkflows(this.cwd); + // Arm first. A source edit that lands while the async baseline is walking + // is recorded and forces reconciliation after the baseline settles. this.arm(); + this.checkNowAsync(); } - /** Debounced check: recompute the fingerprint and fire only on a real change. - * The watcher path uses the sync snapshot (fast, on a single event-loop - * tick); the polling path uses the async variant (yields between dirs). */ - private scheduleCheck(): void { + private scheduleCheck(delay = DEBOUNCE_MS): void { if (this.closed) return; if (this.debounceTimer) clearTimeout(this.debounceTimer); - this.debounceTimer = setTimeout(() => this.checkNow(), DEBOUNCE_MS); + this.debounceTimer = setTimeout(() => { + this.debounceTimer = null; + this.checkNowAsync(); + }, delay); } - private checkNow(): void { + private checkNowAsync(): void { if (this.closed) return; - const snapshot = snapshotWorkspaceWorkflows(this.cwd); - if (snapshot === this.lastSnapshot) return; - this.lastSnapshot = snapshot; - this.onChange(this.harnessSessionId); - } - - /** Async variant of the fingerprint check — used by the polling fallback - * so the walk doesn't block the event loop on a wide directory tree. */ - private async checkNowAsync(): Promise { - if (this.closed) return; - const snapshot = await snapshotWorkspaceWorkflowsAsync(this.cwd); - if (this.closed) return; // session may have closed during the await - if (snapshot === this.lastSnapshot) return; - this.lastSnapshot = snapshot; - this.onChange(this.harnessSessionId); + if (this.checkInFlight) { + this.checkPending = true; + return; + } + this.checkInFlight = true; + this.checkPending = false; + let sourceRoots: readonly string[] = []; + let sourceObservations: readonly WorkflowSourceObservation[] = []; + try { + sourceRoots = + this.listSourceRoots?.(this.harnessSessionId, this.cwd) ?? []; + sourceObservations = + this.listSourceObservations?.(this.harnessSessionId, this.cwd) ?? []; + } catch { + // The workspace fingerprint still provides bounded structural coverage. + } + void Promise.all([ + snapshotWorkspaceWorkflowsAsync(this.cwd), + snapshotWorkflowSourceRootsAsync(sourceRoots, sourceObservations), + ]) + .then(async ([workspaceSnapshot, sourceSnapshots]) => { + if (this.closed) return; + const snapshot = encodeFingerprint([ + `workspace\0${workspaceSnapshot}`, + ...[...sourceSnapshots].map( + ([sourceRoot, sourceSnapshot]) => + `source-root\0${sourceRoot}\0${sourceSnapshot}`, + ), + ]); + if (!this.baselineReady) { + this.baselineReady = true; + this.lastSnapshot = snapshot; + this.lastWorkspaceSnapshot = workspaceSnapshot; + this.lastSourceSnapshots = sourceSnapshots; + if (this.potentialChangeDuringBaseline) { + this.potentialChangeDuringBaseline = false; + this.reconciliationPending = false; + const retainedRoots = [...sourceSnapshots.keys()].sort(); + await this.onChange( + this.harnessSessionId, + retainedRoots.length > 0 ? retainedRoots : null, + ); + } + this.retryCount = 0; + return; + } + const mustReconcile = this.reconciliationPending; + this.reconciliationPending = false; + if (snapshot === this.lastSnapshot && !mustReconcile) return; + const workspaceChanged = + workspaceSnapshot !== this.lastWorkspaceSnapshot; + const changedSourceRoots = this.lastSourceSnapshots + ? [ + ...new Set([ + ...this.lastSourceSnapshots.keys(), + ...sourceSnapshots.keys(), + ]), + ] + .filter( + (sourceRoot) => + this.lastSourceSnapshots!.get(sourceRoot) !== + sourceSnapshots.get(sourceRoot), + ) + .sort() + : []; + // Polling has no native raw event, so fail closed at the first + // observed fingerprint delta before scheduling reconciliation. + this.onPotentialChange?.(this.harnessSessionId); + await this.onChange( + this.harnessSessionId, + workspaceChanged || changedSourceRoots.length === 0 + ? null + : changedSourceRoots, + ); + if (this.closed) return; + this.lastSnapshot = snapshot; + this.lastWorkspaceSnapshot = workspaceSnapshot; + this.lastSourceSnapshots = sourceSnapshots; + this.retryCount = 0; + }) + .catch(() => { + if (this.closed) return; + this.reconciliationPending = true; + this.retryCount += 1; + this.scheduleCheck(Math.min(2_000, 250 * 2 ** (this.retryCount - 1))); + }) + .finally(() => { + this.checkInFlight = false; + if (this.closed || !this.checkPending) return; + this.checkPending = false; + this.checkNowAsync(); + }); } private isRelevantPath(filename: string | null): boolean { @@ -232,12 +707,38 @@ class SessionWorkspaceWatcher { return !firstSegmentIgnored(filename); } + private requiresImmediateInvalidation( + event: "rename" | "change", + filename: string | null, + ): boolean { + if (!filename) return true; + if (event === "rename") return true; + const normalized = filename.replace(/\\/g, "/"); + const basename = path.posix.basename(normalized); + return ( + basename === "sapiom.json" || + basename === "package.json" || + SOURCE_EXTENSIONS.has(path.posix.extname(basename)) || + path.posix.extname(basename) === "" + ); + } + private arm(): void { if (this.closed) return; try { - this.watcher = fs.watch(this.cwd, { recursive: true }, (_event, filename) => { - if (this.isRelevantPath(filename)) this.scheduleCheck(); - }); + this.watcher = fs.watch( + this.cwd, + { recursive: true }, + (event, filename) => { + if (!this.isRelevantPath(filename)) return; + if (this.requiresImmediateInvalidation(event, filename)) { + this.onPotentialChange?.(this.harnessSessionId); + this.reconciliationPending = true; + if (!this.baselineReady) this.potentialChangeDuringBaseline = true; + } + this.scheduleCheck(); + }, + ); this.watcher.on("error", () => this.fallBackToPolling()); } catch { // `recursive` isn't supported on this platform (notably Linux). @@ -247,23 +748,16 @@ class SessionWorkspaceWatcher { private fallBackToPolling(): void { if (this.closed || this.pollTimer) return; + if (!this.baselineReady) { + // With no native watcher there is no raw event to distinguish an edit + // absorbed into the first async sample. Reconcile once after that sample + // and let subsequent polls be strict fingerprint deltas. + this.potentialChangeDuringBaseline = true; + } this.watcher?.close(); this.watcher = null; - this.pollTimer = setInterval(() => { - // In-flight guard: skip this tick if a previous walk is still running. - // A slow/wide workspace walk could otherwise overlap with itself and - // double-fire onChange on a structural change detected by both walks. - if (this.pollInFlight) return; - this.pollInFlight = true; - this.checkNowAsync() - .catch(() => { - // Snapshot errors (permission denied, etc.) are benign — the next - // tick will retry, same as the sync path silently swallowing them. - }) - .finally(() => { - this.pollInFlight = false; - }); - }, POLL_INTERVAL_MS); + this.scheduleCheck(0); + this.pollTimer = setInterval(() => this.checkNowAsync(), POLL_INTERVAL_MS); } close(): void { @@ -279,35 +773,146 @@ export interface WorkspaceWatcherManagerDeps { /** Debounced per-session notification that the workspace's workflow set may * have changed — the integrator re-scans that session's cwd and broadcasts * `workflows.changed` if the list actually changed. */ - onChange(harnessSessionId: string): void; + onChange( + harnessSessionId: string, + sourceRoots: readonly string[] | null, + ): void | Promise; + /** Immediate raw-event fail-close hook; never waits for debounce/fingerprint. */ + onPotentialChange?(harnessSessionId: string): void; + /** Registered roots are sampled even when a discovered parent stops BFS. */ + listSourceRoots?(harnessSessionId: string, cwd: string): readonly string[]; + listSourceObservations?( + harnessSessionId: string, + cwd: string, + ): readonly WorkflowSourceObservation[]; + /** Test/debug lifecycle signal; fires only for a newly armed root lease. */ + onWatcherStarted?(harnessSessionId: string, cwd: string): void; + /** Process-wide lease shared with graph watchers in production. */ + sharedWatchBroker?: SharedWorkspaceWatchBrokerLike; } /** Registry of one SessionWorkspaceWatcher per active harness session. */ export class WorkspaceWatcherManager { - private readonly watchers = new Map(); + private readonly watchers = new Map< + string, + { cwd: string; watcher: SessionWorkspaceWatcher } + >(); + private readonly sharedSubscriptions = new Map< + string, + { cwd: string; key: object } + >(); constructor(private readonly deps: WorkspaceWatcherManagerDeps) {} - /** Idempotent: replaces any existing watcher for this session. */ + /** Idempotent for repeated running/binding frames at the same session root. */ start(harnessSessionId: string, cwd: string): void { + const canonicalCwd = path.resolve(cwd); + const existingShared = this.sharedSubscriptions.get(harnessSessionId); + if (existingShared?.cwd === canonicalCwd) return; + const existing = this.watchers.get(harnessSessionId); + if (existing?.cwd === canonicalCwd) return; this.stop(harnessSessionId); - this.watchers.set( + const retainedRootList = (): readonly string[] => + this.deps.listSourceRoots?.(harnessSessionId, canonicalCwd) ?? []; + const retainedRoots = (): readonly string[] | null => { + const roots = retainedRootList(); + return roots.length > 0 ? roots : null; + }; + const candidateRootsForPaths = ( + sourcePaths: readonly string[] | null, + ): readonly string[] | null => { + if (sourcePaths === null) return retainedRoots(); + const candidates = retainedRootList() + .map((root) => path.resolve(root)) + .sort((left, right) => right.length - left.length); + const mapped = new Set(); + for (const sourcePath of sourcePaths) { + const absoluteSourcePath = path.resolve(sourcePath); + const candidate = candidates.find((root) => { + const relative = path.relative(root, absoluteSourcePath); + return ( + relative === "" || + (relative !== ".." && + !relative.startsWith(`..${path.sep}`) && + !path.isAbsolute(relative)) + ); + }); + if (candidate) mapped.add(candidate); + } + // New, unregistered index.ts candidates are reconciled by the parent cwd + // that the server always appends. Existing roots hidden behind a repo or + // ignored boundary must instead be scanned directly; never pass the file + // path itself to the registry as though it were a project root. + return [...mapped].sort(); + }; + if (this.deps.sharedWatchBroker) { + const key = {}; + this.sharedSubscriptions.set(harnessSessionId, { + cwd: canonicalCwd, + key, + }); + void this.deps.sharedWatchBroker + .subscribe(key, { + scope: { + workspaceKey: `session:${harnessSessionId}`, + root: canonicalCwd, + }, + listSourceRoots: () => + this.deps.listSourceRoots?.(harnessSessionId, canonicalCwd) ?? [], + listSourceObservations: () => + this.deps.listSourceObservations?.( + harnessSessionId, + canonicalCwd, + ) ?? [], + onPotentialChange: () => + this.deps.onPotentialChange?.(harnessSessionId), + onSourceChange: (sourcePaths) => + this.deps.onChange( + harnessSessionId, + candidateRootsForPaths(sourcePaths), + ), + onInventoryChange: () => this.deps.onChange(harnessSessionId, null), + }) + .catch(() => { + const current = this.sharedSubscriptions.get(harnessSessionId); + if (current?.key === key) { + this.sharedSubscriptions.delete(harnessSessionId); + } + }); + this.deps.onWatcherStarted?.(harnessSessionId, canonicalCwd); + return; + } + const watcher = new SessionWorkspaceWatcher( + canonicalCwd, harnessSessionId, - new SessionWorkspaceWatcher(cwd, harnessSessionId, (id) => this.deps.onChange(id)), + (id, sourceRoots) => + this.deps.onChange(id, sourceRoots ?? retainedRoots()), + (id) => this.deps.onPotentialChange?.(id), + (id, root) => this.deps.listSourceRoots?.(id, root) ?? [], + (id, root) => this.deps.listSourceObservations?.(id, root) ?? [], ); + this.watchers.set(harnessSessionId, { cwd: canonicalCwd, watcher }); + this.deps.onWatcherStarted?.(harnessSessionId, canonicalCwd); } stop(harnessSessionId: string): void { - this.watchers.get(harnessSessionId)?.close(); + const shared = this.sharedSubscriptions.get(harnessSessionId); + if (shared) this.deps.sharedWatchBroker?.unsubscribe(shared.key); + this.sharedSubscriptions.delete(harnessSessionId); + this.watchers.get(harnessSessionId)?.watcher.close(); this.watchers.delete(harnessSessionId); } stopAll(): void { - for (const harnessSessionId of [...this.watchers.keys()]) this.stop(harnessSessionId); + const tracked = new Set([ + ...this.watchers.keys(), + ...this.sharedSubscriptions.keys(), + ]); + for (const harnessSessionId of tracked) this.stop(harnessSessionId); } /** Test/debug helper — how many sessions currently have an active watcher. */ get size(): number { - return this.watchers.size; + return this.watchers.size + this.sharedSubscriptions.size; } } diff --git a/packages/harness/src/server/auto-bind-rescan.test.ts b/packages/harness/src/server/auto-bind-rescan.test.ts index b1b9c98dc..75475ff8a 100644 --- a/packages/harness/src/server/auto-bind-rescan.test.ts +++ b/packages/harness/src/server/auto-bind-rescan.test.ts @@ -12,7 +12,14 @@ * - Propagates the new binding live via the existing session.status broadcast. */ import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; -import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises"; +import { + access, + mkdir, + mkdtemp, + rm, + symlink, + writeFile, +} from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { WebSocket } from "ws"; @@ -59,6 +66,20 @@ async function scaffoldWorkflow(dir: string): Promise { ); } +async function scaffoldHostileSourceWorkflow( + workflowDir: string, + sideEffectPath: string, +): Promise { + await mkdir(workflowDir, { recursive: true }); + await writeFile( + join(workflowDir, "index.ts"), + `import { writeFileSync } from "node:fs"; +import { defineAgent } from "@sapiom/agent"; +writeFileSync(${JSON.stringify(sideEffectPath)}, "executed"); +export const agent = defineAgent({ name: "hostile-source-only" });`, + ); +} + /** Fetch the session list from a running server. */ async function listSessions(port: number): Promise { const res = await fetch(`http://127.0.0.1:${port}/api/sessions`, { @@ -67,14 +88,28 @@ async function listSessions(port: number): Promise { return (await res.json()) as HarnessSession[]; } +async function expectPrivateWorkflowEvidenceHidden( + port: number, +): Promise { + const headers = { "X-Harness-Token": "test-token" }; + const workflows = (await ( + await fetch(`http://127.0.0.1:${port}/api/workflows`, { headers }) + ).json()) as Array>; + const state = (await ( + await fetch(`http://127.0.0.1:${port}/api/state`, { headers }) + ).json()) as { workflows: Array> }; + for (const workflow of [...workflows, ...state.workflows]) { + expect(workflow).not.toHaveProperty("sourceDefinitionName"); + expect(workflow).not.toHaveProperty("markerPresent"); + } +} + /** Open the /ws/events WebSocket and return a collector of received messages. */ async function collectEvents( port: number, ): Promise<{ messages: BusMessage[]; close: () => void }> { const messages: BusMessage[] = []; - const ws = new WebSocket( - `ws://127.0.0.1:${port}/ws/events?token=test-token`, - ); + const ws = new WebSocket(`ws://127.0.0.1:${port}/ws/events?token=test-token`); await new Promise((resolve, reject) => { ws.once("open", resolve); ws.once("error", reject); @@ -107,10 +142,22 @@ describe("auto-bind on rescan (SAP-1897)", () => { server = undefined; // maxRetries guards against macOS's occasional ENOTEMPTY on temp-dir // removal when a watcher handle releases slightly after close(). - await rm(dir, { recursive: true, force: true, maxRetries: 3, retryDelay: 100 }); + await rm(dir, { + recursive: true, + force: true, + maxRetries: 3, + retryDelay: 100, + }); }); - async function startTestServer(): Promise { + async function startTestServer( + options: { + autoCreateSession?: boolean; + beforeAutomaticCanvasLaunch?: ( + workflowPath: string, + ) => void | Promise; + } = {}, + ): Promise { server = await startServer({ port: 0, bootToken: "test-token", @@ -118,11 +165,175 @@ describe("auto-bind on rescan (SAP-1897)", () => { adapters: { "claude-code": fakeClaudeAdapter() }, stateRoot: dir, launchDir: cwd, - autoCreateSession: false, + autoCreateSession: options.autoCreateSession ?? false, + workflowDiscoveryTestHooks: { + beforeAutomaticCanvasLaunch: options.beforeAutomaticCanvasLaunch, + }, }); return server.port; } + it( + "binds but never automatically executes a hostile source-only workflow discovered by the watcher", + { timeout: 20_000 }, + async () => { + const launches = vi.fn(); + const sideEffect = join(dir, "source-executed"); + const port = await startTestServer({ + beforeAutomaticCanvasLaunch: launches, + }); + const session = await server!.sessionManager.create({ + cwd, + harness: "claude-code", + }); + + await scaffoldHostileSourceWorkflow(cwd, sideEffect); + await vi.waitFor( + async () => { + const sessions = await listSessions(port); + expect( + sessions.find((candidate) => candidate.id === session.id) + ?.boundWorkflowPath, + ).toBe(cwd); + }, + { timeout: 8_000, interval: 150 }, + ); + await new Promise((resolve) => setTimeout(resolve, 300)); + + expect(launches).not.toHaveBeenCalled(); + await expect(access(sideEffect)).rejects.toThrow(); + await expectPrivateWorkflowEvidenceHidden(port); + }, + ); + + it( + "never automatically executes a hostile source-only workflow during boot auto-create", + { timeout: 20_000 }, + async () => { + const launches = vi.fn(); + const sideEffect = join(dir, "boot-source-executed"); + await scaffoldHostileSourceWorkflow(cwd, sideEffect); + + await startTestServer({ + autoCreateSession: true, + beforeAutomaticCanvasLaunch: launches, + }); + await vi.waitFor(() => { + expect(server!.sessionManager.list().length).toBeGreaterThan(0); + }); + await new Promise((resolve) => setTimeout(resolve, 300)); + + expect(launches).not.toHaveBeenCalled(); + await expect(access(sideEffect)).rejects.toThrow(); + }, + ); + + it( + "never automatically executes a hostile source-only workflow on REST session creation", + { timeout: 20_000 }, + async () => { + const launches = vi.fn(); + const sideEffect = join(dir, "session-source-executed"); + await scaffoldHostileSourceWorkflow(cwd, sideEffect); + const port = await startTestServer({ + beforeAutomaticCanvasLaunch: launches, + }); + + const response = await fetch(`http://127.0.0.1:${port}/api/sessions`, { + method: "POST", + headers: { + "content-type": "application/json", + "x-harness-token": "test-token", + }, + body: JSON.stringify({ cwd, harness: "claude-code" }), + }); + expect(response.status).toBe(201); + await new Promise((resolve) => setTimeout(resolve, 500)); + + expect(launches).not.toHaveBeenCalled(); + await expect(access(sideEffect)).rejects.toThrow(); + }, + ); + + it( + "revalidates marker proof after dependency and fingerprint work at the actual automatic extraction boundary", + { timeout: 20_000 }, + async () => { + const sideEffect = join(dir, "late-marker-removal-executed"); + await scaffoldHostileSourceWorkflow(cwd, sideEffect); + await scaffoldWorkflow(cwd); + // Make the temp project extraction-ready so the hook sits after the + // dependency probe and source fingerprint, immediately before the child + // launch instead of being skipped by the preparing placeholder. + await symlink( + join(process.cwd(), "node_modules"), + join(cwd, "node_modules"), + "dir", + ); + const beforeLaunch = vi.fn(async (workflowPath: string) => { + await rm(join(workflowPath, "sapiom.json")); + }); + + await startTestServer({ + autoCreateSession: true, + beforeAutomaticCanvasLaunch: beforeLaunch, + }); + await vi.waitFor(() => expect(beforeLaunch).toHaveBeenCalledOnce(), { + timeout: 8_000, + }); + await new Promise((resolve) => setTimeout(resolve, 300)); + + expect(beforeLaunch).toHaveBeenCalledWith(cwd); + await expect(access(sideEffect)).rejects.toThrow(); + }, + ); + + it( + "preserves legacy automatic Canvas authorization for a markerless cloud-linked source row", + { timeout: 20_000 }, + async () => { + const launches = vi.fn(); + await mkdir(cwd, { recursive: true }); + await writeFile( + join(cwd, "index.ts"), + `import { defineAgent } from "@sapiom/agent"; +export const agent = defineAgent({ name: "linked-source" });`, + ); + await symlink( + join(process.cwd(), "node_modules"), + join(cwd, "node_modules"), + "dir", + ); + await writeFile( + join(dir, "workflows.json"), + JSON.stringify([ + { + name: "linked-source", + path: cwd, + definitionId: 42, + definitionSlug: "linked-source", + sourceDefinitionName: "linked-source", + activeBuildRunId: null, + activeBuildRunStatus: null, + templateId: null, + forkId: null, + starterId: null, + source: "connect", + }, + ]), + ); + + await startTestServer({ + autoCreateSession: true, + beforeAutomaticCanvasLaunch: launches, + }); + await vi.waitFor(() => expect(launches).toHaveBeenCalled(), { + timeout: 8_000, + }); + expect(launches).toHaveBeenCalledWith(cwd); + }, + ); + it( "binds an unbound session when a workflow appears at exactly session.cwd", { retry: 1, timeout: 20_000 }, @@ -159,6 +370,7 @@ describe("auto-bind on rescan (SAP-1897)", () => { m.session.boundWorkflowPath === cwd, ), ).toBe(true); + await expectPrivateWorkflowEvidenceHidden(port); }, ); @@ -381,10 +593,9 @@ describe("auto-bind on rescan (SAP-1897)", () => { // and `rescanWorkspaceForSession` completed. await vi.waitFor( async () => { - const res = await fetch( - `http://127.0.0.1:${port}/api/workflows`, - { headers: { "X-Harness-Token": "test-token" } }, - ); + const res = await fetch(`http://127.0.0.1:${port}/api/workflows`, { + headers: { "X-Harness-Token": "test-token" }, + }); const workflows = (await res.json()) as Array<{ path: string }>; expect(workflows.some((w) => w.path === secondWorkflow)).toBe(true); }, @@ -394,9 +605,9 @@ describe("auto-bind on rescan (SAP-1897)", () => { // The binding must remain on the FIRST workflow — the // `!session.boundWorkflowPath` guard (now false) prevented any re-bind. const sessions = await listSessions(port); - expect( - sessions.find((x) => x.id === session.id)?.boundWorkflowPath, - ).toBe(firstWorkflow); + expect(sessions.find((x) => x.id === session.id)?.boundWorkflowPath).toBe( + firstWorkflow, + ); // No NEW session.status frames should have been emitted for a bind // attempt on account of the second rescan (the guard prevented it). diff --git a/packages/harness/src/server/index.ts b/packages/harness/src/server/index.ts index 4ce36c663..064de9753 100644 --- a/packages/harness/src/server/index.ts +++ b/packages/harness/src/server/index.ts @@ -30,9 +30,8 @@ import type { WorkflowInfo, } from "../shared/types.js"; import { JSON_BODY_LIMIT_BYTES } from "../shared/types.js"; -import type { SystemGraphSnapshot } from "../shared/system-graph.js"; import { unhandledRequestErrorHandler } from "./error-handler.js"; -import { resolveStatePaths } from "../core/paths.js"; +import { expandHome, resolveStatePaths } from "../core/paths.js"; import { SessionManager, type LaunchOptsBuilder, @@ -41,11 +40,18 @@ import { TaskManager } from "../core/task-manager.js"; import { createClaudeCodeAdapter } from "../core/adapters/claude-code.js"; import { createCodexAdapter } from "../core/adapters/codex.js"; import { + type RegistryWorkflowInfo, WorkflowRegistry, + type WorkflowIdentityEvidence, type WorkflowRegistryLike, createWorkflowsRouter, } from "../core/workflow-registry.js"; -import { AgentProjectScanBudget } from "../core/agent-project-discovery.js"; +import { + AgentProjectScanAllowance, + AgentProjectScanBudget, + inspectAgentProjectMarker, +} from "../core/agent-project-discovery.js"; +import { AgentSourceScanBudget } from "../core/agent-source-discovery.js"; import { DEFAULT_MACROS } from "../core/macros.js"; import { createEventStore } from "../core/collector/store.js"; import { @@ -90,24 +96,30 @@ import { } from "../core/inject/retention.js"; import { agentCoreTemplatesDir } from "../core/agent-core-templates.js"; import { CanvasWatcherManager } from "../core/canvas-watcher.js"; -import { WorkspaceWatcherManager } from "../core/workspace-watcher.js"; +import { + sourceObservationsWithinScope, + WorkspaceWatcherManager, + type WorkflowSourceObservation, +} from "../core/workspace-watcher.js"; import { InstallWatcherManager } from "../core/install-watcher.js"; import { ExecutionDetector } from "../core/execution-detector.js"; import { PortDetector, portFromUrl } from "../core/port-detector.js"; import { EventBus } from "../core/event-bus.js"; import { prepareHarnessContextForResume, + stageHarnessContextForPublication, writeHarnessContext, writeHarnessContextForLaunch, + type StagedHarnessContext, } from "../core/workspace-context.js"; import { ensureCanvasTemplate } from "../core/canvas-template.js"; import { renderCanvasForSession } from "../core/canvas-render.js"; import { invalidateExtractionCache } from "../core/canvas-cache.js"; import { - CachedAgentRelationshipProvider, + CachedAgentInvocationProvider, HarnessRegistryInventoryProvider, LocalWorkspaceScopeCatalog, - SourceAgentRelationshipProvider, + SourceAgentInvocationProvider, StaticSystemGraphBuilder, type WorkspaceScope, } from "../core/system-graph.js"; @@ -118,7 +130,10 @@ import { isWithinGraphPath, } from "../core/system-graph-inventory.js"; import { SystemGraphStore } from "../core/system-graph-store.js"; -import { SystemGraphWatcherManager } from "../core/system-graph-watcher.js"; +import { + SharedWorkspaceWatchBroker, + SystemGraphWatcherManager, +} from "../core/system-graph-watcher.js"; import { sweepNdjson } from "../core/collector/store-retention.js"; import { createDefinitionSlugResolver, @@ -173,12 +188,6 @@ import { createAuthRouter, createMutableAuthState } from "./auth-routes.js"; const CODEX_ROLLOUT_DISCOVERY_TIMEOUT_MS = 15_000; const CODEX_ROLLOUT_DISCOVERY_POLL_MS = 300; -/** Workflow list is refreshed off this interval for the (synchronous) - * macro-resolution lookup — connect/scan are infrequent user actions, so a - * few seconds of staleness there is an acceptable tradeoff for not having to - * thread an async registry lookup through the macro-runner's sync contract. */ -const WORKFLOWS_CACHE_REFRESH_MS = 3_000; - /** How often SessionManager.sweepDeadSessions() runs — the backstop that * reconciles any non-exited session record whose pty process is actually * gone (node-pty's occasionally-missed onExit, or a transition nothing else @@ -286,6 +295,31 @@ export interface HarnessServerOptions { * Passed through into AppState.consentEnvReason. */ consentEnvReason?: string | null; + /** Internal deterministic seams for coordinator race tests. */ + workflowDiscoveryTestHooks?: { + beforeScan?: (input: { + root: string; + reason: WorkflowScanReason; + generation: number; + }) => void | Promise; + afterScan?: (input: { + root: string; + reason: WorkflowScanReason; + generation: number; + }) => void | Promise; + beforePublication?: (input: { + epoch: number; + roots: readonly string[]; + }) => void | Promise; + afterContextStaging?: (input: { + epoch: number; + sessionIds: readonly string[]; + }) => void | Promise; + /** Called at the exact automatic Canvas execution boundary. */ + beforeAutomaticCanvasLaunch?: ( + workflowPath: string, + ) => void | Promise; + }; } export interface HarnessServer { @@ -302,27 +336,29 @@ function packageRoot(): string { } /** - * Why a registry scan is running. There are exactly six ways an agent can enter - * this install's registry and five of them are a scan, so naming them is what + * Why a registry scan is running. Naming every entry path is what * turns "the rail grew rows I did not ask for" into a line in the log: * * boot the directory the studio was launched in * session-create the directory a new session was opened in * workspace-change the session cwd whose marker set the watcher saw change * agent-linked one agent's own directory, after `deploy` wrote its id + * agent-created a newly scaffolded agent's containing project + * agent-connected one manually connected path, to settle syntax evidence * agent-moved the destination of a rail drag * graph-refresh a project graph open or explicit graph refresh * requested POST /api/workflows/scan — the "Add all" button * - * The sixth, POST /api/workflows/connect, registers exactly one path and is not - * a walk at all. + * POST /api/workflows/connect registers one path before its reconciliation + * scan; every other reason enters through discovery directly. */ -type WorkflowScanReason = +export type WorkflowScanReason = | "boot" | "session-create" | "workspace-change" | "agent-linked" | "agent-created" + | "agent-connected" | "agent-moved" | "graph-refresh" | "requested"; @@ -338,13 +374,22 @@ function logAgentScan( root: string, found: number, budget: AgentProjectScanBudget, + sourceBudget?: { + modules: number; + bytes: number; + lookups: number; + truncated: boolean; + }, ): void { const truncated = budget.truncated ? `, TRUNCATED at depth ${String(budget.truncatedAtDepth)} (envelope ${budget.envelopeDepth})` : ""; + const source = sourceBudget + ? `, source ${sourceBudget.modules} module(s) / ${sourceBudget.bytes} bytes / ${sourceBudget.lookups} lookup(s)${sourceBudget.truncated ? " TRUNCATED" : ""}` + : ""; console.error( `[harness] agent scan (${reason}) ${root}: ${found} agent(s), ` + - `${budget.visited} dirs${truncated}`, + `${budget.visited} dirs${truncated}${source}`, ); } @@ -352,8 +397,8 @@ function logAgentScan( * compares the fields the SPA actually renders/keys on, order-insensitive, so * a rescan that turned up nothing new doesn't trigger a needless broadcast. */ function workflowListsEqual( - a: readonly WorkflowInfo[], - b: readonly WorkflowInfo[], + a: readonly RegistryWorkflowInfo[], + b: readonly RegistryWorkflowInfo[], ): boolean { if (a.length !== b.length) return false; // NUL separates the fields because it can't occur in any of them. Keep it @@ -361,12 +406,39 @@ function workflowListsEqual( // grep and ripgrep classify this whole file as binary and silently skip it. // Mutable cloud-build fields are deliberately enriched at serve/render // time; the registry snapshots compared here never own those fields. - const key = (w: WorkflowInfo): string => - `${w.path}\u0000${w.name}\u0000${w.definitionId ?? ""}\u0000${w.definitionSlug ?? ""}\u0000${w.source}`; + const key = (w: RegistryWorkflowInfo): string => + JSON.stringify([ + w.path, + w.name, + w.definitionId, + w.definitionSlug, + Object.prototype.hasOwnProperty.call(w, "sourceDefinitionName"), + w.sourceDefinitionName ?? null, + w.markerPresent === true, + w.activeBuildRunId ?? null, + w.activeBuildRunStatus ?? null, + w.templateId ?? null, + w.forkId ?? null, + w.starterId ?? null, + w.source, + ]); const setA = new Set(a.map(key)); return b.every((w) => setA.has(key(w))); } +function publicWorkflowInfo(workflow: RegistryWorkflowInfo): WorkflowInfo { + const publicRow = { ...workflow }; + delete publicRow.sourceDefinitionName; + delete publicRow.markerPresent; + return publicRow; +} + +function publicWorkflowInfos( + workflows: readonly RegistryWorkflowInfo[], +): WorkflowInfo[] { + return workflows.map(publicWorkflowInfo); +} + function readVersion(): string { try { const pkg = JSON.parse( @@ -525,8 +597,8 @@ export const startServer = async ( * so it is refreshed even when the stable slug is already present. * Resolves all lookups in parallel. Never mutates the registry. */ const enrichWorkflows = async ( - workflows: WorkflowInfo[], - ): Promise => { + workflows: RegistryWorkflowInfo[], + ): Promise => { return Promise.all( workflows.map(async (workflow) => { if (workflow.definitionId == null) return workflow; @@ -640,13 +712,111 @@ export const startServer = async ( } catch (err) { console.error("[harness] recent-dirs prune failed:", err); } - let workflowsCache: WorkflowInfo[] = await workflowRegistry.list(); - const workflowsCacheTimer = setInterval(() => { - void workflowRegistry.list().then((list) => { - workflowsCache = list; + let workflowsCache: RegistryWorkflowInfo[] = await workflowRegistry.list(); + const initialInventorySnapshot = + await workflowRegistry.inventorySnapshot(launchDir); + type AcceptedCanonicalWorkflowRoot = { + workflowPath: string; + canonicalRoot: string; + identityEvidence: WorkflowIdentityEvidence; + }; + let acceptedInventoryGeneration = initialInventorySnapshot.generation; + let acceptedCanonicalWorkflowRoots: AcceptedCanonicalWorkflowRoot[] = + initialInventorySnapshot.canonicalWorkflowRoots.map((entry) => ({ + ...entry, + })); + let acceptedSourceObservations: WorkflowSourceObservation[] = + initialInventorySnapshot.sourceObservations.map((entry) => ({ + ...entry, + paths: [...entry.paths], + })); + const acceptedScopeStatusByCanonicalRoot = new Map< + string, + "complete" | "degraded" + >([ + [ + initialInventorySnapshot.canonicalScopeRoot, + initialInventorySnapshot.status, + ], + ]); + const acceptedCanonicalScopeByLexicalRoot = new Map([ + [ + resolve(expandHome(launchDir)), + initialInventorySnapshot.canonicalScopeRoot, + ], + [ + initialInventorySnapshot.canonicalScopeRoot, + initialInventorySnapshot.canonicalScopeRoot, + ], + ]); + + const acceptedCanonicalScopeRoot = (root: string): string => { + const lexicalRoot = resolve(expandHome(root)); + return ( + acceptedCanonicalScopeByLexicalRoot.get(lexicalRoot) ?? + canonicalGraphPath(lexicalRoot) + ); + }; + const acceptedInventorySnapshot = (scope: WorkspaceScope) => { + const canonicalScopeRoot = acceptedCanonicalScopeRoot(scope.root); + return { + workflows: workflowsCache, + status: + acceptedScopeStatusByCanonicalRoot.get(canonicalScopeRoot) ?? + ("degraded" as const), + generation: acceptedInventoryGeneration, + canonicalScopeRoot, + canonicalWorkflowRoots: acceptedCanonicalWorkflowRoots, + sourceObservations: acceptedSourceObservations, + }; + }; + + const acceptedSourceObservationsForRoot = ( + root: string, + ): WorkflowSourceObservation[] => { + const canonicalRoot = acceptedCanonicalScopeRoot(root); + return sourceObservationsWithinScope(canonicalRoot, [ + ...acceptedSourceObservations, + ...systemGraphInvocations.invocationObservations(), + ]); + }; + + const markAcceptedInventoryDirty = (root: string): void => { + const canonicalRoot = acceptedCanonicalScopeRoot(root); + let changed = false; + let sawIntersectingStatus = false; + for (const [scopeRoot, status] of acceptedScopeStatusByCanonicalRoot) { + if ( + isWithinGraphPath(scopeRoot, canonicalRoot) || + isWithinGraphPath(canonicalRoot, scopeRoot) + ) { + sawIntersectingStatus = true; + if (status !== "degraded") { + acceptedScopeStatusByCanonicalRoot.set(scopeRoot, "degraded"); + changed = true; + } + } + } + if (!sawIntersectingStatus) { + acceptedScopeStatusByCanonicalRoot.set(canonicalRoot, "degraded"); + changed = true; + } + const nextRoots = acceptedCanonicalWorkflowRoots.map((entry) => { + if ( + entry.identityEvidence === "unknown" || + (!isWithinGraphPath(canonicalRoot, entry.canonicalRoot) && + !isWithinGraphPath(entry.canonicalRoot, canonicalRoot)) + ) { + return entry; + } + changed = true; + return { ...entry, identityEvidence: "unknown" as const }; }); - }, WORKFLOWS_CACHE_REFRESH_MS); - workflowsCacheTimer.unref?.(); + if (changed) { + acceptedCanonicalWorkflowRoots = nextRoots; + acceptedInventoryGeneration += 1; + } + }; const boundWorkflowForSession = ( session: HarnessSession, @@ -667,11 +837,32 @@ export const startServer = async ( */ const writeSessionContext = async ( session: HarnessSession, + workflows: WorkflowInfo[] = workflowsCache, + isCurrent?: () => boolean, ): Promise => { - await writeHarnessContext( + const boundWorkflow = session.boundWorkflowPath + ? (workflows.find( + (workflow) => workflow.path === session.boundWorkflowPath, + ) ?? null) + : null; + await writeHarnessContext(session, boundWorkflow, workflows, isCurrent); + }; + + const stageAcceptedSessionContext = async ( + session: HarnessSession, + workflows: WorkflowInfo[], + isCurrent: () => boolean, + ): Promise => { + const boundWorkflow = session.boundWorkflowPath + ? (workflows.find( + (workflow) => workflow.path === session.boundWorkflowPath, + ) ?? null) + : null; + return stageHarnessContextForPublication( session, - boundWorkflowForSession(session), - workflowsCache, + boundWorkflow, + workflows, + isCurrent, ); }; @@ -856,13 +1047,35 @@ export const startServer = async ( ...(await loadSettings(statePaths.settings)).recentDirs, ...sessionManager.list().map((session) => session.cwd), ]); - const systemGraphRelationships = new CachedAgentRelationshipProvider( - new SourceAgentRelationshipProvider(), - ); const activeSystemGraphScopes = new Map(); + const systemGraphInvocations = new CachedAgentInvocationProvider( + new SourceAgentInvocationProvider(), + undefined, + { + onChange: (sourceRoots) => { + const canonicalSourceRoots = sourceRoots.map(canonicalGraphPath); + for (const scope of activeSystemGraphScopes.values()) { + const canonicalScope = { + workspaceKey: scope.workspaceKey, + root: canonicalGraphPath(scope.root), + }; + if ( + systemGraphStore.peek(canonicalScope.workspaceKey) && + canonicalSourceRoots.some((sourceRoot) => + isWithinGraphPath(canonicalScope.root, sourceRoot), + ) + ) { + systemGraphStore.requestRefresh(canonicalScope); + } + } + }, + }, + ); const systemGraphInventory = new HarnessRegistryInventoryProvider({ listWorkflows: () => workflowsCache, - inspectManifestName, + inventorySnapshot: acceptedInventorySnapshot, + inspectManifestName: (sourceRoot, extractionOptions) => + inspectManifestName(sourceRoot, undefined, extractionOptions), onIdentityChange: (sourceRoots) => { const canonicalSourceRoots = sourceRoots.map(canonicalGraphPath); for (const scope of activeSystemGraphScopes.values()) { @@ -882,10 +1095,7 @@ export const startServer = async ( }, }); const systemGraphStore = new SystemGraphStore( - new StaticSystemGraphBuilder( - systemGraphInventory, - systemGraphRelationships, - ), + new StaticSystemGraphBuilder(systemGraphInventory, systemGraphInvocations), { onChange: ({ workspaceKey, revision, state }) => { bus.publish({ @@ -1036,40 +1246,22 @@ export const startServer = async ( // marker disappears or becomes invalid; manually connected folders remain. const rescanWorkspaceForSession = async ( harnessSessionId: string, + dirty = true, + sourceRoots: readonly string[] | null = null, ): Promise => { const session = sessionManager.get(harnessSessionId); if (!session || session.status === "exited") return; - const before = workflowsCache; - await workflowRegistry.prune(); - const rescanBudget = new AgentProjectScanBudget(); - const rescanFound = await workflowRegistry.scan(session.cwd, rescanBudget); - logAgentScan( - "workspace-change", - session.cwd, - rescanFound.length, - rescanBudget, + const roots = [...(sourceRoots ?? []), session.cwd]; + const discoveryBudget = workspaceDiscoveryBudget(session.cwd); + await Promise.all( + [...new Set(roots)].map((root) => + scanWorkflowsAndBroadcast(root, "workspace-change", { + dirty, + discoveryBudget, + }), + ), ); - const after = await workflowRegistry.list(); - workflowsCache = after; - const inventoryChanged = !workflowListsEqual(before, after); - if (inventoryChanged) { - await refreshSystemGraphScopesForRoot(session.cwd); - } - - // A removed project must not leave a live session permanently bound to a - // path that the registry can no longer resolve. Clear only stale bindings; - // the triggering session may immediately auto-bind to another candidate - // below its cwd in the block that follows. - const registeredPaths = new Set(after.map((workflow) => workflow.path)); - for (const openSession of sessionManager.list()) { - if ( - openSession.status !== "exited" && - openSession.boundWorkflowPath && - !registeredPaths.has(openSession.boundWorkflowPath) - ) { - sessionManager.setBoundWorkflowPath(openSession.id, null); - } - } + const after = workflowsCache; // Auto-bind: if this session is still unbound, find the workflow at or // directly under its cwd and bind it — same mechanism as @@ -1092,20 +1284,55 @@ export const startServer = async ( // so the session reference we already hold already carries the new // binding — pass it directly, same as the PATCH handler does. await writeSessionContext(session); - await renderCanvas(session).catch((err: unknown) => { + await autoRenderCanvas(session).catch((err: unknown) => { console.error("[harness] auto-bind canvas render failed:", err); }); } } - - if (!inventoryChanged) return; - await Promise.all(sessionManager.list().map((s) => writeSessionContext(s))); - bus.publish({ type: "workflows.changed" }); }; + const sharedWorkspaceWatchBroker = new SharedWorkspaceWatchBroker({ + onLastLeaseReleased: (root) => { + if (!coordinatorActive) return; + // Losing continuous observation invalidates freshness, but it is not a + // filesystem mutation and must not manufacture a trailing destructive + // scan after the final watcher has gone away. Supersede any publication + // based on pre-retirement proof and retain the last-known rows degraded; + // the next lease's normal background scan will reconcile the interval. + supersedePublication(); + coordinatorEpoch += 1; + systemGraphInventory.invalidateScope(root); + systemGraphInvocations.invalidateScope(root); + workflowRegistry.markDiscoveryDirty(root); + markAcceptedInventoryDirty(root); + }, + }); const workspaceWatcher = new WorkspaceWatcherManager({ - onChange: (harnessSessionId) => { - rescanWorkspaceForSession(harnessSessionId).catch((err: unknown) => { + sharedWatchBroker: sharedWorkspaceWatchBroker, + listSourceRoots: (_harnessSessionId, cwd) => + graphSourceRootsWithinScope( + cwd, + workflowsCache.map((workflow) => workflow.path), + ), + listSourceObservations: (_harnessSessionId, cwd) => + acceptedSourceObservationsForRoot(cwd), + onPotentialChange: (harnessSessionId) => { + const session = sessionManager.get(harnessSessionId); + if (session && session.status !== "exited") { + prepareDirtyWorkflowRoot( + session.cwd, + undefined, + workspaceDiscoveryBudget(session.cwd), + ); + } + }, + onChange: (harnessSessionId, sourceRoots) => { + return rescanWorkspaceForSession( + harnessSessionId, + true, + sourceRoots, + ).catch((err: unknown) => { console.error("[harness] workspace rescan failed:", err); + throw err; }); }, }); @@ -1236,62 +1463,673 @@ export const startServer = async ( // nobody had opened, and reconstructing which root did it took a filesystem // archaeology session. Now every scan says its root, its reason, what it // found, what it cost, and what it declined to enter. - const scanWorkflowsAndBroadcast = async ( - scanRoot: string, - reason: WorkflowScanReason, - options: { refreshGraphs?: () => Promise } = {}, - ): Promise => { - // The registry keys rows by path, so two spellings of one directory - // register the same agent twice. Only the graph-refresh caller resolved - // symlinks: booting under a symlinked launch dir (macOS `os.tmpdir()` is - // `/var/...` for `/private/var/...`) registered every agent under the - // symlinked path, then the first graph open registered them all again - // under the real one. The duplicates collided into `local:` fallback keys, - // so every cross-agent target became ambiguous and its edge vanished. - // Resolved here rather than inside the registry: registry paths are - // compared by exact string elsewhere (a session auto-binds on - // `path === cwd`, index.ts:1057), so rewriting stored paths unbinds them. - const root = canonicalGraphPath(scanRoot); - const before = workflowsCache; - const budget = new AgentProjectScanBudget(); - const found = await workflowRegistry.scan(root, budget); - logAgentScan(reason, root, found.length, budget); - const after = await workflowRegistry.list(); - workflowsCache = after; - const changed = !workflowListsEqual(before, after); - // Rewrite every open session's context file before broadcasting — a - // listener reacting to workflows.changed (the SPA, or an agent that - // happens to re-read the file right then) must never see the - // notification before the file it describes is actually updated. - if (changed) { - await Promise.all( - sessionManager.list().map((session) => writeSessionContext(session)), + interface CoordinatedScanResult { + found: WorkflowInfo[]; + repositoryBoundaries: string[]; + } + interface ScanFlight { + canonicalRoot: string; + lexicalRoot: string; + token: string; + generation: number; + acceptedGeneration: number; + acceptedChanged: boolean; + pending: boolean; + dirty: boolean; + reason: WorkflowScanReason; + discoveryBudget: WorkspaceDiscoveryBudget; + promise: Promise; + } + interface WorkspaceDiscoveryBudget { + project: AgentProjectScanAllowance; + source: AgentSourceScanBudget; + } + interface PublicationWaiter { + promise: Promise; + resolve: (published: boolean) => void; + reject: (error: unknown) => void; + } + const scanFlights = new Map(); + const outstandingDirtyPrerequisites = new Map(); + let coordinatorEpoch = 0; + let mutationTokenSequence = 0; + let coordinatorActive = true; + let publicationWaiter: PublicationWaiter | null = null; + let publicationQueue: Promise = Promise.resolve(); + const sameTurnDirtyPreparations = new Map< + string, + { canonicalRoot: string; lexicalRoot: string; token: string } + >(); + const sameTurnDiscoveryBudgets = new Map(); + const workspaceDiscoveryBudget = (root: string): WorkspaceDiscoveryBudget => { + const canonicalRoot = canonicalGraphPath(resolve(expandHome(root))); + const existing = sameTurnDiscoveryBudgets.get(canonicalRoot); + if (existing) return existing; + const created = { + project: new AgentProjectScanAllowance(), + source: new AgentSourceScanBudget(), + }; + sameTurnDiscoveryBudgets.set(canonicalRoot, created); + queueMicrotask(() => { + if (sameTurnDiscoveryBudgets.get(canonicalRoot) === created) { + sameTurnDiscoveryBudgets.delete(canonicalRoot); + } + }); + return created; + }; + + const intersectingGraphScopes = (changedRoot: string): WorkspaceScope[] => { + const canonicalChangedRoot = canonicalGraphPath(changedRoot); + const scopes: WorkspaceScope[] = []; + for (const scope of activeSystemGraphScopes.values()) { + const canonicalScope = { + workspaceKey: scope.workspaceKey, + root: canonicalGraphPath(scope.root), + }; + if ( + isWithinGraphPath(canonicalScope.root, canonicalChangedRoot) || + isWithinGraphPath(canonicalChangedRoot, canonicalScope.root) + ) { + scopes.push(canonicalScope); + } + } + return scopes; + }; + const staleSystemGraphScopesForRoot = ( + changedRoot: string, + token: string, + ): void => { + for (const scope of intersectingGraphScopes(changedRoot)) { + systemGraphStore.markStale(scope, token); + } + }; + const releaseSystemGraphPrerequisite = (token: string): void => { + // A symlink scope can be retargeted by the accepted scan, so release by + // token rather than recomputing containment against its former target. + for (const scope of activeSystemGraphScopes.values()) { + systemGraphStore.releasePrerequisite( + { + workspaceKey: scope.workspaceKey, + root: canonicalGraphPath(scope.root), + }, + token, ); } - if (options.refreshGraphs) await options.refreshGraphs(); - else if (changed) refreshSystemGraphScopesForRoot(root); - if (changed) bus.publish({ type: "workflows.changed" }); - return found; + outstandingDirtyPrerequisites.delete(token); }; - - const refreshSystemGraphInventory = async (scope: WorkspaceScope) => { + const cancelSystemGraphPrerequisite = (token: string): void => { + for (const scope of activeSystemGraphScopes.values()) { + systemGraphStore.cancelPrerequisite(scope.workspaceKey, token); + } + outstandingDirtyPrerequisites.delete(token); + }; + const attachOutstandingPrerequisites = (scope: WorkspaceScope): void => { const canonicalScope = { workspaceKey: scope.workspaceKey, root: canonicalGraphPath(scope.root), }; - let refreshPromise: Promise | null = null; - const refreshGraph = (): Promise => { - refreshSystemGraphScopesForRoot( - canonicalScope.root, - canonicalScope.workspaceKey, - ); - refreshPromise = systemGraphStore.refresh(canonicalScope); - return refreshPromise.then(() => undefined); + for (const [token, dirtyRoot] of outstandingDirtyPrerequisites) { + if ( + isWithinGraphPath(canonicalScope.root, dirtyRoot) || + isWithinGraphPath(dirtyRoot, canonicalScope.root) + ) { + systemGraphStore.markStale(canonicalScope, token); + } + } + }; + const reportSystemGraphRefreshFailure = (changedRoot: string): void => { + for (const scope of intersectingGraphScopes(changedRoot)) { + systemGraphStore.reportRefreshFailure(scope); + } + }; + const allFlightsAccepted = (): boolean => + [...scanFlights.values()].every( + (flight) => + !flight.pending && flight.acceptedGeneration === flight.generation, + ); + const getPublicationWaiter = (): PublicationWaiter => { + if (publicationWaiter) return publicationWaiter; + let resolve!: (published: boolean) => void; + let reject!: (error: unknown) => void; + const promise = new Promise((resolvePromise, rejectPromise) => { + resolve = resolvePromise; + reject = rejectPromise; + }); + publicationWaiter = { promise, resolve, reject }; + return publicationWaiter; + }; + const supersedePublication = (): void => { + const waiter = publicationWaiter; + if (!waiter) return; + publicationWaiter = null; + waiter.resolve(false); + }; + const rejectPublication = (error: unknown): void => { + const waiter = publicationWaiter; + if (!waiter) return; + publicationWaiter = null; + waiter.reject(error); + }; + const prepareDirtyWorkflowRoot = ( + root: string, + tokenOverride?: string, + discoveryBudget?: WorkspaceDiscoveryBudget, + ): { canonicalRoot: string; lexicalRoot: string; token: string } => { + const lexicalRoot = resolve(expandHome(root)); + const canonicalRoot = canonicalGraphPath(lexicalRoot); + if (!tokenOverride) { + const existingPreparation = sameTurnDirtyPreparations.get(canonicalRoot); + if (existingPreparation) return existingPreparation; + } + const token = tokenOverride ?? `inventory:${canonicalRoot}`; + supersedePublication(); + coordinatorEpoch += 1; + systemGraphInventory.invalidateScope(lexicalRoot); + systemGraphInvocations.invalidateScope(lexicalRoot); + workflowRegistry.markDiscoveryDirty(lexicalRoot); + markAcceptedInventoryDirty(lexicalRoot); + outstandingDirtyPrerequisites.set(token, canonicalRoot); + staleSystemGraphScopesForRoot(lexicalRoot, token); + const currentFlight = scanFlights.get(canonicalRoot); + if (currentFlight && !currentFlight.pending) { + currentFlight.generation += 1; + currentFlight.acceptedGeneration = 0; + currentFlight.acceptedChanged = false; + currentFlight.pending = true; + currentFlight.dirty = true; + // Every edit generation gets fresh memoization/counters. Reusing the + // prior AgentSourceScanBudget can return old file promises after a raw + // save, while an exhausted project allowance makes the trailing proof a + // permanent false-negative. + currentFlight.discoveryBudget = + discoveryBudget ?? workspaceDiscoveryBudget(lexicalRoot); + } + const prepared = { canonicalRoot, lexicalRoot, token }; + if (!tokenOverride) { + sameTurnDirtyPreparations.set(canonicalRoot, prepared); + queueMicrotask(() => { + if (sameTurnDirtyPreparations.get(canonicalRoot) === prepared) { + sameTurnDirtyPreparations.delete(canonicalRoot); + } + }); + } + return prepared; + }; + + const requestAcceptedPublication = (): Promise => { + const waiter = getPublicationWaiter(); + const attempt = publicationQueue + .catch(() => {}) + .then(async () => { + if ( + publicationWaiter !== waiter || + !coordinatorActive || + !allFlightsAccepted() + ) { + return; + } + const epoch = coordinatorEpoch; + const isCurrent = (): boolean => + publicationWaiter === waiter && + coordinatorActive && + coordinatorEpoch === epoch && + allFlightsAccepted(); + try { + await options.workflowDiscoveryTestHooks?.beforePublication?.({ + epoch, + roots: [...scanFlights.values()].map( + (flight) => flight.lexicalRoot, + ), + }); + if (!isCurrent()) return; + const snapshotRoots = [ + ...new Set([ + launchDir, + ...[...scanFlights.values()].map((flight) => flight.lexicalRoot), + ...[...activeSystemGraphScopes.values()].map( + (scope) => scope.root, + ), + ]), + ]; + const snapshots: Array<{ + lexicalRoot: string; + snapshot: Awaited< + ReturnType + >; + }> = []; + for (const root of snapshotRoots) { + snapshots.push({ + lexicalRoot: resolve(expandHome(root)), + snapshot: await workflowRegistry.inventorySnapshot(root), + }); + if (!isCurrent()) return; + } + const snapshot = snapshots[0]!.snapshot; + const before = workflowsCache; + const after = [...snapshot.workflows]; + const rowsChanged = !workflowListsEqual(before, after); + const acceptedFlights = [...scanFlights.values()]; + const nextCanonicalWorkflowRoots = + snapshot.canonicalWorkflowRoots.map((entry) => ({ ...entry })); + const nextSourceObservations = snapshot.sourceObservations.map( + (entry) => ({ ...entry, paths: [...entry.paths] }), + ); + const nextScopeStatuses = new Map(acceptedScopeStatusByCanonicalRoot); + const nextCanonicalScopeByLexicalRoot = new Map( + acceptedCanonicalScopeByLexicalRoot, + ); + for (const entry of snapshots) { + nextCanonicalScopeByLexicalRoot.set( + entry.lexicalRoot, + entry.snapshot.canonicalScopeRoot, + ); + nextCanonicalScopeByLexicalRoot.set( + entry.snapshot.canonicalScopeRoot, + entry.snapshot.canonicalScopeRoot, + ); + nextScopeStatuses.set( + entry.snapshot.canonicalScopeRoot, + entry.snapshot.status, + ); + } + const inventoryProjectionChanged = + JSON.stringify(acceptedCanonicalWorkflowRoots) !== + JSON.stringify(nextCanonicalWorkflowRoots) || + JSON.stringify(acceptedSourceObservations) !== + JSON.stringify(nextSourceObservations) || + JSON.stringify([...acceptedScopeStatusByCanonicalRoot].sort()) !== + JSON.stringify([...nextScopeStatuses].sort()) || + JSON.stringify([...acceptedCanonicalScopeByLexicalRoot].sort()) !== + JSON.stringify([...nextCanonicalScopeByLexicalRoot].sort()); + let stagedContexts: StagedHarnessContext[] = []; + if (rowsChanged) { + let contextsStable = false; + for (let pass = 0; pass < 3; pass += 1) { + for (const staged of stagedContexts) staged.discard(); + stagedContexts = []; + const sessions = sessionManager + .list() + .filter((session) => session.status !== "exited") + .sort((left, right) => left.id.localeCompare(right.id)); + const sessionProjection = JSON.stringify( + sessions.map((session) => [ + session.id, + session.cwd, + session.harness, + session.boundWorkflowPath, + ]), + ); + const stageResults = await Promise.allSettled( + sessions.map((session) => + stageAcceptedSessionContext(session, after, isCurrent), + ), + ); + for (const [index, result] of stageResults.entries()) { + if (result.status === "fulfilled") { + if (result.value) stagedContexts.push(result.value); + continue; + } + console.error( + `[harness] failed to stage accepted context for ${sessions[index]?.id ?? "unknown session"}:`, + result.reason, + ); + } + if (!isCurrent()) { + for (const staged of stagedContexts) staged.discard(); + return; + } + await options.workflowDiscoveryTestHooks?.afterContextStaging?.({ + epoch, + sessionIds: sessions.map((session) => session.id), + }); + if (!isCurrent()) { + for (const staged of stagedContexts) staged.discard(); + return; + } + const currentSessionProjection = JSON.stringify( + sessionManager + .list() + .filter((session) => session.status !== "exited") + .sort((left, right) => left.id.localeCompare(right.id)) + .map((session) => [ + session.id, + session.cwd, + session.harness, + session.boundWorkflowPath, + ]), + ); + if (currentSessionProjection === sessionProjection) { + contextsStable = true; + break; + } + } + if (!contextsStable) { + for (const staged of stagedContexts) staged.discard(); + coordinatorEpoch += 1; + supersedePublication(); + return; + } + } + if (!isCurrent()) { + for (const staged of stagedContexts) staged.discard(); + return; + } + + // From this point through event publication there is no await: the + // cache, bindings, graph prerequisites, and bus frame become one + // accepted process-local snapshot before another raw signal can run. + for (const staged of stagedContexts) { + try { + staged.commit(); + } catch (error) { + staged.discard(); + console.error( + `[harness] failed to commit accepted context ${staged.filePath}:`, + error, + ); + } + } + workflowsCache = after; + acceptedCanonicalWorkflowRoots = nextCanonicalWorkflowRoots; + acceptedSourceObservations = nextSourceObservations; + acceptedScopeStatusByCanonicalRoot.clear(); + for (const [scopeRoot, status] of nextScopeStatuses) { + acceptedScopeStatusByCanonicalRoot.set(scopeRoot, status); + } + acceptedCanonicalScopeByLexicalRoot.clear(); + for (const [ + lexicalRoot, + canonicalRoot, + ] of nextCanonicalScopeByLexicalRoot) { + acceptedCanonicalScopeByLexicalRoot.set(lexicalRoot, canonicalRoot); + } + if (rowsChanged || inventoryProjectionChanged) { + acceptedInventoryGeneration += 1; + } + if (rowsChanged) { + const registeredPaths = new Set( + after.map((workflow) => workflow.path), + ); + for (const openSession of sessionManager.list()) { + if ( + openSession.status !== "exited" && + openSession.boundWorkflowPath && + !registeredPaths.has(openSession.boundWorkflowPath) + ) { + sessionManager.setBoundWorkflowPath(openSession.id, null); + } + } + } + if (rowsChanged || inventoryProjectionChanged) { + // A scan prunes confirmed-missing rows registry-wide, not only below + // its requested root. Refresh every active projection so an + // unrelated workspace cannot retain a ghost node/navigation target. + for (const scope of activeSystemGraphScopes.values()) { + systemGraphStore.requestRefresh({ + workspaceKey: scope.workspaceKey, + root: canonicalGraphPath(scope.root), + }); + } + } else { + for (const flight of acceptedFlights) { + if (flight.acceptedChanged && !flight.dirty) { + refreshSystemGraphScopesForRoot(flight.lexicalRoot); + } + } + } + const acceptedCanonicalRoots = new Set( + acceptedFlights.map((flight) => flight.canonicalRoot), + ); + for (const [token, dirtyRoot] of [...outstandingDirtyPrerequisites]) { + // A terminal dirty attempt deliberately keeps its token armed. A + // later ordinary scan of that exact root inherits the proof by + // publication: once its newest generation is in this quiescent + // accepted snapshot, release every producer token for that root. + if (acceptedCanonicalRoots.has(dirtyRoot)) { + releaseSystemGraphPrerequisite(token); + } + } + if (rowsChanged) bus.publish({ type: "workflows.changed" }); + publicationWaiter = null; + waiter.resolve(true); + } catch (error) { + if (publicationWaiter === waiter) { + publicationWaiter = null; + waiter.reject(error); + } + } + }); + publicationQueue = attempt.then( + () => undefined, + () => undefined, + ); + return waiter.promise; + }; + + const scanWorkflowsAndBroadcast = ( + root: string, + reason: WorkflowScanReason, + scanOptions: { + dirty?: boolean; + discoveryBudget?: WorkspaceDiscoveryBudget; + } = {}, + ): Promise => { + if (!coordinatorActive) { + return Promise.reject(new Error("Agent discovery coordinator is closed")); + } + const prepared = scanOptions.dirty + ? prepareDirtyWorkflowRoot(root) + : { + lexicalRoot: resolve(expandHome(root)), + canonicalRoot: canonicalGraphPath(resolve(expandHome(root))), + token: `inventory:${canonicalGraphPath(resolve(expandHome(root)))}`, + }; + const { lexicalRoot, canonicalRoot, token } = prepared; + if (!scanOptions.dirty) { + supersedePublication(); + coordinatorEpoch += 1; + } + const existing = scanFlights.get(canonicalRoot); + if (existing) { + if (!existing.pending) { + existing.generation += 1; + existing.acceptedGeneration = 0; + existing.acceptedChanged = false; + existing.discoveryBudget = + scanOptions.discoveryBudget ?? workspaceDiscoveryBudget(lexicalRoot); + } + existing.pending = true; + existing.dirty ||= scanOptions.dirty === true; + if (scanOptions.discoveryBudget) { + existing.discoveryBudget = scanOptions.discoveryBudget; + } + existing.lexicalRoot = lexicalRoot; + existing.reason = reason; + return existing.promise; + } + + const flight: ScanFlight = { + canonicalRoot, + lexicalRoot, + token, + generation: 1, + acceptedGeneration: 0, + acceptedChanged: false, + pending: false, + dirty: scanOptions.dirty === true, + reason, + discoveryBudget: + scanOptions.discoveryBudget ?? workspaceDiscoveryBudget(lexicalRoot), + promise: Promise.resolve({ + found: [] as WorkflowInfo[], + repositoryBoundaries: [] as string[], + }), }; - await scanWorkflowsAndBroadcast(canonicalScope.root, "graph-refresh", { - refreshGraphs: refreshGraph, + flight.promise = (async () => { + let retries = 0; + let retryGeneration = 0; + while (coordinatorActive) { + // Shared watcher fanout invokes session and graph subscribers in the + // same turn. Let every sibling register/coalesce before one pass + // captures the generation; a genuinely later edit still increments + // it during the held scan and gets exactly one trailing pass. + await Promise.resolve(); + const generation = flight.generation; + const discoveryBudget = flight.discoveryBudget; + if (retryGeneration !== generation) { + retryGeneration = generation; + retries = 0; + } + const passReason = flight.reason; + flight.pending = false; + try { + await options.workflowDiscoveryTestHooks?.beforeScan?.({ + root: flight.lexicalRoot, + reason: passReason, + generation, + }); + if (!coordinatorActive) { + throw new Error("Agent discovery coordinator is closed"); + } + const outcome = await workflowRegistry.scanDetailed( + flight.lexicalRoot, + new AgentProjectScanBudget({}, discoveryBudget.project), + discoveryBudget.source, + ); + await options.workflowDiscoveryTestHooks?.afterScan?.({ + root: flight.lexicalRoot, + reason: passReason, + generation, + }); + logAgentScan( + passReason, + flight.lexicalRoot, + outcome.found.length, + outcome.budget, + outcome.sourceBudget, + ); + if (!coordinatorActive) { + throw new Error("Agent discovery coordinator is closed"); + } + if (generation !== flight.generation || flight.pending) { + continue; + } + flight.acceptedGeneration = generation; + flight.acceptedChanged = outcome.changed; + let published = false; + while ( + !published && + generation === flight.generation && + !flight.pending + ) { + published = await requestAcceptedPublication(); + } + if (!published) continue; + if (generation !== flight.generation || flight.pending) { + continue; + } + const acceptedFoundByPath = new Map(); + for (const found of outcome.found) { + const exact = workflowsCache.find( + (workflow) => workflow.path === found.path, + ); + const foundCanonicalRoot = canonicalGraphPath(found.path); + const canonicalEntry = acceptedCanonicalWorkflowRoots.find( + (entry) => + entry.workflowPath === exact?.path || + entry.canonicalRoot === foundCanonicalRoot, + ); + const accepted = + exact ?? + (canonicalEntry + ? workflowsCache.find( + (workflow) => workflow.path === canonicalEntry.workflowPath, + ) + : undefined); + if ( + accepted && + canonicalEntry && + (canonicalEntry.identityEvidence === "marker" || + canonicalEntry.identityEvidence === "source") + ) { + acceptedFoundByPath.set(accepted.path, accepted); + } + } + return { + found: [...acceptedFoundByPath.values()], + repositoryBoundaries: outcome.repositoryBoundaries, + }; + } catch (error) { + flight.acceptedGeneration = 0; + flight.acceptedChanged = false; + if (!coordinatorActive) throw error; + if (generation !== flight.generation || flight.pending) { + continue; + } + if (retries < 3) { + const delay = 50 * 2 ** retries; + retries += 1; + await new Promise((resolve) => setTimeout(resolve, delay)); + if (generation === flight.generation && !flight.pending) { + flight.discoveryBudget = workspaceDiscoveryBudget( + flight.lexicalRoot, + ); + } + continue; + } + reportSystemGraphRefreshFailure(flight.lexicalRoot); + supersedePublication(); + throw error; + } + } + throw new Error("Agent discovery coordinator is closed"); + })().finally(() => { + if (scanFlights.get(canonicalRoot) === flight) { + scanFlights.delete(canonicalRoot); + if (publicationWaiter) { + void requestAcceptedPublication().catch(() => { + // The owning accepted flight observes publication failures. + }); + } + } }); - return await (refreshPromise ?? systemGraphStore.refresh(canonicalScope)); + scanFlights.set(canonicalRoot, flight); + return flight.promise; + }; + + const refreshSystemGraphInventory = async ( + scope: WorkspaceScope, + includeRetainedRoots = true, + ) => { + const canonicalScope = { + workspaceKey: scope.workspaceKey, + root: canonicalGraphPath(scope.root), + }; + const roots = includeRetainedRoots + ? [ + ...graphSourceRootsWithinScope( + scope.root, + workflowsCache.map((workflow) => workflow.path), + ), + scope.root, + ] + : [scope.root]; + const discoveryBudget = workspaceDiscoveryBudget(scope.root); + await Promise.all( + [...new Set(roots)].map((root) => + scanWorkflowsAndBroadcast(root, "graph-refresh", { + dirty: true, + discoveryBudget, + }), + ), + ); + const refreshed = await systemGraphStore.waitForCurrentRefresh( + canonicalScope.workspaceKey, + ); + if (!refreshed) { + throw new Error("Workspace graph scope retired during refresh"); + } + return refreshed; }; const workflowRootsForGraphScope = (scope: WorkspaceScope): string[] => @@ -1300,35 +2138,76 @@ export const startServer = async ( workflowsCache.map((workflow) => workflow.path), ); - const systemGraphWatcher = new SystemGraphWatcherManager({ - listSourceRoots: workflowRootsForGraphScope, - onSourceChange: (scope, sourcePaths) => { - const canonicalScope = { - workspaceKey: scope.workspaceKey, - root: canonicalGraphPath(scope.root), - }; - const dirtyRoots = dirtyGraphSourceRoots( - canonicalScope.root, - workflowsCache.map((workflow) => workflow.path), - sourcePaths, - ); - if (dirtyRoots.length === 0) return; - for (const workflowRoot of dirtyRoots) { - systemGraphInventory.invalidateSource(workflowRoot); - systemGraphRelationships.invalidateSource(workflowRoot); - } - systemGraphStore.requestRefresh(canonicalScope); - }, - onInventoryChange: async (scope) => { - try { - await refreshSystemGraphInventory(scope); - } catch (err) { - systemGraphStore.reportRefreshFailure(scope); - console.error("[harness] workspace graph inventory refresh failed"); - throw err; - } + const systemGraphWatcher = new SystemGraphWatcherManager( + { + listSourceRoots: workflowRootsForGraphScope, + listSourceObservations: (scope) => + acceptedSourceObservationsForRoot(scope.root), + onPotentialChange: (scope, sourcePaths) => { + const discoveryBudget = workspaceDiscoveryBudget(scope.root); + prepareDirtyWorkflowRoot(scope.root, undefined, discoveryBudget); + if (sourcePaths === null) { + systemGraphInventory.invalidateScope(scope.root); + systemGraphInvocations.invalidateScope(scope.root); + } else { + for (const root of dirtyGraphSourceRoots( + scope.root, + workflowsCache.map((workflow) => workflow.path), + sourcePaths, + )) { + prepareDirtyWorkflowRoot(root, undefined, discoveryBudget); + systemGraphInventory.invalidateSource(root); + systemGraphInvocations.invalidateSource(root); + } + } + }, + onSourceChange: async (scope, sourcePaths) => { + const canonicalScope = { + workspaceKey: scope.workspaceKey, + root: canonicalGraphPath(scope.root), + }; + const dirtyRoots = + sourcePaths === null + ? workflowRootsForGraphScope(scope) + : dirtyGraphSourceRoots( + canonicalScope.root, + workflowsCache.map((workflow) => workflow.path), + sourcePaths, + ); + if (sourcePaths === null) { + systemGraphInventory.invalidateScope(canonicalScope.root); + systemGraphInvocations.invalidateScope(canonicalScope.root); + } else { + for (const workflowRoot of dirtyRoots) { + systemGraphInventory.invalidateSource(workflowRoot); + systemGraphInvocations.invalidateSource(workflowRoot); + } + } + const discoveryBudget = workspaceDiscoveryBudget(scope.root); + await Promise.all( + [...new Set([...dirtyRoots, scope.root])].map((root) => + scanWorkflowsAndBroadcast(root, "graph-refresh", { + dirty: true, + discoveryBudget, + }), + ), + ); + }, + onInventoryChange: async (scope) => { + try { + // A structural boundary event (for example `candidate/.git`) must + // first be reconciled from the containing scope. Treating every + // retained row as an explicit selection here would immediately + // direct-scan and resurrect the candidate the parent just retired. + await refreshSystemGraphInventory(scope, false); + } catch (err) { + console.error("[harness] workspace graph inventory refresh failed"); + throw err; + } + }, }, - }); + { sharedBroker: sharedWorkspaceWatchBroker }, + ); const listWorkspaceScopesAndRetain = async () => { const scopes = await workspaceScopeCatalog.list(); @@ -1362,6 +2241,32 @@ export const startServer = async ( return workflows; }; + const automaticCanvasAuthorized = async ( + session: Pick, + expectedWorkflowPath = session.boundWorkflowPath, + ): Promise => { + if ( + !expectedWorkflowPath || + session.boundWorkflowPath !== expectedWorkflowPath + ) { + return false; + } + const workflow = workflowsCache.find( + (candidate) => candidate.path === expectedWorkflowPath, + ); + if (!workflow) return false; + // A retained cloud definition link is explicit legacy authorization even + // when the local marker is no longer present. Marker-only rows require + // both fresh accepted proof and a hardened launch-time recheck so a queued + // automatic render cannot race an offline marker removal. + if (workflow.definitionId !== null) return true; + const evidence = acceptedCanonicalWorkflowRoots.find( + (entry) => entry.workflowPath === workflow.path, + )?.identityEvidence; + if (evidence !== "marker") return false; + return (await inspectAgentProjectMarker(workflow.path)).status === "valid"; + }; + // Renders a session's bound workflow via the fully deterministic pipeline — // against the live workflowsCache plus the bound definition's current cloud // build projection; structure + derived annotations, no LLM, no user token. @@ -1394,11 +2299,21 @@ export const startServer = async ( reactToRenderOutcome(session, outcome); }; const autoRenderCanvas = async (session: HarnessSession): Promise => { - const outcome = await renderCanvasForSession( - session, - await canvasWorkflowsForSession(session), - { preserveExistingOnFailure: true }, - ); + const workflowPath = session.boundWorkflowPath; + if (!(await automaticCanvasAuthorized(session, workflowPath))) return; + const workflows = await canvasWorkflowsForSession(session); + if (!(await automaticCanvasAuthorized(session, workflowPath))) return; + const outcome = await renderCanvasForSession(session, workflows, { + preserveExistingOnFailure: true, + authorizeBeforeExtraction: () => + automaticCanvasAuthorized(session, workflowPath), + beforeExtractionLaunchAuthorization: workflowPath + ? () => + options.workflowDiscoveryTestHooks?.beforeAutomaticCanvasLaunch?.( + workflowPath, + ) + : undefined, + }); reactToRenderOutcome(session, outcome); }; // Used only by the install-watcher timeout: forces extraction even with deps @@ -1408,11 +2323,21 @@ export const startServer = async ( const renderCanvasSurfacingDepErrors = async ( session: HarnessSession, ): Promise => { - const outcome = await renderCanvasForSession( - session, - await canvasWorkflowsForSession(session), - { surfaceErrorOnMissingDeps: true }, - ); + const workflowPath = session.boundWorkflowPath; + if (!(await automaticCanvasAuthorized(session, workflowPath))) return; + const workflows = await canvasWorkflowsForSession(session); + if (!(await automaticCanvasAuthorized(session, workflowPath))) return; + const outcome = await renderCanvasForSession(session, workflows, { + surfaceErrorOnMissingDeps: true, + authorizeBeforeExtraction: () => + automaticCanvasAuthorized(session, workflowPath), + beforeExtractionLaunchAuthorization: workflowPath + ? () => + options.workflowDiscoveryTestHooks?.beforeAutomaticCanvasLaunch?.( + workflowPath, + ) + : undefined, + }); reactToRenderOutcome(session, outcome); }; @@ -1421,7 +2346,10 @@ export const startServer = async ( "boot", ).catch((err: unknown) => { console.error("[harness] initial agent scan failed:", err); - return [] as WorkflowInfo[]; + return { + found: [] as WorkflowInfo[], + repositoryBoundaries: [] as string[], + }; }); // Boot-time retention sweep: keeps events.ndjson within the 50 MB / 30-day @@ -1540,7 +2468,8 @@ export const startServer = async ( organizationName: identity.organizationName, } : null, - listWorkflows: () => workflowRegistry.list().then(enrichWorkflows), + listWorkflows: async () => + publicWorkflowInfos(await enrichWorkflows(workflowsCache)), listWorkspaceScopes: listWorkspaceScopesAndRetain, listMacros: () => DEFAULT_MACROS, findWorkflow: (workflowPath) => @@ -1549,8 +2478,8 @@ export const startServer = async ( renderCanvas, onTelemetryOptInChange: (optIn) => batcher.setTelemetryOptIn(optIn), onSessionCreated: (cwd, harnessSessionId) => { - scanWorkflowsAndBroadcast(cwd, "session-create") - .then((found) => { + scanWorkflowsAndBroadcast(cwd, "session-create", { dirty: true }) + .then(({ found }) => { // Only auto-render when THIS session's own directory turned up a // workflow — an unrelated project scanned earlier elsewhere in // the registry shouldn't unprompt-render into a brand new, @@ -1592,15 +2521,36 @@ export const startServer = async ( scopeResolver: workspaceScopeCatalog, store: systemGraphStore, onScopeAccess: (scope) => { + const firstAccess = !activeSystemGraphScopes.has(scope.workspaceKey); activeSystemGraphScopes.set(scope.workspaceKey, scope); + // A destructive signal may predate this store entry (or arrive while + // the scope was retired). Attach every intersecting producer token + // synchronously before store.get can project old clickable inventory. + attachOutstandingPrerequisites(scope); + if (firstAccess) { + // A scope can be reopened after an interval with no continuous + // watcher lease. Its accepted rows remain useful for the immediate + // cache-backed graph, but pre-lease completeness/identity proof is + // no longer fresh enough to report ready or authorize legacy work. + markAcceptedInventoryDirty(scope.root); + void scanWorkflowsAndBroadcast(scope.root, "graph-refresh").catch( + (err: unknown) => { + console.error("[harness] workspace graph discovery failed:", err); + }, + ); + } return systemGraphWatcher.start(scope); }, onScopeRefresh: async (scope) => { try { systemGraphInventory.retryFailedInspections(scope); + systemGraphInvocations.retryFailed(scope.root); return await refreshSystemGraphInventory(scope); } catch { console.error("[harness] workspace graph manual refresh failed"); + if (!systemGraphStore.peek(scope.workspaceKey)) { + throw new Error("Workspace graph scope is no longer active"); + } return systemGraphStore.reportRefreshFailure(scope); } }, @@ -1625,25 +2575,49 @@ export const startServer = async ( // wrapped; scan/connect write through to the real registry untouched. Typed // as WorkflowRegistryLike so this wrapper needs no unsafe cast. const enrichedWorkflowRegistry: WorkflowRegistryLike = { - list: () => workflowRegistry.list().then(enrichWorkflows), - scan: (root: string) => scanWorkflowsAndBroadcast(root, "requested"), + list: async () => + publicWorkflowInfos(await enrichWorkflows(workflowsCache)), + scan: (root: string) => + scanWorkflowsAndBroadcast(root, "requested", { dirty: true }).then( + (outcome) => publicWorkflowInfos(outcome.found), + ), connectPath: async (inputPath: string) => { - const workflow = await workflowRegistry.connectPath(inputPath); - workflowsCache = await workflowRegistry.list(); - await Promise.all( - sessionManager.list().map((session) => writeSessionContext(session)), + mutationTokenSequence += 1; + const prepared = prepareDirtyWorkflowRoot( + inputPath, + `connect:${mutationTokenSequence}`, ); - await refreshSystemGraphScopesForRoot(workflow.path); - bus.publish({ type: "workflows.changed" }); - return workflow; + try { + const workflow = await workflowRegistry.connectPath(inputPath); + const scan = scanWorkflowsAndBroadcast( + workflow.path, + "agent-connected", + { dirty: true }, + ); + // scanWorkflowsAndBroadcast synchronously installs its own flight + // token before returning. The mutation token no longer owns freshness. + cancelSystemGraphPrerequisite(prepared.token); + const outcome = await scan; + return publicWorkflowInfo( + workflowsCache.find( + (candidate) => candidate.path === workflow.path, + ) ?? + outcome.found.find( + (candidate) => candidate.path === workflow.path, + ) ?? + workflow, + ); + } catch (error) { + cancelSystemGraphPrerequisite(prepared.token); + reportSystemGraphRefreshFailure(prepared.lexicalRoot); + throw error; + } }, scanWithBoundaries: async (root: string) => { - const budget = new AgentProjectScanBudget(); - const found = await workflowRegistry.scan(root, budget); - logAgentScan("requested", root, found.length, budget); - // The boundaries the walk stopped at travel with the result so the rail - // can explain an empty project instead of misdescribing one. - return { found, repositoryBoundaries: budget.repositoryBoundaries }; + const outcome = await scanWorkflowsAndBroadcast(root, "requested", { + dirty: true, + }); + return { ...outcome, found: publicWorkflowInfos(outcome.found) }; }, }; app.use( @@ -1694,7 +2668,9 @@ export const startServer = async ( // which the watcher's directory-set diff cannot see — rescan that project // so the Draft→Deployed chip and the deploy-gated actions update. onLinked: async (workflow) => { - await scanWorkflowsAndBroadcast(workflow.path, "agent-linked"); + await scanWorkflowsAndBroadcast(workflow.path, "agent-linked", { + dirty: true, + }); }, }), ); @@ -1758,16 +2734,14 @@ export const startServer = async ( // NEW path rather than from a stale registry row. onMoved: async (from, to) => { remapSessions(sessionManager.list(), from, to); - await workflowRegistry.prune(); - await scanWorkflowsAndBroadcast(dirname(to), "agent-moved", { - refreshGraphs: async () => { - // A cross-project move changes two containment inventories. Refresh - // every active parent/nested graph touching either side so neither - // keeps a ghost node or misses the arrival. - refreshSystemGraphScopesForRoot(dirname(from)); - refreshSystemGraphScopesForRoot(dirname(to)); - }, - }); + await Promise.all([ + scanWorkflowsAndBroadcast(dirname(from), "agent-moved", { + dirty: true, + }), + scanWorkflowsAndBroadcast(dirname(to), "agent-moved", { + dirty: true, + }), + ]); }, }), ); @@ -1826,7 +2800,9 @@ export const startServer = async ( // scan broadcasts `workflows.changed` so the row is there before the // dialog's caller opens a session on it. onScaffolded: async (agentDir) => { - await scanWorkflowsAndBroadcast(dirname(agentDir), "agent-created"); + await scanWorkflowsAndBroadcast(dirname(agentDir), "agent-created", { + dirty: true, + }); }, }), ); @@ -2121,7 +3097,7 @@ export const startServer = async ( // Reuses the scan already kicked off above rather than scanning // launchDir twice — only renders when it actually found something, // same "discoverable" gate as the REST onSessionCreated path. - const found = await initialWorkflowScan; + const { found } = await initialWorkflowScan; if (found.length > 0) await autoRenderCanvas(session); }) .catch((err: unknown) => { @@ -2141,14 +3117,17 @@ export const startServer = async ( port: actualPort, sessionManager, close: async () => { - clearInterval(workflowsCacheTimer); + coordinatorActive = false; + coordinatorEpoch += 1; + await workflowRegistry.retirePendingDiscovery(); + rejectPublication(new Error("Agent discovery coordinator is closed")); clearInterval(sessionSweepTimer); clearInterval(ndjsonRetentionTimer); canvasWatcher.stopAll(); workspaceWatcher.stopAll(); systemGraphWatcher.stopAll(); activeSystemGraphScopes.clear(); - systemGraphRelationships.clear(); + systemGraphInvocations.clear(); systemGraphInventory.clear(); systemGraphStore.clear(); installWatcher.stopAll(); diff --git a/packages/harness/src/server/rest.test.ts b/packages/harness/src/server/rest.test.ts index b31686924..58d267107 100644 --- a/packages/harness/src/server/rest.test.ts +++ b/packages/harness/src/server/rest.test.ts @@ -4,7 +4,6 @@ import * as os from "node:os"; import * as path from "node:path"; import express from "express"; import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; -import { createBootTokenMiddleware } from "./auth.js"; let tmpHome: string; @@ -13,10 +12,29 @@ vi.mock("node:os", async (importOriginal) => { return { ...actual, homedir: () => tmpHome }; }); -import type { HarnessAdapter, HarnessKind, HarnessSession, MacroDef, SessionRecord, SessionSummary, SpawnSpec, WorkflowInfo } from "../shared/types.js"; -import { SessionManager, SessionNotReadyError, UnknownSessionError } from "../core/session-manager.js"; +import type { + HarnessAdapter, + HarnessKind, + HarnessSession, + MacroDef, + SessionRecord, + SessionSummary, + SpawnSpec, + WorkflowInfo, +} from "../shared/types.js"; +import { + SessionManager, + SessionNotReadyError, + UnknownSessionError, +} from "../core/session-manager.js"; import type { SessionRecordReader } from "../core/session-record.js"; -import { AdapterNotFoundError, ExternalHarnessError, SessionAlreadyLiveError, SessionNotResumeableError, SpawnTargetError } from "../core/errors.js"; +import { + AdapterNotFoundError, + ExternalHarnessError, + SessionAlreadyLiveError, + SessionNotResumeableError, + SpawnTargetError, +} from "../core/errors.js"; import { createRestRouter, type RestRouterOptions } from "./rest.js"; const TOKEN_HEADER = { "X-Harness-Token": "unused-in-router-tests" }; @@ -35,28 +53,38 @@ function fakeSessionManager(initial: HarnessSession[] = []) { const session = sessions.get(id); if (session) session.boundWorkflowPath = workflowPath; }), - registerHistorical: vi.fn((input: { agentSessionId: string; harness: HarnessKind; cwd: string; title: string; lastActiveAt: string }) => { - const session: HarnessSession = { - id: `adopted-${input.agentSessionId}`, - agentSessionId: input.agentSessionId, - harness: input.harness, - cwd: input.cwd, - title: input.title, - status: "exited", - createdAt: input.lastActiveAt, - lastActiveAt: input.lastActiveAt, - exitCode: null, - boundWorkflowPath: null, - ready: false, - }; - sessions.set(session.id, session); - return session; - }), + registerHistorical: vi.fn( + (input: { + agentSessionId: string; + harness: HarnessKind; + cwd: string; + title: string; + lastActiveAt: string; + }) => { + const session: HarnessSession = { + id: `adopted-${input.agentSessionId}`, + agentSessionId: input.agentSessionId, + harness: input.harness, + cwd: input.cwd, + title: input.title, + status: "exited", + createdAt: input.lastActiveAt, + lastActiveAt: input.lastActiveAt, + exitCode: null, + boundWorkflowPath: null, + ready: false, + }; + sessions.set(session.id, session); + return session; + }, + ), } as unknown as RestRouterOptions["sessionManager"]; } /** An exited registry session — the shape a past-sessions row is built from. */ -function exitedSession(overrides: Partial = {}): HarnessSession { +function exitedSession( + overrides: Partial = {}, +): HarnessSession { return { id: "sess-1", agentSessionId: "agent-1", @@ -75,15 +103,22 @@ function exitedSession(overrides: Partial = {}): HarnessSession /** A history adapter whose resumability answer and transcript listing are both * controllable — the two independent inputs the history endpoint merges. */ -function historyAdapter(opts: { - canResume?: HarnessAdapter["canResume"]; - listPastSessions?: HarnessAdapter["listPastSessions"]; -} = {}): HarnessAdapter { +function historyAdapter( + opts: { + canResume?: HarnessAdapter["canResume"]; + listPastSessions?: HarnessAdapter["listPastSessions"]; + } = {}, +): HarnessAdapter { return { id: "claude-code", eventSource: "hooks" as const, doctor: async () => [], - launch: (o): SpawnSpec => ({ command: "fake-claude", args: [], env: {}, cwd: o.cwd }), + launch: (o): SpawnSpec => ({ + command: "fake-claude", + args: [], + env: {}, + cwd: o.cwd, + }), resume: (agentSessionId, o): SpawnSpec => ({ command: "fake-claude", args: ["--resume", agentSessionId], @@ -245,7 +280,11 @@ describe("createRestRouter", () => { start({ sessionManager: fakeSessionManager([session]), - identity: { userId: "user-1", tenantId: "user-1", organizationName: "Acme" }, + identity: { + userId: "user-1", + tenantId: "user-1", + organizationName: "Acme", + }, listWorkflows: async () => [workflow], listMacros: () => [macro], }); @@ -445,7 +484,10 @@ describe("createRestRouter", () => { const res = await fetch(`${baseUrl}/sessions`, { method: "POST", headers: { ...TOKEN_HEADER, "content-type": "application/json" }, - body: JSON.stringify({ cwd: "/tmp//projects/../proj", harness: "claude-code" }), + body: JSON.stringify({ + cwd: "/tmp//projects/../proj", + harness: "claude-code", + }), }); expect(res.status).toBe(201); @@ -460,7 +502,9 @@ describe("createRestRouter", () => { // user what to do was discarded. The dialog renders this body verbatim. const sessionManager = fakeSessionManager(); (sessionManager.create as ReturnType).mockRejectedValue( - new SpawnTargetError('cannot spawn "claude" on Windows: not found on PATH'), + new SpawnTargetError( + 'cannot spawn "claude" on Windows: not found on PATH', + ), ); start({ sessionManager }); @@ -490,7 +534,9 @@ describe("createRestRouter", () => { }); expect(res.status).toBe(409); - expect(((await res.json()) as { code: string }).code).toBe("HARNESS_EXTERNAL"); + expect(((await res.json()) as { code: string }).code).toBe( + "HARNESS_EXTERNAL", + ); }); }); @@ -802,7 +848,8 @@ describe("createRestRouter", () => { expect(res.status).toBe(400); expect(await res.json()).toEqual({ - error: "Unknown agent path '/not/registered' — scan or connect it before binding a session to it", + error: + "Unknown agent path '/not/registered' — scan or connect it before binding a session to it", }); expect(sessionManager.setBoundWorkflowPath).not.toHaveBeenCalled(); expect(writeWorkspaceContext).not.toHaveBeenCalled(); @@ -924,9 +971,12 @@ describe("createRestRouter", () => { describe("GET /sessions/history — server-verified resumeMode", () => { async function history(cwd: string): Promise { - const res = await fetch(`${baseUrl}/sessions/history?cwd=${encodeURIComponent(cwd)}`, { - headers: TOKEN_HEADER, - }); + const res = await fetch( + `${baseUrl}/sessions/history?cwd=${encodeURIComponent(cwd)}`, + { + headers: TOKEN_HEADER, + }, + ); expect(res.status).toBe(200); return (await res.json()) as SessionSummary[]; } @@ -934,12 +984,18 @@ describe("createRestRouter", () => { it("marks a registry row the agent still holds as agent-resume", async () => { start({ sessionManager: fakeSessionManager([exitedSession()]), - adapters: { "claude-code": historyAdapter({ canResume: async () => true }) }, + adapters: { + "claude-code": historyAdapter({ canResume: async () => true }), + }, }); const rows = await history("/tmp/proj"); expect(rows).toHaveLength(1); - expect(rows[0]).toMatchObject({ agentSessionId: "agent-1", source: "registry", resumeMode: "agent-resume" }); + expect(rows[0]).toMatchObject({ + agentSessionId: "agent-1", + source: "registry", + resumeMode: "agent-resume", + }); }); it("marks a PHANTOM registry row as rehydrate — an agentSessionId is not evidence of a conversation", async () => { @@ -948,11 +1004,16 @@ describe("createRestRouter", () => { // button whose only possible outcome was exit 1. start({ sessionManager: fakeSessionManager([exitedSession()]), - adapters: { "claude-code": historyAdapter({ canResume: async () => false }) }, + adapters: { + "claude-code": historyAdapter({ canResume: async () => false }), + }, }); const rows = await history("/tmp/proj"); - expect(rows[0]).toMatchObject({ source: "registry", resumeMode: "rehydrate" }); + expect(rows[0]).toMatchObject({ + source: "registry", + resumeMode: "rehydrate", + }); }); it("probes with the row's own agentSessionId and cwd", async () => { @@ -987,12 +1048,20 @@ describe("createRestRouter", () => { }); const rows = await history("/tmp/proj"); - expect(rows[0]).toMatchObject({ source: "transcript", resumeMode: "agent-resume" }); + expect(rows[0]).toMatchObject({ + source: "transcript", + resumeMode: "agent-resume", + }); }); it("resolves each row independently — a phantom and a live transcript in one directory", async () => { start({ - sessionManager: fakeSessionManager([exitedSession({ id: "sess-phantom", agentSessionId: "agent-phantom" })]), + sessionManager: fakeSessionManager([ + exitedSession({ + id: "sess-phantom", + agentSessionId: "agent-phantom", + }), + ]), adapters: { "claude-code": historyAdapter({ canResume: async (id) => id !== "agent-phantom", @@ -1010,7 +1079,12 @@ describe("createRestRouter", () => { }, }); - const byId = new Map((await history("/tmp/proj")).map((row) => [row.agentSessionId, row.resumeMode])); + const byId = new Map( + (await history("/tmp/proj")).map((row) => [ + row.agentSessionId, + row.resumeMode, + ]), + ); expect(byId.get("agent-phantom")).toBe("rehydrate"); expect(byId.get("agent-real")).toBe("agent-resume"); }); @@ -1033,13 +1107,18 @@ describe("createRestRouter", () => { ]); start({ sessionManager: fakeSessionManager([exitedSession()]), - adapters: { "claude-code": historyAdapter({ canResume, listPastSessions }) }, + adapters: { + "claude-code": historyAdapter({ canResume, listPastSessions }), + }, }); const rows = await history("/tmp/proj"); expect(rows).toHaveLength(1); // Registry row wins the merge (it carries live status) and is resumable. - expect(rows[0]).toMatchObject({ source: "registry", resumeMode: "agent-resume" }); + expect(rows[0]).toMatchObject({ + source: "registry", + resumeMode: "agent-resume", + }); expect(listPastSessions).toHaveBeenCalledTimes(1); expect(canResume).not.toHaveBeenCalled(); }); @@ -1068,7 +1147,12 @@ describe("createRestRouter", () => { }, }); - const byId = new Map((await history("/tmp/proj")).map((r) => [r.agentSessionId, r.resumeMode])); + const byId = new Map( + (await history("/tmp/proj")).map((r) => [ + r.agentSessionId, + r.resumeMode, + ]), + ); expect(byId.get("agent-found")).toBe("agent-resume"); expect(byId.get("agent-missed")).toBe("rehydrate"); expect(canResume.mock.calls).toEqual([["agent-missed", "/tmp/proj"]]); @@ -1078,7 +1162,9 @@ describe("createRestRouter", () => { // e.g. an external-mode harness, or a kind persisted by another build: // unverifiable is not the same as resumable. start({ - sessionManager: fakeSessionManager([exitedSession({ harness: "conductor" as HarnessKind })]), + sessionManager: fakeSessionManager([ + exitedSession({ harness: "conductor" as HarnessKind }), + ]), adapters: {}, }); @@ -1122,27 +1208,45 @@ describe("createRestRouter", () => { it("registers a transcript-only row and resumes it for real", async () => { const sessionManager = fakeSessionManager(); - (sessionManager.resume as ReturnType).mockImplementation(async (id: string) => ({ - ...exitedSession({ id, agentSessionId: body.agentSessionId }), - status: "running", - })); - start({ sessionManager, adapters: { "claude-code": historyAdapter({ canResume: async () => true }) } }); + (sessionManager.resume as ReturnType).mockImplementation( + async (id: string) => ({ + ...exitedSession({ id, agentSessionId: body.agentSessionId }), + status: "running", + }), + ); + start({ + sessionManager, + adapters: { + "claude-code": historyAdapter({ canResume: async () => true }), + }, + }); const res = await adopt(body); expect(res.status).toBe(200); - expect((await res.json()) as HarnessSession).toMatchObject({ status: "running" }); + expect((await res.json()) as HarnessSession).toMatchObject({ + status: "running", + }); expect(sessionManager.registerHistorical).toHaveBeenCalledWith(body); // The whole point: a real resume, not a fresh session. - expect(sessionManager.resume).toHaveBeenCalledWith("adopted-agent-transcript"); + expect(sessionManager.resume).toHaveBeenCalledWith( + "adopted-agent-transcript", + ); }); it("409s SESSION_NOT_RESUMEABLE without registering anything when the agent no longer holds it", async () => { const sessionManager = fakeSessionManager(); - start({ sessionManager, adapters: { "claude-code": historyAdapter({ canResume: async () => false }) } }); + start({ + sessionManager, + adapters: { + "claude-code": historyAdapter({ canResume: async () => false }), + }, + }); const res = await adopt(body); expect(res.status).toBe(409); - expect((await res.json()) as { code: string }).toMatchObject({ code: "SESSION_NOT_RESUMEABLE" }); + expect((await res.json()) as { code: string }).toMatchObject({ + code: "SESSION_NOT_RESUMEABLE", + }); // No phantom record left behind by a stale history row. expect(sessionManager.registerHistorical).not.toHaveBeenCalled(); expect(sessionManager.resume).not.toHaveBeenCalled(); @@ -1151,17 +1255,33 @@ describe("createRestRouter", () => { it("re-verifies server-side — a client claiming resumability cannot force a registration", async () => { const sessionManager = fakeSessionManager(); const canResume = vi.fn(async () => false); - start({ sessionManager, adapters: { "claude-code": historyAdapter({ canResume }) } }); + start({ + sessionManager, + adapters: { "claude-code": historyAdapter({ canResume }) }, + }); - expect((await adopt({ ...body, resumeMode: "agent-resume" })).status).toBe(409); + expect( + (await adopt({ ...body, resumeMode: "agent-resume" })).status, + ).toBe(409); expect(canResume).toHaveBeenCalledWith(body.agentSessionId, body.cwd); }); it("is idempotent: an already-tracked row resumes its existing record instead of duplicating it", async () => { - const existing = exitedSession({ id: "sess-existing", agentSessionId: body.agentSessionId }); + const existing = exitedSession({ + id: "sess-existing", + agentSessionId: body.agentSessionId, + }); const sessionManager = fakeSessionManager([existing]); - (sessionManager.resume as ReturnType).mockResolvedValue({ ...existing, status: "running" }); - start({ sessionManager, adapters: { "claude-code": historyAdapter({ canResume: async () => true }) } }); + (sessionManager.resume as ReturnType).mockResolvedValue({ + ...existing, + status: "running", + }); + start({ + sessionManager, + adapters: { + "claude-code": historyAdapter({ canResume: async () => true }), + }, + }); expect((await adopt(body)).status).toBe(200); expect(sessionManager.registerHistorical).not.toHaveBeenCalled(); @@ -1181,7 +1301,12 @@ describe("createRestRouter", () => { // `POST /sessions/:id` exists to catch it. This pins that, so adding // such a route later can't silently reroute adopt through resume(). const sessionManager = fakeSessionManager(); - start({ sessionManager, adapters: { "claude-code": historyAdapter({ canResume: async () => false }) } }); + start({ + sessionManager, + adapters: { + "claude-code": historyAdapter({ canResume: async () => false }), + }, + }); const res = await adopt(body); expect(res.status).toBe(409); @@ -1193,11 +1318,18 @@ describe("createRestRouter", () => { (sessionManager.resume as ReturnType).mockRejectedValue( new SessionAlreadyLiveError("adopted-agent-transcript"), ); - start({ sessionManager, adapters: { "claude-code": historyAdapter({ canResume: async () => true }) } }); + start({ + sessionManager, + adapters: { + "claude-code": historyAdapter({ canResume: async () => true }), + }, + }); const res = await adopt(body); expect(res.status).toBe(409); - expect((await res.json()) as { code: string }).toMatchObject({ code: "SESSION_ALREADY_LIVE" }); + expect((await res.json()) as { code: string }).toMatchObject({ + code: "SESSION_ALREADY_LIVE", + }); }); }); @@ -1423,7 +1555,9 @@ describe("createRestRouter", () => { limitations: [], }; - function stubRecords(overrides: Partial = {}): SessionRecordReader { + function stubRecords( + overrides: Partial = {}, + ): SessionRecordReader { const find = async (id: string): Promise => id === "sess-1" || id === "agent-1" ? record : null; return { @@ -1437,26 +1571,34 @@ describe("createRestRouter", () => { it("GET /sessions/:id/record returns the reconstructed record", async () => { start({ sessionRecords: stubRecords() }); - const res = await fetch(`${baseUrl}/sessions/sess-1/record`, { headers: TOKEN_HEADER }); + const res = await fetch(`${baseUrl}/sessions/sess-1/record`, { + headers: TOKEN_HEADER, + }); expect(res.status).toBe(200); expect(await res.json()).toEqual(record); }); it("GET /sessions/:id/record resolves an agent session id too", async () => { start({ sessionRecords: stubRecords() }); - const res = await fetch(`${baseUrl}/sessions/agent-1/record`, { headers: TOKEN_HEADER }); + const res = await fetch(`${baseUrl}/sessions/agent-1/record`, { + headers: TOKEN_HEADER, + }); expect(res.status).toBe(200); }); it("GET /sessions/:id/record is 404 when nothing was recorded for the session", async () => { start({ sessionRecords: stubRecords() }); - const res = await fetch(`${baseUrl}/sessions/unknown/record`, { headers: TOKEN_HEADER }); + const res = await fetch(`${baseUrl}/sessions/unknown/record`, { + headers: TOKEN_HEADER, + }); expect(res.status).toBe(404); }); it("GET /sessions/:id/record is 501 when the server has no record reader", async () => { start(); - const res = await fetch(`${baseUrl}/sessions/sess-1/record`, { headers: TOKEN_HEADER }); + const res = await fetch(`${baseUrl}/sessions/sess-1/record`, { + headers: TOKEN_HEADER, + }); expect(res.status).toBe(501); }); @@ -1473,11 +1615,17 @@ describe("createRestRouter", () => { lastActiveAt: "2026-07-01T10:00:05.000Z", ready: false, }; - start({ sessionManager: fakeSessionManager([session]), sessionRecords: stubRecords() }); - - const res = await fetch(`${baseUrl}/sessions/history?cwd=${encodeURIComponent("/repo")}`, { - headers: TOKEN_HEADER, + start({ + sessionManager: fakeSessionManager([session]), + sessionRecords: stubRecords(), }); + + const res = await fetch( + `${baseUrl}/sessions/history?cwd=${encodeURIComponent("/repo")}`, + { + headers: TOKEN_HEADER, + }, + ); expect(res.status).toBe(200); const body = (await res.json()) as SessionSummary[]; expect(body).toHaveLength(1); @@ -1506,9 +1654,12 @@ describe("createRestRouter", () => { }), }); - const res = await fetch(`${baseUrl}/sessions/history?cwd=${encodeURIComponent("/repo")}`, { - headers: TOKEN_HEADER, - }); + const res = await fetch( + `${baseUrl}/sessions/history?cwd=${encodeURIComponent("/repo")}`, + { + headers: TOKEN_HEADER, + }, + ); expect(res.status).toBe(200); const body = (await res.json()) as SessionSummary[]; expect(body).toHaveLength(1); diff --git a/packages/harness/src/server/studio-rail.ts b/packages/harness/src/server/studio-rail.ts index e9a12f917..e49cd67a9 100644 --- a/packages/harness/src/server/studio-rail.ts +++ b/packages/harness/src/server/studio-rail.ts @@ -150,7 +150,7 @@ export async function removeStudioRailFile(root: string): Promise { * Launch edges across every registered agent, as parent-name to child-slug. * * Uses the shared syntax-only extractor (`core/canvas-interconnections.ts`) — - * the same detector the canvas and system graph use for agent relationships. A + * the same detector the canvas and system graph use for direct agent invocations. A * second edge detector would be a second answer to "what does this launch", * and the Group axis and system graph are supposed to read one graph. * diff --git a/packages/harness/src/server/system-graph-freshness.test.ts b/packages/harness/src/server/system-graph-freshness.test.ts index d9b589367..c48f08172 100644 --- a/packages/harness/src/server/system-graph-freshness.test.ts +++ b/packages/harness/src/server/system-graph-freshness.test.ts @@ -4,8 +4,16 @@ import * as path from "node:path"; import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; import { WebSocket } from "ws"; -import type { AppState, BusMessage, WorkflowInfo } from "../shared/types.js"; +import type { + AppState, + BusMessage, + HarnessAdapter, + LaunchOpts, + SpawnSpec, + WorkflowInfo, +} from "../shared/types.js"; import type { SystemGraphSnapshot } from "../shared/system-graph.js"; +import type { RegistryWorkflowInfo } from "../core/workflow-registry.js"; import { startServer, type HarnessServer } from "./index.js"; async function scaffoldAgent( @@ -43,6 +51,35 @@ export default defineAgent({ name: ${JSON.stringify(name)}, entry: "run", steps: `; } +function deferred(): { + promise: Promise; + resolve: () => void; +} { + let resolve!: () => void; + const promise = new Promise((settle) => { + resolve = settle; + }); + return { promise, resolve }; +} + +function fakeClaudeAdapter(): HarnessAdapter { + const spec = (options: LaunchOpts): SpawnSpec => ({ + command: "bash", + args: [], + env: {}, + cwd: options.cwd, + }); + return { + id: "claude-code", + eventSource: "hooks", + doctor: async () => [], + launch: spec, + resume: (_agentSessionId, options) => spec(options), + listPastSessions: async () => [], + canResume: async () => true, + }; +} + describe("workspace graph freshness wiring", () => { let tempRoot: string; let stateRoot: string; @@ -66,13 +103,19 @@ describe("workspace graph freshness wiring", () => { afterEach(async () => { socket?.close(); + await server?.sessionManager.flush(); await server?.close(); server = undefined; - await fs.rm(tempRoot, { recursive: true, force: true }); + await fs.rm(tempRoot, { + recursive: true, + force: true, + maxRetries: 3, + retryDelay: 100, + }); }); it( - "refreshes source relationships and agent inventory without a session", + "refreshes source invocations and agent inventory without a session", { retry: 1, timeout: 30_000 }, async () => { const researchRoot = await scaffoldAgent(workspaceRoot, "research"); @@ -131,16 +174,20 @@ describe("workspace graph freshness wiring", () => { return JSON.parse(raw) as SystemGraphSnapshot; }; const initial = await readGraph(); - // These fixtures intentionally have no defineAgent export. The graph is - // useful immediately through their marker identities while background - // source inspection honestly leaves the inventory degraded. - expect(initial).toMatchObject({ state: "degraded" }); - expect(initial.graph?.edges).toEqual([]); + // Cold discovery is detached: cached inventory renders immediately, + // conservatively degraded until this process accepts fresh evidence. + expect(initial.state).toBe("degraded"); + expect(initial.graph?.nodes.map((node) => node.agentKey).sort()).toEqual([ + "growth", + "research", + ]); let absentSettled!: SystemGraphSnapshot; await vi.waitFor( async () => { absentSettled = await readGraph(); expect(absentSettled.revision).toBeGreaterThan(initial.revision); + expect(absentSettled.state).toBe("degraded"); + expect(absentSettled.graph?.edges).toEqual([]); expect( absentSettled.graph?.warnings.some( (warning) => warning.code === "inventory-extraction-failed", @@ -271,13 +318,735 @@ describe("workspace graph freshness wiring", () => { expect(manualRetryResponse.status).toBe(200); const manualRetry = (await manualRetryResponse.json()) as SystemGraphSnapshot; - // Manual Retry does not unset identities already proven absent. It - // rebuilds the graph, but the settled inventory remains cacheable. - expect(manualRetry).toMatchObject({ state: "ready" }); + // Manual Retry rebuilds immediately from accepted inventory, then direct + // invocation extraction completes in the background. + expect(manualRetry).toMatchObject({ state: "degraded" }); expect(manualRetry.revision).toBeGreaterThan(beforeManualRetry.revision); }, ); + it("serves persisted cold inventory without awaiting discovery", async () => { + const within = async (promise: Promise, label: string): Promise => + await Promise.race([ + promise, + new Promise((_resolve, reject) => { + setTimeout(() => reject(new Error(`timed out: ${label}`)), 1_000); + }), + ]); + const coldRoot = await scaffoldAgent(workspaceRoot, "cold"); + await fs.writeFile( + path.join(stateRoot, "workflows.json"), + JSON.stringify([ + { + name: "cold", + path: coldRoot, + definitionId: null, + definitionSlug: "cold", + templateId: null, + forkId: null, + starterId: null, + activeBuildRunId: null, + activeBuildRunStatus: null, + markerPresent: true, + source: "scan", + } satisfies RegistryWorkflowInfo, + ]), + ); + const scanGate = deferred(); + const scanEntered = deferred(); + let blockFirstScan = true; + server = await startServer({ + port: 0, + bootToken: "test-token", + telemetryOptIn: false, + adapters: {}, + stateRoot, + launchDir: workspaceRoot, + autoCreateSession: false, + workflowDiscoveryTestHooks: { + beforeScan: async () => { + if (!blockFirstScan) return; + blockFirstScan = false; + scanEntered.resolve(); + await scanGate.promise; + }, + }, + }); + const baseUrl = `http://127.0.0.1:${server.port}`; + const headers = { "X-Harness-Token": "test-token" }; + await within(scanEntered.promise, "scan entry"); + const state = (await ( + await within(fetch(`${baseUrl}/api/state`, { headers }), "state") + ).json()) as AppState; + const workspaceKey = state.workspaceScopes?.find( + (scope) => scope.cwd === workspaceRoot, + )?.workspaceKey; + expect(workspaceKey).toBeTruthy(); + const response = await within( + fetch(`${baseUrl}/api/workspaces/${workspaceKey}/system-graph`, { + headers, + }), + "graph", + ); + expect(response.status).toBe(200); + const cached = (await response.json()) as SystemGraphSnapshot; + expect(cached.state).toBe("degraded"); + expect(cached.graph?.nodes.some((node) => node.agentKey === "cold")).toBe( + true, + ); + + scanGate.resolve(); + const acceptedScan = await fetch(`${baseUrl}/api/workflows/scan`, { + method: "POST", + headers: { ...headers, "Content-Type": "application/json" }, + body: JSON.stringify({ root: workspaceRoot }), + }); + expect(acceptedScan.status).toBe(200); + await within( + vi.waitFor(async () => { + const settled = (await ( + await fetch( + `${baseUrl}/api/workspaces/${workspaceKey}/system-graph`, + { headers }, + ) + ).json()) as SystemGraphSnapshot; + expect(settled.graph?.nodes).toHaveLength(1); + }), + "settled graph", + ); + }); + + it("supersedes a paused publication and commits only the newest scan", async () => { + const agentRoot = await scaffoldAgent(workspaceRoot, "initial"); + const publicationGate = deferred(); + const publicationEntered = deferred(); + let blockNextPublication = false; + server = await startServer({ + port: 0, + bootToken: "test-token", + telemetryOptIn: false, + adapters: {}, + stateRoot, + launchDir: workspaceRoot, + autoCreateSession: false, + workflowDiscoveryTestHooks: { + beforePublication: async () => { + if (!blockNextPublication) return; + blockNextPublication = false; + publicationEntered.resolve(); + await publicationGate.promise; + }, + }, + }); + const baseUrl = `http://127.0.0.1:${server.port}`; + const headers = { + "X-Harness-Token": "test-token", + "Content-Type": "application/json", + }; + await vi.waitFor(async () => { + const workflows = (await ( + await fetch(`${baseUrl}/api/workflows`, { headers }) + ).json()) as WorkflowInfo[]; + expect(workflows[0]?.definitionSlug).toBe("initial"); + }); + blockNextPublication = true; + await fs.writeFile( + path.join(agentRoot, "sapiom.json"), + JSON.stringify({ name: "intermediate", definitionId: null }), + ); + const first = fetch(`${baseUrl}/api/workflows/scan`, { + method: "POST", + headers, + body: JSON.stringify({ root: workspaceRoot }), + }); + await publicationEntered.promise; + + await fs.writeFile( + path.join(agentRoot, "sapiom.json"), + JSON.stringify({ name: "newest", definitionId: null }), + ); + const second = fetch(`${baseUrl}/api/workflows/scan`, { + method: "POST", + headers, + body: JSON.stringify({ root: workspaceRoot }), + }); + publicationGate.resolve(); + const responses = await Promise.all([first, second]); + expect(responses.map((response) => response.status)).toEqual([200, 200]); + const workflows = (await ( + await fetch(`${baseUrl}/api/workflows`, { headers }) + ).json()) as WorkflowInfo[]; + expect(workflows).toHaveLength(1); + expect(workflows[0]?.definitionSlug).toBe("newest"); + }); + + it("uses fresh source and project budgets for a generation superseded after scanning", async () => { + await fs.writeFile( + path.join(workspaceRoot, "index.ts"), + `import { defineAgent } from "@sapiom/agent"; +export const agent = defineAgent({ name: "budget-v1" });`, + ); + const scanReturned = deferred(); + const releaseScan = deferred(); + let blockNextRequestedResult = false; + server = await startServer({ + port: 0, + bootToken: "test-token", + telemetryOptIn: false, + adapters: {}, + stateRoot, + launchDir: workspaceRoot, + autoCreateSession: false, + workflowDiscoveryTestHooks: { + afterScan: async ({ reason }) => { + if (!blockNextRequestedResult || reason !== "requested") return; + blockNextRequestedResult = false; + scanReturned.resolve(); + await releaseScan.promise; + }, + }, + }); + const baseUrl = `http://127.0.0.1:${server.port}`; + const headers = { + "X-Harness-Token": "test-token", + "Content-Type": "application/json", + }; + await vi.waitFor(async () => { + expect( + (await ( + await fetch(`${baseUrl}/api/workflows`, { headers }) + ).json()) as WorkflowInfo[], + ).toHaveLength(1); + }); + const state = (await ( + await fetch(`${baseUrl}/api/state`, { headers }) + ).json()) as AppState; + const workspaceKey = state.workspaceScopes?.find( + (scope) => scope.cwd === workspaceRoot, + )?.workspaceKey; + expect(workspaceKey).toBeTruthy(); + const graphUrl = `${baseUrl}/api/workspaces/${workspaceKey}/system-graph`; + await fetch(graphUrl, { headers }); + + blockNextRequestedResult = true; + const first = fetch(`${baseUrl}/api/workflows/scan`, { + method: "POST", + headers, + body: JSON.stringify({ root: workspaceRoot }), + }); + await scanReturned.promise; + await fs.writeFile( + path.join(workspaceRoot, "index.ts"), + `import { defineAgent } from "@sapiom/agent"; +export const agent = defineAgent({ name: "budget-v2-final" });`, + ); + const second = fetch(`${baseUrl}/api/workflows/scan`, { + method: "POST", + headers, + body: JSON.stringify({ root: workspaceRoot }), + }); + releaseScan.resolve(); + expect( + (await Promise.all([first, second])).map((response) => response.status), + ).toEqual([200, 200]); + await vi.waitFor(async () => { + const snapshot = (await ( + await fetch(graphUrl, { headers }) + ).json()) as SystemGraphSnapshot; + expect(snapshot.graph?.nodes.map((node) => node.agentKey)).toEqual([ + "budget-v2-final", + ]); + }); + }); + + it("lets an ordinary first graph scan release a failed dirty prerequisite", async () => { + await scaffoldAgent(workspaceRoot, "recoverable"); + let failuresRemaining = 0; + server = await startServer({ + port: 0, + bootToken: "test-token", + telemetryOptIn: false, + adapters: {}, + stateRoot, + launchDir: workspaceRoot, + autoCreateSession: false, + workflowDiscoveryTestHooks: { + beforeScan: () => { + if (failuresRemaining <= 0) return; + failuresRemaining -= 1; + throw new Error("held dirty reconciliation failed"); + }, + }, + }); + const baseUrl = `http://127.0.0.1:${server.port}`; + const headers = { + "X-Harness-Token": "test-token", + "Content-Type": "application/json", + }; + await vi.waitFor(async () => { + const workflows = (await ( + await fetch(`${baseUrl}/api/workflows`, { headers }) + ).json()) as WorkflowInfo[]; + expect(workflows.map((workflow) => workflow.definitionSlug)).toEqual([ + "recoverable", + ]); + }); + const state = (await ( + await fetch(`${baseUrl}/api/state`, { headers }) + ).json()) as AppState; + const workspaceKey = state.workspaceScopes?.find( + (scope) => scope.cwd === workspaceRoot, + )?.workspaceKey; + expect(workspaceKey).toBeTruthy(); + + failuresRemaining = 4; + const failed = await fetch(`${baseUrl}/api/workflows/scan`, { + method: "POST", + headers, + body: JSON.stringify({ root: workspaceRoot }), + }); + expect(failed.status).toBe(500); + expect(failuresRemaining).toBe(0); + + const graphUrl = `${baseUrl}/api/workspaces/${workspaceKey}/system-graph`; + // The first GET attaches the prior dirty token before it starts the + // ordinary background recovery scan. It may initially be building, but + // the accepted exact-root proof must release that inherited token. + await fetch(graphUrl, { headers }); + await vi.waitFor( + async () => { + const response = await fetch(graphUrl, { headers }); + const snapshot = (await response.json()) as SystemGraphSnapshot; + expect(snapshot.state).not.toBe("building"); + expect( + snapshot.graph?.nodes.some((node) => node.agentKey === "recoverable"), + ).toBe(true); + }, + { timeout: 8_000, interval: 100 }, + ); + }); + + it( + "coalesces two sessions and a graph subscriber into one pass plus one held-edit trailing pass", + { timeout: 25_000 }, + async () => { + await fs.writeFile( + path.join(workspaceRoot, "index.ts"), + `import { defineAgent } from "@sapiom/agent"; +export const agent = defineAgent({ name: "shared-v0" });`, + ); + const firstPassEntered = deferred(); + const releaseFirstPass = deferred(); + let observePasses = false; + let passCount = 0; + server = await startServer({ + port: 0, + bootToken: "test-token", + telemetryOptIn: false, + adapters: { "claude-code": fakeClaudeAdapter() }, + stateRoot, + launchDir: workspaceRoot, + autoCreateSession: false, + workflowDiscoveryTestHooks: { + beforeScan: async ({ root }) => { + if (!observePasses || path.resolve(root) !== workspaceRoot) return; + passCount += 1; + if (passCount !== 1) return; + firstPassEntered.resolve(); + await releaseFirstPass.promise; + }, + }, + }); + const headers = { "X-Harness-Token": "test-token" }; + const baseUrl = `http://127.0.0.1:${server.port}`; + await server.sessionManager.create({ + cwd: workspaceRoot, + harness: "claude-code", + }); + await server.sessionManager.create({ + cwd: workspaceRoot, + harness: "claude-code", + }); + const state = (await ( + await fetch(`${baseUrl}/api/state`, { headers }) + ).json()) as AppState; + const workspaceKey = state.workspaceScopes?.find( + (scope) => scope.cwd === workspaceRoot, + )?.workspaceKey; + expect(workspaceKey).toBeTruthy(); + await fetch(`${baseUrl}/api/workspaces/${workspaceKey}/system-graph`, { + headers, + }); + + // Let the shared broker's one conservative initial reconciliation drain + // before counting the edit under test. + await new Promise((resolve) => setTimeout(resolve, 2_500)); + observePasses = true; + await fs.writeFile( + path.join(workspaceRoot, "index.ts"), + `import { defineAgent } from "@sapiom/agent"; +export const agent = defineAgent({ name: "shared-v1" });`, + ); + await firstPassEntered.promise; + + // A second save while the first registry pass is held must supersede it + // immediately. The overlapping source callback reaches the coordinator + // without waiting behind the older subscriber fanout. + await fs.writeFile( + path.join(workspaceRoot, "index.ts"), + `import { defineAgent } from "@sapiom/agent"; +export const agent = defineAgent({ name: "shared-v2-final" });`, + ); + await new Promise((resolve) => setTimeout(resolve, 2_300)); + releaseFirstPass.resolve(); + + await vi.waitFor(() => expect(passCount).toBe(2), { + timeout: 8_000, + interval: 50, + }); + await new Promise((resolve) => setTimeout(resolve, 2_300)); + expect(passCount).toBe(2); + }, + ); + + it.each(["graph-first", "session-first"] as const)( + "reconciles a newly foreign repository from the parent regardless of %s subscriber order", + async (subscriberOrder) => { + const checkout = path.join(workspaceRoot, "checkout"); + await fs.mkdir(checkout, { recursive: true }); + await fs.writeFile( + path.join(checkout, "index.ts"), + `import { defineAgent } from "@sapiom/agent"; +export const agent = defineAgent({ name: "checkout-agent" });`, + ); + server = await startServer({ + port: 0, + bootToken: "test-token", + telemetryOptIn: false, + adapters: { "claude-code": fakeClaudeAdapter() }, + stateRoot, + launchDir: workspaceRoot, + autoCreateSession: false, + }); + const baseUrl = `http://127.0.0.1:${server.port}`; + const headers = { + "X-Harness-Token": "test-token", + "Content-Type": "application/json", + }; + await vi.waitFor(async () => { + const workflows = (await ( + await fetch(`${baseUrl}/api/workflows`, { headers }) + ).json()) as WorkflowInfo[]; + expect(workflows.map((workflow) => workflow.path)).toEqual([checkout]); + }); + const state = (await ( + await fetch(`${baseUrl}/api/state`, { headers }) + ).json()) as AppState; + const workspaceKey = state.workspaceScopes?.find( + (scope) => scope.cwd === workspaceRoot, + )?.workspaceKey; + expect(workspaceKey).toBeTruthy(); + const startGraph = () => + fetch(`${baseUrl}/api/workspaces/${workspaceKey}/system-graph`, { + headers, + }); + const startSession = () => + server!.sessionManager.create({ + cwd: workspaceRoot, + harness: "claude-code", + }); + if (subscriberOrder === "graph-first") { + await startGraph(); + await startSession(); + } else { + await startSession(); + await startGraph(); + } + await new Promise((resolve) => setTimeout(resolve, 2_500)); + + await fs.mkdir(path.join(checkout, ".git")); + await vi.waitFor( + async () => { + const workflows = (await ( + await fetch(`${baseUrl}/api/workflows`, { headers }) + ).json()) as WorkflowInfo[]; + expect(workflows).toEqual([]); + }, + { timeout: 8_000, interval: 100 }, + ); + + const direct = await fetch(`${baseUrl}/api/workflows/scan`, { + method: "POST", + headers, + body: JSON.stringify({ root: checkout }), + }); + expect(direct.status).toBe(200); + await vi.waitFor(async () => { + const workflows = (await ( + await fetch(`${baseUrl}/api/workflows`, { headers }) + ).json()) as WorkflowInfo[]; + expect(workflows.map((workflow) => workflow.path)).toEqual([checkout]); + }); + }, + 20_000, + ); + + it( + "keeps staged session contexts invisible when publication is superseded and commits only the newest rows", + { timeout: 20_000 }, + async () => { + await scaffoldAgent(workspaceRoot, "initial"); + const stagingEntered = deferred(); + const releaseStaging = deferred(); + let blockNextStaging = false; + server = await startServer({ + port: 0, + bootToken: "test-token", + telemetryOptIn: false, + adapters: { "claude-code": fakeClaudeAdapter() }, + stateRoot, + launchDir: workspaceRoot, + autoCreateSession: false, + workflowDiscoveryTestHooks: { + afterContextStaging: async () => { + if (!blockNextStaging) return; + blockNextStaging = false; + stagingEntered.resolve(); + await releaseStaging.promise; + }, + }, + }); + const sessionRoots = [ + path.join(workspaceRoot, "session-a"), + path.join(workspaceRoot, "session-b"), + ]; + const baseUrl = `http://127.0.0.1:${server.port}`; + const headers = { + "X-Harness-Token": "test-token", + "Content-Type": "application/json", + }; + await vi.waitFor(async () => { + const workflows = (await ( + await fetch(`${baseUrl}/api/workflows`, { headers }) + ).json()) as WorkflowInfo[]; + expect(workflows.map((workflow) => workflow.name)).toEqual(["initial"]); + }); + await Promise.all( + sessionRoots.map((root) => fs.mkdir(root, { recursive: true })), + ); + await Promise.all( + sessionRoots.map((cwd) => + server!.sessionManager.create({ cwd, harness: "claude-code" }), + ), + ); + const readAgentNames = async (cwd: string): Promise => { + const context = JSON.parse( + await fs.readFile( + path.join(cwd, ".sapiom", "harness-context.json"), + "utf8", + ), + ) as { agents: Array<{ name: string }> }; + return context.agents.map((agent) => agent.name).sort(); + }; + await vi.waitFor(async () => { + for (const cwd of sessionRoots) { + expect(await readAgentNames(cwd)).toEqual(["initial"]); + } + }); + + const intermediate = await scaffoldAgent(workspaceRoot, "intermediate"); + blockNextStaging = true; + const first = fetch(`${baseUrl}/api/workflows/scan`, { + method: "POST", + headers, + body: JSON.stringify({ root: workspaceRoot }), + }); + await stagingEntered.promise; + for (const cwd of sessionRoots) { + expect(await readAgentNames(cwd)).toEqual(["initial"]); + } + + await fs.rm(intermediate, { recursive: true, force: true }); + await scaffoldAgent(workspaceRoot, "newest"); + const second = fetch(`${baseUrl}/api/workflows/scan`, { + method: "POST", + headers, + body: JSON.stringify({ root: workspaceRoot }), + }); + releaseStaging.resolve(); + expect( + (await Promise.all([first, second])).map((response) => response.status), + ).toEqual([200, 200]); + + await vi.waitFor(async () => { + const workflows = (await ( + await fetch(`${baseUrl}/api/workflows`, { headers }) + ).json()) as WorkflowInfo[]; + expect(workflows.map((workflow) => workflow.name).sort()).toEqual([ + "initial", + "newest", + ]); + for (const cwd of sessionRoots) { + expect(await readAgentNames(cwd)).toEqual(["initial", "newest"]); + } + }); + }, + ); + + it("publishes globally when one active session context cannot be staged", async () => { + await scaffoldAgent(workspaceRoot, "initial"); + server = await startServer({ + port: 0, + bootToken: "test-token", + telemetryOptIn: false, + adapters: { "claude-code": fakeClaudeAdapter() }, + stateRoot, + launchDir: workspaceRoot, + autoCreateSession: false, + }); + const baseUrl = `http://127.0.0.1:${server.port}`; + const headers = { + "X-Harness-Token": "test-token", + "Content-Type": "application/json", + }; + await vi.waitFor(async () => { + expect( + (await ( + await fetch(`${baseUrl}/api/workflows`, { headers }) + ).json()) as WorkflowInfo[], + ).toHaveLength(1); + }); + const goodCwd = path.join(workspaceRoot, "good-session"); + const badCwd = path.join(workspaceRoot, "bad-session"); + await Promise.all([ + fs.mkdir(goodCwd, { recursive: true }), + fs.mkdir(badCwd, { recursive: true }), + ]); + await Promise.all([ + server.sessionManager.create({ cwd: goodCwd, harness: "claude-code" }), + server.sessionManager.create({ cwd: badCwd, harness: "claude-code" }), + ]); + await fs.rm(path.join(badCwd, ".sapiom"), { + recursive: true, + force: true, + }); + await fs.writeFile(path.join(badCwd, ".sapiom"), "blocked"); + + await scaffoldAgent(workspaceRoot, "newest"); + const response = await fetch(`${baseUrl}/api/workflows/scan`, { + method: "POST", + headers, + body: JSON.stringify({ root: workspaceRoot }), + }); + expect(response.status).toBe(200); + const goodContext = JSON.parse( + await fs.readFile( + path.join(goodCwd, ".sapiom", "harness-context.json"), + "utf8", + ), + ) as { agents: Array<{ name: string }> }; + expect(goodContext.agents.map((agent) => agent.name).sort()).toEqual([ + "initial", + "newest", + ]); + const workflows = (await ( + await fetch(`${baseUrl}/api/workflows`, { headers }) + ).json()) as WorkflowInfo[]; + expect(workflows.map((workflow) => workflow.name).sort()).toEqual([ + "initial", + "newest", + ]); + }); + + it("does not re-promote evidence from a publication paused across the last watch lease", async () => { + const agentRoot = await scaffoldAgent(workspaceRoot, "offline-edit"); + const publicationGate = deferred(); + const publicationEntered = deferred(); + const reopenScanGate = deferred(); + let blockPublication = false; + let blockReopenScan = false; + server = await startServer({ + port: 0, + bootToken: "test-token", + telemetryOptIn: false, + adapters: {}, + stateRoot, + launchDir: workspaceRoot, + autoCreateSession: false, + workflowDiscoveryTestHooks: { + beforePublication: async () => { + if (!blockPublication) return; + blockPublication = false; + publicationEntered.resolve(); + await publicationGate.promise; + }, + beforeScan: async () => { + if (blockReopenScan) await reopenScanGate.promise; + }, + }, + }); + const baseUrl = `http://127.0.0.1:${server.port}`; + const headers = { + "X-Harness-Token": "test-token", + "Content-Type": "application/json", + }; + await vi.waitFor(async () => { + const workflows = (await ( + await fetch(`${baseUrl}/api/workflows`, { headers }) + ).json()) as WorkflowInfo[]; + expect(workflows).toHaveLength(1); + }); + const initialState = (await ( + await fetch(`${baseUrl}/api/state`, { headers }) + ).json()) as AppState; + const workspaceKey = initialState.workspaceScopes?.find( + (scope) => scope.cwd === workspaceRoot, + )?.workspaceKey; + expect(workspaceKey).toBeTruthy(); + const graphUrl = `${baseUrl}/api/workspaces/${workspaceKey}/system-graph`; + await fetch(graphUrl, { headers }); // acquire the only continuous lease + + blockPublication = true; + const oldScan = fetch(`${baseUrl}/api/workflows/scan`, { + method: "POST", + headers, + body: JSON.stringify({ root: workspaceRoot }), + }); + await publicationEntered.promise; + + await fs.writeFile( + path.join(stateRoot, "settings.json"), + JSON.stringify({ recentDirs: [] }), + ); + await fetch(`${baseUrl}/api/state`, { headers }); // retires the last lease + await fs.rm(path.join(agentRoot, "sapiom.json")); // unobserved interval + publicationGate.resolve(); + expect((await oldScan).status).toBe(200); + + await fs.writeFile( + path.join(stateRoot, "settings.json"), + JSON.stringify({ recentDirs: [workspaceRoot] }), + ); + const restoredState = (await ( + await fetch(`${baseUrl}/api/state`, { headers }) + ).json()) as AppState; + expect( + restoredState.workspaceScopes?.some( + (scope) => scope.workspaceKey === workspaceKey, + ), + ).toBe(true); + blockReopenScan = true; + + const reopened = (await ( + await fetch(graphUrl, { headers }) + ).json()) as SystemGraphSnapshot; + + expect(reopened.state).toBe("degraded"); + expect(reopened.state).not.toBe("ready"); + expect(reopened.graph?.nodes).toHaveLength(1); + reopenScanGate.resolve(); + }); + it( "registers each agent once when the launch directory is a symlink", { timeout: 30_000 }, @@ -338,9 +1107,12 @@ describe("workspace graph freshness wiring", () => { const readGraph = async (): Promise => (await ( - await fetch(`${baseUrl}/api/workspaces/${workspaceKey}/system-graph`, { - headers, - }) + await fetch( + `${baseUrl}/api/workspaces/${workspaceKey}/system-graph`, + { + headers, + }, + ) ).json()) as SystemGraphSnapshot; await vi.waitFor( @@ -350,9 +1122,8 @@ describe("workspace graph freshness wiring", () => { { timeout: 8_000, interval: 150 }, ); - // The duplicate rows only appear once a SECOND scan runs under the - // resolved spelling, which is what a graph refresh does. Adding an agent - // is the cheapest way to make the watcher trigger one. + // A second scan under the resolved spelling must not register duplicate + // rows or make the existing invocation target ambiguous. await scaffoldAgent(workspaceRoot, "reporting"); await vi.waitFor( async () => { diff --git a/packages/harness/src/server/system-graph.test.ts b/packages/harness/src/server/system-graph.test.ts index dbca7ba93..7e1014db5 100644 --- a/packages/harness/src/server/system-graph.test.ts +++ b/packages/harness/src/server/system-graph.test.ts @@ -28,7 +28,7 @@ const graph: SystemGraph = { from: "agent:research", to: "agent:growth", kind: "invokes", - basis: "static", + basis: "static-invocation", mode: "async", }, ], diff --git a/packages/harness/src/server/system-graph.ts b/packages/harness/src/server/system-graph.ts index cfbb31292..40b36366b 100644 --- a/packages/harness/src/server/system-graph.ts +++ b/packages/harness/src/server/system-graph.ts @@ -49,10 +49,12 @@ export function createSystemGraphRouter( return; } try { - await options.onScopeAccess?.(scope); + void Promise.resolve(options.onScopeAccess?.(scope)).catch(() => { + // Watcher/discovery setup is best-effort and detached. Graph reads + // are strictly cache-backed and never await a baseline or scan. + }); } catch { - // Watcher setup is best-effort. A graph read must remain available - // even when automatic freshness cannot be armed. + // Synchronous setup failures are best-effort too. } await (refresh ? (options.onScopeRefresh?.(scope) ?? options.store.refresh(scope)) @@ -87,9 +89,12 @@ export function createSystemGraphRouter( return; } try { - await options.onScopeAccess?.(scope); + void Promise.resolve(options.onScopeAccess?.(scope)).catch(() => { + // Resolver reads remain cache-backed while freshness arms in the + // background. + }); } catch { - // Resolver reads remain available when freshness watching cannot arm. + // Synchronous setup failures are best-effort too. } await options.store.ensureInitialized(scope); if (!options.store.peek(scope.workspaceKey)) { diff --git a/packages/harness/src/server/workflow-graph.test.ts b/packages/harness/src/server/workflow-graph.test.ts index 4243300a4..207425f70 100644 --- a/packages/harness/src/server/workflow-graph.test.ts +++ b/packages/harness/src/server/workflow-graph.test.ts @@ -30,7 +30,9 @@ const GRAPH: CanvasGraph = { }; /** A successful derivation, as `deriveWorkflowCanvas` would return it. */ -function okDerivation(overrides: Partial = {}): WorkflowCanvasDerivation { +function okDerivation( + overrides: Partial = {}, +): WorkflowCanvasDerivation { return { status: "ok", graph: GRAPH, @@ -56,20 +58,26 @@ describe("GET /api/workflows/:path/graph", () => { /** GET the route for `agentPath`, encoded the way the SPA encodes it. */ async function get(agentPath: string): Promise { - return fetch(`${baseUrl}/api/workflows/${encodeURIComponent(agentPath)}/graph`); + return fetch( + `${baseUrl}/api/workflows/${encodeURIComponent(agentPath)}/graph`, + ); } async function makeTmpDir(): Promise { // realpath: macOS hands back /var, which is a symlink to /private/var — // the route realpaths too, so the fixture must compare like with like. - const dir = await fs.realpath(await fs.mkdtemp(path.join(os.tmpdir(), "wf-graph-"))); + const dir = await fs.realpath( + await fs.mkdtemp(path.join(os.tmpdir(), "wf-graph-")), + ); tmpDirs.push(dir); return dir; } afterEach(async () => { await new Promise((r) => server.close(() => r())); - await Promise.all(tmpDirs.splice(0).map((d) => fs.rm(d, { recursive: true, force: true }))); + await Promise.all( + tmpDirs.splice(0).map((d) => fs.rm(d, { recursive: true, force: true })), + ); }); // ------------------------------------------------------------------------- @@ -77,10 +85,17 @@ describe("GET /api/workflows/:path/graph", () => { // ------------------------------------------------------------------------- it("returns a graph derived from disk for a registered agent, with no session involved", async () => { - const deriveCanvas = vi.fn().mockResolvedValue(okDerivation({ cached: true })); + const deriveCanvas = vi + .fn() + .mockResolvedValue(okDerivation({ cached: true })); start({ - resolveWorkflow: () => ({ path: "/registered/agent", name: "order-triage", definitionId: 7 }), - inspectMarker: () => Promise.resolve({ status: "valid", marker: { definitionId: 7 } }), + resolveWorkflow: () => ({ + path: "/registered/agent", + name: "order-triage", + definitionId: 7, + }), + inspectMarker: () => + Promise.resolve({ status: "valid", marker: { definitionId: 7 } }), realpath: (p) => Promise.resolve(p), deriveCanvas, }); @@ -98,18 +113,67 @@ describe("GET /api/workflows/:path/graph", () => { expect(body.document).toContain(""); // The registry's identity is what gets rendered — badges and the panel // title come from it, exactly as the session-bound render builds them. - expect(deriveCanvas).toHaveBeenCalledWith({ - path: "/registered/agent", - name: "order-triage", - definitionId: 7, - activeBuildRunStatus: null, + expect(deriveCanvas).toHaveBeenCalledWith( + { + path: "/registered/agent", + name: "order-triage", + definitionId: 7, + activeBuildRunStatus: null, + }, + { authorizeBeforeExtraction: expect.any(Function) }, + ); + }); + + it("rechecks marker proof at the extractor launch boundary", async () => { + let markerPresent = true; + const actualExtractorLaunch = vi.fn(); + const inspectMarker = vi.fn(async () => + markerPresent + ? ({ status: "valid", marker: {} } as const) + : ({ status: "absent" } as const), + ); + start({ + resolveWorkflow: () => ({ + path: "/registered/agent", + name: "order-triage", + definitionId: null, + }), + inspectMarker, + realpath: (p) => Promise.resolve(p), + deriveCanvas: async (_workflow, options) => { + // Models the cache's asynchronous dependency/fingerprint window. + markerPresent = false; + if (await options?.authorizeBeforeExtraction?.()) { + actualExtractorLaunch(); + return okDerivation(); + } + return okDerivation({ + status: "cancelled", + graph: null, + enrichment: null, + reason: null, + }); + }, }); + + const body = (await ( + await get("/registered/agent") + ).json()) as WorkflowGraphResponse; + + expect(actualExtractorLaunch).not.toHaveBeenCalled(); + expect(body.status).toBe("empty"); + expect(body.reason).toContain("no sapiom.json"); }); it("serves an agent that has never had a session — nothing in the request names one", async () => { start({ - resolveWorkflow: () => ({ path: "/never/sessioned", name: "fresh", definitionId: null }), - inspectMarker: () => Promise.resolve({ status: "valid", marker: { definitionId: null } }), + resolveWorkflow: () => ({ + path: "/never/sessioned", + name: "fresh", + definitionId: null, + }), + inspectMarker: () => + Promise.resolve({ status: "valid", marker: { definitionId: null } }), realpath: (p) => Promise.resolve(p), deriveCanvas: () => Promise.resolve(okDerivation()), }); @@ -122,7 +186,11 @@ describe("GET /api/workflows/:path/graph", () => { it("reads sapiom.json off real disk through the default marker inspection", async () => { const dir = await makeTmpDir(); - await fs.writeFile(path.join(dir, "sapiom.json"), JSON.stringify({ definitionId: 42 }), "utf8"); + await fs.writeFile( + path.join(dir, "sapiom.json"), + JSON.stringify({ definitionId: 42 }), + "utf8", + ); start({ resolveWorkflow: () => ({ path: dir, name: "on-disk", definitionId: 42 }), deriveCanvas: () => Promise.resolve(okDerivation()), @@ -145,16 +213,39 @@ describe("GET /api/workflows/:path/graph", () => { // resolves through an ancestor node_modules, exactly as a real installed // agent project does; a bare os.tmpdir() copy would read as deps-missing // and render the "preparing" placeholder instead. - const packageDir = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..", ".."); - const agent = await fs.realpath(await fs.mkdtemp(path.join(packageDir, ".tmp-wf-graph-"))); + const packageDir = path.resolve( + path.dirname(fileURLToPath(import.meta.url)), + "..", + "..", + ); + const agent = await fs.realpath( + await fs.mkdtemp(path.join(packageDir, ".tmp-wf-graph-")), + ); tmpDirs.push(agent); await fs.copyFile( - path.join(packageDir, "src", "core", "__fixtures__", "order-triage", "index.ts"), + path.join( + packageDir, + "src", + "core", + "__fixtures__", + "order-triage", + "index.ts", + ), path.join(agent, "index.ts"), ); - await fs.writeFile(path.join(agent, "sapiom.json"), JSON.stringify({ definitionId: null }), "utf8"); + await fs.writeFile( + path.join(agent, "sapiom.json"), + JSON.stringify({ definitionId: null }), + "utf8", + ); - start({ resolveWorkflow: () => ({ path: agent, name: "order-triage", definitionId: null }) }); + start({ + resolveWorkflow: () => ({ + path: agent, + name: "order-triage", + definitionId: null, + }), + }); const res = await get(agent); const body = (await res.json()) as WorkflowGraphResponse; @@ -162,7 +253,13 @@ describe("GET /api/workflows/:path/graph", () => { expect(res.status).toBe(200); expect(body.status).toBe("ok"); expect(body.graph?.nodes.map((n) => n.id)).toEqual( - expect.arrayContaining(["intake", "classify", "route", "auto_resolve", "escalate"]), + expect.arrayContaining([ + "intake", + "classify", + "route", + "auto_resolve", + "escalate", + ]), ); expect(body.graph?.edges.length).toBeGreaterThan(0); expect(body.enrichment?.summary).toBeTruthy(); @@ -190,29 +287,57 @@ describe("GET /api/workflows/:path/graph", () => { }); it.each([ - { label: "a `..` climb", input: "/registered/agent/../../etc", error: "agent path must not contain a '..' segment" }, - { label: "a bare `..` segment", input: "/../etc/passwd", error: "agent path must not contain a '..' segment" }, - { label: "a relative path", input: "registered/agent", error: "agent path must be absolute" }, + { + label: "a `..` climb", + input: "/registered/agent/../../etc", + error: "agent path must not contain a '..' segment", + }, + { + label: "a bare `..` segment", + input: "/../etc/passwd", + error: "agent path must not contain a '..' segment", + }, + { + label: "a relative path", + input: "registered/agent", + error: "agent path must be absolute", + }, { label: "an empty path", input: " ", error: "agent path is required" }, - ])("400s $label without consulting the registry", async ({ input, error }) => { - const resolveWorkflow = vi.fn().mockReturnValue({ path: "/x", name: "x", definitionId: null }); - const deriveCanvas = vi.fn(); - start({ resolveWorkflow, inspectMarker: () => Promise.resolve({ status: "valid", marker: {} }), deriveCanvas }); - - const res = await get(input); - - expect(res.status).toBe(400); - expect(await res.json()).toEqual({ error }); - expect(resolveWorkflow).not.toHaveBeenCalled(); - expect(deriveCanvas).not.toHaveBeenCalled(); - }); + ])( + "400s $label without consulting the registry", + async ({ input, error }) => { + const resolveWorkflow = vi + .fn() + .mockReturnValue({ path: "/x", name: "x", definitionId: null }); + const deriveCanvas = vi.fn(); + start({ + resolveWorkflow, + inspectMarker: () => Promise.resolve({ status: "valid", marker: {} }), + deriveCanvas, + }); + + const res = await get(input); + + expect(res.status).toBe(400); + expect(await res.json()).toEqual({ error }); + expect(resolveWorkflow).not.toHaveBeenCalled(); + expect(deriveCanvas).not.toHaveBeenCalled(); + }, + ); it("rejects traversal even when normalization would land on a registered path", async () => { // `/registered/agent/../agent` normalizes to `/registered/agent`, which IS // registered. Resolving first and asking questions later would have served // it; the raw-value guard refuses the shape outright. - const resolveWorkflow = vi.fn().mockReturnValue({ path: "/registered/agent", name: "a", definitionId: null }); - start({ resolveWorkflow, deriveCanvas: () => Promise.resolve(okDerivation()) }); + const resolveWorkflow = vi.fn().mockReturnValue({ + path: "/registered/agent", + name: "a", + definitionId: null, + }); + start({ + resolveWorkflow, + deriveCanvas: () => Promise.resolve(okDerivation()), + }); const res = await get("/registered/agent/../agent"); @@ -227,11 +352,18 @@ describe("GET /api/workflows/:path/graph", () => { await fs.mkdir(agent); await fs.mkdir(outside); await fs.writeFile(path.join(outside, "secrets.json"), "{}", "utf8"); - await fs.symlink(path.join(outside, "secrets.json"), path.join(agent, "sapiom.json")); + await fs.symlink( + path.join(outside, "secrets.json"), + path.join(agent, "sapiom.json"), + ); const inspectMarker = vi.fn(); start({ - resolveWorkflow: () => ({ path: agent, name: "agent", definitionId: null }), + resolveWorkflow: () => ({ + path: agent, + name: "agent", + definitionId: null, + }), inspectMarker, deriveCanvas: () => Promise.resolve(okDerivation()), }); @@ -239,7 +371,9 @@ describe("GET /api/workflows/:path/graph", () => { const res = await get(agent); expect(res.status).toBe(400); - expect(await res.json()).toEqual({ error: "sapiom.json resolves outside the agent directory" }); + expect(await res.json()).toEqual({ + error: "sapiom.json resolves outside the agent directory", + }); expect(inspectMarker).not.toHaveBeenCalled(); }); @@ -248,11 +382,19 @@ describe("GET /api/workflows/:path/graph", () => { const real = path.join(dir, "real"); const link = path.join(dir, "link"); await fs.mkdir(real); - await fs.writeFile(path.join(real, "sapiom.json"), JSON.stringify({ definitionId: null }), "utf8"); + await fs.writeFile( + path.join(real, "sapiom.json"), + JSON.stringify({ definitionId: null }), + "utf8", + ); await fs.symlink(real, link); start({ - resolveWorkflow: () => ({ path: link, name: "linked", definitionId: null }), + resolveWorkflow: () => ({ + path: link, + name: "linked", + definitionId: null, + }), deriveCanvas: () => Promise.resolve(okDerivation()), }); @@ -263,36 +405,58 @@ describe("GET /api/workflows/:path/graph", () => { }); it.each([ - { status: "absent" as const, reason: "This agent has no sapiom.json, so there is no graph to render yet." }, - { status: "invalid" as const, reason: "This agent's sapiom.json is not valid JSON, so its graph can't be read." }, - { status: "unreadable" as const, reason: "This agent's sapiom.json could not be read." }, - ])("returns 200 + an explicit empty graph when the marker is $status", async ({ status, reason }) => { - const deriveCanvas = vi.fn(); - start({ - resolveWorkflow: () => ({ path: "/registered/agent", name: "order-triage", definitionId: null }), - inspectMarker: () => Promise.resolve({ status }), - realpath: (p) => Promise.resolve(p), - deriveCanvas, - }); + { + status: "absent" as const, + reason: + "This agent has no sapiom.json, so there is no graph to render yet.", + }, + { + status: "invalid" as const, + reason: + "This agent's sapiom.json is not valid JSON, so its graph can't be read.", + }, + { + status: "unreadable" as const, + reason: "This agent's sapiom.json could not be read.", + }, + ])( + "returns 200 + an explicit empty graph when the marker is $status", + async ({ status, reason }) => { + const deriveCanvas = vi.fn(); + start({ + resolveWorkflow: () => ({ + path: "/registered/agent", + name: "order-triage", + definitionId: null, + }), + inspectMarker: () => Promise.resolve({ status }), + realpath: (p) => Promise.resolve(p), + deriveCanvas, + }); - const res = await get("/registered/agent"); - const body = (await res.json()) as WorkflowGraphResponse; + const res = await get("/registered/agent"); + const body = (await res.json()) as WorkflowGraphResponse; - // 200, not 404 and not 422: absent ⇒ empty, not an error. A consumer tells - // this apart from "no route" by the status code alone, and apart from a - // real board by `status`. - expect(res.status).toBe(200); - expect(body.status).toBe("empty"); - expect(body.graph).toBeNull(); - expect(body.reason).toBe(reason); - // Still a renderable page, so the pane is never mutely blank. - expect(body.document).toContain("Nothing rendered yet"); - expect(deriveCanvas).not.toHaveBeenCalled(); - }); + // 200, not 404 and not 422: absent ⇒ empty, not an error. A consumer tells + // this apart from "no route" by the status code alone, and apart from a + // real board by `status`. + expect(res.status).toBe(200); + expect(body.status).toBe("empty"); + expect(body.graph).toBeNull(); + expect(body.reason).toBe(reason); + // Still a renderable page, so the pane is never mutely blank. + expect(body.document).toContain("Nothing rendered yet"); + expect(deriveCanvas).not.toHaveBeenCalled(); + }, + ); it("returns 200 empty (not 404) for a registered agent whose directory is gone", async () => { start({ - resolveWorkflow: () => ({ path: "/registered/vanished", name: "vanished", definitionId: null }), + resolveWorkflow: () => ({ + path: "/registered/vanished", + name: "vanished", + definitionId: null, + }), deriveCanvas: () => Promise.resolve(okDerivation()), }); @@ -320,10 +484,15 @@ describe("GET /api/workflows/:path/graph", () => { }; process.on("unhandledRejection", onRejection); start({ - resolveWorkflow: () => ({ path: "/registered/agent", name: "order-triage", definitionId: null }), + resolveWorkflow: () => ({ + path: "/registered/agent", + name: "order-triage", + definitionId: null, + }), inspectMarker: () => Promise.resolve({ status: "valid", marker: {} }), realpath: (p) => Promise.resolve(p), - deriveCanvas: () => Promise.reject(new Error("esbuild exited with code 1")), + deriveCanvas: () => + Promise.reject(new Error("esbuild exited with code 1")), }); try { @@ -349,13 +518,20 @@ describe("GET /api/workflows/:path/graph", () => { // The other unguarded await: reading `sapiom.json` off a directory that // EACCESes mid-read throws just as readily as the extraction does. start({ - resolveWorkflow: () => ({ path: "/registered/agent", name: "order-triage", definitionId: null }), - inspectMarker: () => Promise.reject(new Error("EACCES: permission denied")), + resolveWorkflow: () => ({ + path: "/registered/agent", + name: "order-triage", + definitionId: null, + }), + inspectMarker: () => + Promise.reject(new Error("EACCES: permission denied")), realpath: (p) => Promise.resolve(p), deriveCanvas: () => Promise.resolve(okDerivation()), }); - const body = (await (await get("/registered/agent")).json()) as WorkflowGraphResponse; + const body = (await ( + await get("/registered/agent") + ).json()) as WorkflowGraphResponse; expect(body.status).toBe("error"); expect(body.reason).toContain("EACCES"); @@ -370,15 +546,23 @@ describe("GET /api/workflows/:path/graph", () => { // with a different directory than the one asked for is what proves it: // every disk call, and the reported path, follow the registry. const realpath = vi.fn((p: string) => Promise.resolve(p)); - const inspectMarker = vi.fn(() => Promise.resolve({ status: "valid" as const, marker: {} })); + const inspectMarker = vi.fn(() => + Promise.resolve({ status: "valid" as const, marker: {} }), + ); start({ - resolveWorkflow: () => ({ path: "/registry/says/here", name: "order-triage", definitionId: null }), + resolveWorkflow: () => ({ + path: "/registry/says/here", + name: "order-triage", + definitionId: null, + }), inspectMarker, realpath, deriveCanvas: () => Promise.resolve(okDerivation()), }); - const body = (await (await get("/request/says/there")).json()) as WorkflowGraphResponse; + const body = (await ( + await get("/request/says/there") + ).json()) as WorkflowGraphResponse; expect(realpath).toHaveBeenCalledWith("/registry/says/here"); expect(realpath).not.toHaveBeenCalledWith("/request/says/there"); @@ -387,26 +571,47 @@ describe("GET /api/workflows/:path/graph", () => { }); it.each([ - { derived: { status: "error" as const, reason: "Could not resolve @sapiom/agent" }, expectReason: "Could not resolve @sapiom/agent" }, - { derived: { status: "preparing" as const, reason: null }, expectReason: null }, - ])("passes through a $derived.status derivation as a 200", async ({ derived, expectReason }) => { - start({ - resolveWorkflow: () => ({ path: "/registered/agent", name: "order-triage", definitionId: null }), - inspectMarker: () => Promise.resolve({ status: "valid", marker: {} }), - realpath: (p) => Promise.resolve(p), - deriveCanvas: () => - Promise.resolve( - okDerivation({ ...derived, graph: null, enrichment: null, document: "panel" }), - ), - }); + { + derived: { + status: "error" as const, + reason: "Could not resolve @sapiom/agent", + }, + expectReason: "Could not resolve @sapiom/agent", + }, + { + derived: { status: "preparing" as const, reason: null }, + expectReason: null, + }, + ])( + "passes through a $derived.status derivation as a 200", + async ({ derived, expectReason }) => { + start({ + resolveWorkflow: () => ({ + path: "/registered/agent", + name: "order-triage", + definitionId: null, + }), + inspectMarker: () => Promise.resolve({ status: "valid", marker: {} }), + realpath: (p) => Promise.resolve(p), + deriveCanvas: () => + Promise.resolve( + okDerivation({ + ...derived, + graph: null, + enrichment: null, + document: "panel", + }), + ), + }); - const res = await get("/registered/agent"); - const body = (await res.json()) as WorkflowGraphResponse; + const res = await get("/registered/agent"); + const body = (await res.json()) as WorkflowGraphResponse; - expect(res.status).toBe(200); - expect(body.status).toBe(derived.status); - expect(body.graph).toBeNull(); - expect(body.reason).toBe(expectReason); - expect(body.document).toContain(""); - }); + expect(res.status).toBe(200); + expect(body.status).toBe(derived.status); + expect(body.graph).toBeNull(); + expect(body.reason).toBe(expectReason); + expect(body.document).toContain(""); + }, + ); }); diff --git a/packages/harness/src/server/workflow-graph.ts b/packages/harness/src/server/workflow-graph.ts index 87a33807a..c7e5efe7c 100644 --- a/packages/harness/src/server/workflow-graph.ts +++ b/packages/harness/src/server/workflow-graph.ts @@ -65,7 +65,10 @@ import { type AgentProjectMarkerInspection, } from "../core/agent-project-discovery.js"; import { renderCanvasMessageDocument } from "../core/canvas-template.js"; -import { deriveWorkflowCanvas, type RenderableWorkflow } from "../core/canvas-render.js"; +import { + deriveWorkflowCanvas, + type RenderableWorkflow, +} from "../core/canvas-render.js"; import type { CanvasGraph } from "../core/canvas-graph.js"; import type { CanvasEnrichment } from "../core/canvas-enrichment.js"; import { hasTraversalSegment, resolveWithinRoot } from "../core/path-safety.js"; @@ -113,13 +116,20 @@ export interface WorkflowGraphRouterDeps { const EMPTY_REASONS = { gone: "This agent's directory is no longer on disk.", absent: "This agent has no sapiom.json, so there is no graph to render yet.", - invalid: "This agent's sapiom.json is not valid JSON, so its graph can't be read.", + invalid: + "This agent's sapiom.json is not valid JSON, so its graph can't be read.", unreadable: "This agent's sapiom.json could not be read.", + changed: + "This agent's marker changed before its graph extraction could start.", } as const; /** The empty board — the same message document server/canvas.ts serves, with * the specific reason as its subtitle so the pane is never mutely blank. */ -function emptyResponse(agentPath: string, name: string, reason: string): WorkflowGraphResponse { +function emptyResponse( + agentPath: string, + name: string, + reason: string, +): WorkflowGraphResponse { return { path: agentPath, name, @@ -142,7 +152,11 @@ function emptyResponse(agentPath: string, name: string, reason: string): Workflo * agent's graph could not be extracted" — the same distinction * `server/actions.ts` preserves when its input-contract extraction fails. */ -function errorResponse(agentPath: string, name: string, reason: string): WorkflowGraphResponse { +function errorResponse( + agentPath: string, + name: string, + reason: string, +): WorkflowGraphResponse { return { path: agentPath, name, @@ -161,7 +175,9 @@ function failureReason(err: unknown): string { return `Studio couldn't read this agent's graph: ${detail || "the derivation failed."}`; } -export function createWorkflowGraphRouter(deps: WorkflowGraphRouterDeps): ExpressRouter { +export function createWorkflowGraphRouter( + deps: WorkflowGraphRouterDeps, +): ExpressRouter { const inspectMarker = deps.inspectMarker ?? inspectAgentProjectMarker; const deriveCanvas = deps.deriveCanvas ?? deriveWorkflowCanvas; const realpath = deps.realpath ?? ((p: string) => fsp.realpath(p)); @@ -178,7 +194,9 @@ export function createWorkflowGraphRouter(deps: WorkflowGraphRouterDeps): Expres // error, not something normalization silently lands on another registered // path. (path-safety.ts's segment-aware test — "a..b" is a normal name.) if (hasTraversalSegment(raw)) { - res.status(400).json({ error: "agent path must not contain a '..' segment" }); + res + .status(400) + .json({ error: "agent path must not contain a '..' segment" }); return; } if (!path.isAbsolute(raw)) { @@ -213,9 +231,13 @@ export function createWorkflowGraphRouter(deps: WorkflowGraphRouterDeps): Expres // agent DIRECTORY is legitimate (the user registered it, and realDir is // its target), but a `sapiom.json` symlinked out of the project would make // a path-keyed read endpoint into a file reader — refuse that outright. - const marker = await realpath(path.join(realDir, "sapiom.json")).catch(() => null); + const marker = await realpath(path.join(realDir, "sapiom.json")).catch( + () => null, + ); if (marker !== null && resolveWithinRoot(realDir, marker) === null) { - res.status(400).json({ error: "sapiom.json resolves outside the agent directory" }); + res + .status(400) + .json({ error: "sapiom.json resolves outside the agent directory" }); return; } @@ -226,16 +248,33 @@ export function createWorkflowGraphRouter(deps: WorkflowGraphRouterDeps): Expres try { const inspection = await inspectMarker(realDir); if (inspection.status !== "valid") { - res.json(emptyResponse(agentPath, name, EMPTY_REASONS[inspection.status])); + res.json( + emptyResponse(agentPath, name, EMPTY_REASONS[inspection.status]), + ); return; } - const derived = await deriveCanvas({ - path: workflow.path, - name, - definitionId: workflow.definitionId ?? null, - activeBuildRunStatus: workflow.activeBuildRunStatus ?? null, - }); + const derived = await deriveCanvas( + { + path: workflow.path, + name, + definitionId: workflow.definitionId ?? null, + activeBuildRunStatus: workflow.activeBuildRunStatus ?? null, + }, + { + authorizeBeforeExtraction: async () => + (await inspectMarker(realDir)).status === "valid", + }, + ); + if (derived.status === "cancelled") { + const currentMarker = await inspectMarker(realDir); + const reason = + currentMarker.status === "valid" + ? EMPTY_REASONS.changed + : EMPTY_REASONS[currentMarker.status]; + res.json(emptyResponse(agentPath, name, reason)); + return; + } res.json({ path: agentPath, diff --git a/packages/harness/src/shared/system-graph.ts b/packages/harness/src/shared/system-graph.ts index 1fd10396a..3f7897bfc 100644 --- a/packages/harness/src/shared/system-graph.ts +++ b/packages/harness/src/shared/system-graph.ts @@ -112,14 +112,16 @@ export interface SystemGraphNode { export type AgentInvocationMode = "blocking" | "async"; -export interface SystemGraphEdge { +export interface StaticInvocationGraphEdge { from: string; to: string; kind: "invokes"; - basis: "static"; + basis: "static-invocation"; mode: AgentInvocationMode; } +export type SystemGraphEdge = StaticInvocationGraphEdge; + export interface GraphWarning { code: | "unresolved-target" diff --git a/packages/harness/src/shared/types.ts b/packages/harness/src/shared/types.ts index 5affa9e0e..82a33bd67 100644 --- a/packages/harness/src/shared/types.ts +++ b/packages/harness/src/shared/types.ts @@ -1470,7 +1470,13 @@ export interface HarnessSettings { * server/rest.ts and the picker are all derived from this tuple, so adding an * editor is a one-line change here plus its label. */ -export const EDITOR_KINDS = ["vscode", "vscode-insiders", "cursor", "windsurf", "zed"] as const; +export const EDITOR_KINDS = [ + "vscode", + "vscode-insiders", + "cursor", + "windsurf", + "zed", +] as const; export type EditorKind = (typeof EDITOR_KINDS)[number]; @@ -1514,17 +1520,22 @@ export interface FsListResponse { // Workflows (left rail) // --------------------------------------------------------------------------- -/** An orchestration project on disk, identified by its sapiom.json marker. */ +/** + * An agent project known to Studio. It may have been identified by a valid + * `sapiom.json` marker or by static proof in its regular `index.ts` entrypoint. + */ export interface WorkflowInfo { - /** Directory name (or package.json name when present). */ + /** Stable display name, normally the package name or directory name. */ name: string; - /** Absolute path to the project directory (contains sapiom.json). */ + /** Absolute path to the project directory; a marker is not required. */ path: string; - /** From sapiom.json once linked; null before first link. */ + /** Cloud definition id once explicitly linked; null for local-only rows. */ definitionId: number | null; - /** The deployed agent's slug — the `defineAgent({ name })` that sapiom.json - * caches as `name`, used as the executions-API handle - * (`/agents/v1/definitions/{slug}/executions`). Null before first link. */ + /** + * Cloud definition slug cached by linking/marker metadata, used as the + * executions-API handle (`/agents/v1/definitions/{slug}/executions`). This is + * null for source-only rows; Studio keeps static source identity private. + */ definitionSlug: string | null; /** * Cloud build evidence from the definition-detail projection. An id alone diff --git a/packages/harness/web/e2e/polsia-workspace.spec.ts b/packages/harness/web/e2e/polsia-workspace.spec.ts index 80eaee37d..54449dafa 100644 --- a/packages/harness/web/e2e/polsia-workspace.spec.ts +++ b/packages/harness/web/e2e/polsia-workspace.spec.ts @@ -34,6 +34,14 @@ test("a Polsia-style Project opens its complete graph without a session and reus await expect( project.getByTestId(`workflow-status-${POLSIA}/scripts/tools/rollup`), ).toHaveAttribute("data-deployed", "false"); + // `outreach` is the markerless syntax-discovered fixture: its public rail + // row has null cloud metadata, while the mock's private accepted evidence + // gives the graph its canonical source key (asserted below). + await expect( + project.getByTestId( + `workflow-status-${POLSIA}/backend/src/agents/outreach`, + ), + ).toHaveAttribute("data-deployed", "false"); await expect(page.getByTestId("session-context")).not.toHaveAttribute( "data-session-id", /.+/, @@ -59,7 +67,9 @@ test("a Polsia-style Project opens its complete graph without a session and reus "queue", "local:scripts/tools/rollup", ]) { - await expect(page.getByTestId(`system-graph-node-${agentKey}`)).toBeVisible(); + await expect( + page.getByTestId(`system-graph-node-${agentKey}`), + ).toBeVisible(); } // The fixture exercises fan-out, fan-in, a cycle, mixed call modes, and one @@ -81,9 +91,7 @@ test("a Polsia-style Project opens its complete graph without a session and reus 7, ); await expect( - page.locator( - '[data-testid^="system-graph-edge-"][data-testid*="rollup"]', - ), + page.locator('[data-testid^="system-graph-edge-"][data-testid*="rollup"]'), ).toHaveCount(0); await expect.poll(() => graphRequestCount(page)).toBe(1); diff --git a/packages/harness/web/src/lib/api.test.ts b/packages/harness/web/src/lib/api.test.ts index d84b8c12e..dec7d1555 100644 --- a/packages/harness/web/src/lib/api.test.ts +++ b/packages/harness/web/src/lib/api.test.ts @@ -96,6 +96,152 @@ describe("MockApi deterministic system graph identity and navigation", () => { ); }); + it("gives proven source identity precedence over a legacy marker alias", () => { + const workflow: WorkflowInfo = { + name: "billing-package", + path: "/workspace/billing", + definitionId: null, + definitionSlug: "payments", + activeBuildRunId: null, + activeBuildRunStatus: null, + source: "scan", + }; + + const projection = projectMockSystemGraphInventory( + "/workspace", + [workflow], + { + [workflow.path]: { + kind: "source", + sourceDefinitionName: "billing", + }, + }, + ); + + expect(projection.nodes).toEqual([ + { + id: "agent:billing", + agentKey: "billing", + label: "billing-package", + }, + ]); + expect(projection.targets).toEqual([ + { agentKey: "billing", workflowPath: workflow.path }, + ]); + expect(projection.degraded).toBe(false); + }); + + it("keeps persisted unknown source identity visible but lifecycle-degraded", () => { + const workflow: WorkflowInfo = { + name: "billing-package", + path: "/workspace/billing", + definitionId: null, + definitionSlug: null, + source: "scan", + }; + + const projection = projectMockSystemGraphInventory( + "/workspace", + [workflow], + { + [workflow.path]: { + kind: "unknown", + sourceDefinitionName: "billing", + }, + }, + ); + + expect(projection.nodes[0]?.agentKey).toBe("billing"); + expect(projection.degraded).toBe(true); + }); + + it("falls back deterministically for duplicate proven source identities", () => { + const workflows: WorkflowInfo[] = ["first", "second"].map((name) => ({ + name, + path: `/workspace/${name}`, + definitionId: null, + definitionSlug: null, + source: "scan", + })); + const evidence = Object.fromEntries( + workflows.map((workflow) => [ + workflow.path, + { kind: "source", sourceDefinitionName: "billing" } as const, + ]), + ); + + const projection = projectMockSystemGraphInventory( + "/workspace", + workflows, + evidence, + ); + + expect(projection.nodes.map((node) => node.agentKey)).toEqual([ + "local:first", + "local:second", + ]); + expect(projection.warnings).toEqual([ + { + code: "duplicate-agent-key", + agentKey: "billing", + message: + "Multiple agents use billing; kept each with a local identity.", + }, + ]); + expect(projection.degraded).toBe(true); + }); + + it("invalidates mock rail and graph revisions across source add, edit, and delete", async () => { + const events = await import("./events"); + const publish = vi.spyOn(events, "publishMockBusMessage"); + const api = new MockApi(); + const state = await api.getState(); + const scope = state.workspaceScopes?.find( + (candidate) => candidate.cwd === "/Users/demo/rfq-agent", + ); + expect(scope).toBeDefined(); + const before = await api.getSystemGraph(scope!.workspaceKey); + const row: WorkflowInfo = { + name: "rfq-package", + path: scope!.cwd, + definitionId: null, + definitionSlug: null, + activeBuildRunId: null, + activeBuildRunStatus: null, + source: "scan", + }; + + api.replaceSourceDiscoveredWorkflows([row], { + [row.path]: { kind: "source", sourceDefinitionName: "rfq-current" }, + }); + const added = await api.getSystemGraph(scope!.workspaceKey); + expect(added.revision).toBeGreaterThan(before.revision); + expect(added.graph?.nodes.map((node) => node.agentKey)).toEqual([ + "rfq-current", + ]); + + api.replaceSourceDiscoveredWorkflows([row], { + [row.path]: { kind: "source", sourceDefinitionName: "rfq-next" }, + }); + const edited = await api.getSystemGraph(scope!.workspaceKey); + expect(edited.revision).toBeGreaterThan(added.revision); + expect(edited.graph?.nodes.map((node) => node.agentKey)).toEqual([ + "rfq-next", + ]); + + api.replaceSourceDiscoveredWorkflows([], {}); + const removed = await api.getSystemGraph(scope!.workspaceKey); + expect(removed.revision).toBeGreaterThan(edited.revision); + expect(removed.graph?.nodes).toEqual([]); + await vi.waitFor(() => { + expect( + publish.mock.calls.filter( + ([message]) => message.type === "workflows.changed", + ), + ).toHaveLength(3); + }); + }); + it("uses projection warnings and lifecycle for non-special mock graphs", async () => { const api = new MockApi(); const scope = (await api.getState()).workspaceScopes?.find( diff --git a/packages/harness/web/src/lib/api.ts b/packages/harness/web/src/lib/api.ts index 0e35e3d1d..e8c2ebdfb 100644 --- a/packages/harness/web/src/lib/api.ts +++ b/packages/harness/web/src/lib/api.ts @@ -1203,7 +1203,9 @@ function recordAgentMove(entry: { from: string; to: string }): void { */ function recordCreateStep(kind: "scaffold" | "session", path: string): void { if (typeof window === "undefined") return; - const win = window as unknown as { __HARNESS_TEST__?: Record }; + const win = window as unknown as { + __HARNESS_TEST__?: Record; + }; const previous = (win.__HARNESS_TEST__?.createOrder as string[]) ?? []; win.__HARNESS_TEST__ = { ...(win.__HARNESS_TEST__ ?? {}), @@ -1339,63 +1341,63 @@ const MOCK_POLSIA_ROOT = "/Users/demo/polsia"; * A compact Polsia-style direct-call topology for the deep Project fixture. * Two source records for Outreach -> Mailer deliberately collapse into one * combined connector in the renderer. Rollup stays disconnected so inventory - * coverage is tested independently of relationship extraction. + * coverage is tested independently of direct invocation extraction. */ const MOCK_POLSIA_GRAPH_EDGES: SystemGraph["edges"] = [ { from: "agent:outreach", to: "agent:mailer", kind: "invokes", - basis: "static", + basis: "static-invocation", mode: "blocking", }, { from: "agent:outreach", to: "agent:mailer", kind: "invokes", - basis: "static", + basis: "static-invocation", mode: "async", }, { from: "agent:ads", to: "agent:gateway", kind: "invokes", - basis: "static", + basis: "static-invocation", mode: "blocking", }, { from: "agent:gateway", to: "agent:ads-worker", kind: "invokes", - basis: "static", + basis: "static-invocation", mode: "async", }, { from: "agent:gateway", to: "agent:queue", kind: "invokes", - basis: "static", + basis: "static-invocation", mode: "blocking", }, { from: "agent:ads-worker", to: "agent:queue", kind: "invokes", - basis: "static", + basis: "static-invocation", mode: "async", }, { from: "agent:queue", to: "agent:sender", kind: "invokes", - basis: "static", + basis: "static-invocation", mode: "blocking", }, { from: "agent:sender", to: "agent:gateway", kind: "invokes", - basis: "static", + basis: "static-invocation", mode: "async", }, ]; @@ -1438,10 +1440,23 @@ export interface MockSystemGraphProjection { degraded: boolean; } +/** Process-local discovery proof used by the browser mock. The real REST + * WorkflowInfo intentionally does not expose registry evidence, so mock graph + * projection receives the same information as a separate sidecar. */ +export interface MockWorkflowIdentityEvidence { + kind: "marker" | "source" | "not-agent" | "unknown"; + sourceDefinitionName?: string | null; +} + +export type MockWorkflowIdentityEvidenceByPath = Readonly< + Record +>; + /** Deterministic identity/navigation projection for the browser mock. */ export function projectMockSystemGraphInventory( scopeRoot: string, workflows: readonly WorkflowInfo[], + evidenceByPath: MockWorkflowIdentityEvidenceByPath = {}, ): MockSystemGraphProjection { const rows = workflows .filter((workflow) => isWithinDir(scopeRoot, workflow.path)) @@ -1449,11 +1464,35 @@ export function projectMockSystemGraphInventory( const inventoryPath = mockInventoryPath(scopeRoot, workflow.path); const fallbackKey = `local:${inventoryPath === "." ? "root" : inventoryPath}`; const marker = mockCanonicalIdentity(workflow.definitionSlug); + const evidence = evidenceByPath[workflow.path]; + const hasPersistedSourceName = + evidence !== undefined && + Object.prototype.hasOwnProperty.call(evidence, "sourceDefinitionName"); + const sourceName = hasPersistedSourceName + ? mockCanonicalIdentity(evidence.sourceDefinitionName ?? null) + : null; + const sourceIsAuthoritative = + evidence?.kind === "source" || + (evidence?.kind === "unknown" && hasPersistedSourceName); + // `unknown` may retain the last accepted syntax identity for continuity, + // but it can never make the graph ready until a fresh scan proves it. + const degraded = + evidence?.kind === "unknown" || + (evidence?.kind === "source" && sourceName === null); + const canonical = sourceIsAuthoritative + ? sourceName !== null + : evidence?.kind === "not-agent" + ? false + : marker !== null; return { workflow, inventoryPath, fallbackKey, - candidateKey: marker ?? fallbackKey, + candidateKey: sourceIsAuthoritative + ? (sourceName ?? fallbackKey) + : (marker ?? fallbackKey), + canonical, + degraded, }; }) .sort( @@ -1469,17 +1508,22 @@ export function projectMockSystemGraphInventory( samePath(candidate.workflow.path, row.workflow.path), ) === index, ); - const candidateCounts = new Map(); + const canonicalCounts = new Map(); + const provisionalCounts = new Map(); for (const row of rows) { - candidateCounts.set( - row.candidateKey, - (candidateCounts.get(row.candidateKey) ?? 0) + 1, - ); + const counts = row.canonical ? canonicalCounts : provisionalCounts; + counts.set(row.candidateKey, (counts.get(row.candidateKey) ?? 0) + 1); } const used = new Set(); const projected = rows.map((row) => { - const duplicated = (candidateCounts.get(row.candidateKey) ?? 0) > 1; - const base = duplicated ? row.fallbackKey : row.candidateKey; + const canonicalCount = canonicalCounts.get(row.candidateKey) ?? 0; + const provisionalCount = provisionalCounts.get(row.candidateKey) ?? 0; + const ambiguous = row.canonical + ? canonicalCount > 1 + : canonicalCount === 0 && provisionalCount > 1; + const shadowedByCanonical = !row.canonical && canonicalCount > 0; + const base = + ambiguous || shadowedByCanonical ? row.fallbackKey : row.candidateKey; let agentKey = base; let suffix = 2; while (used.has(agentKey)) { @@ -1494,12 +1538,18 @@ export function projectMockSystemGraphInventory( }; }); projected.sort((left, right) => codeUnitOrder(left.agentKey, right.agentKey)); - const duplicateCandidates = [...candidateCounts] - .filter( - ([candidateKey, count]) => - count > 1 && mockCanonicalIdentity(candidateKey) !== null, - ) - .map(([candidateKey]) => candidateKey) + const duplicateCandidates = [ + ...new Set([...canonicalCounts.keys(), ...provisionalCounts.keys()]), + ] + .filter((candidateKey) => { + const canonicalCount = canonicalCounts.get(candidateKey) ?? 0; + const provisionalCount = provisionalCounts.get(candidateKey) ?? 0; + return ( + (canonicalCount > 1 || + (canonicalCount === 0 && provisionalCount > 1)) && + mockCanonicalIdentity(candidateKey) !== null + ); + }) .sort(codeUnitOrder); return { nodes: projected.map(({ agentKey, label }) => ({ @@ -1516,7 +1566,8 @@ export function projectMockSystemGraphInventory( agentKey: candidateKey, message: `Multiple agents use ${candidateKey}; kept each with a local identity.`, })), - degraded: duplicateCandidates.length > 0, + degraded: + duplicateCandidates.length > 0 || rows.some((row) => row.degraded), }; } @@ -1615,6 +1666,21 @@ export class MockApi implements HarnessApi { ...MOCK_WORKFLOWS, ...(isSearchFixturesEnabled() ? MOCK_SEARCH_WORKFLOWS : []), ].map((workflow) => ({ ...workflow })); + /** Mock-only equivalent of the server's private accepted identity sidecar. */ + private workflowIdentityEvidenceStore: Record< + string, + MockWorkflowIdentityEvidence + > = Object.fromEntries( + this.workflowsStore + .filter( + (workflow) => + workflow.path === `${MOCK_POLSIA_ROOT}/backend/src/agents/outreach`, + ) + .map((workflow) => [ + workflow.path, + { kind: "source", sourceDefinitionName: "outreach" } as const, + ]), + ); /* * Every read of the fixtures goes through the move log (`mockMoves`), so a @@ -1636,6 +1702,31 @@ export class MockApi implements HarnessApi { this.invalidateSystemGraphProjections(); } + private get workflowIdentityEvidence(): MockWorkflowIdentityEvidenceByPath { + if (mockMoves.length === 0) return this.workflowIdentityEvidenceStore; + return Object.fromEntries( + Object.entries(this.workflowIdentityEvidenceStore).map( + ([workflowPath, evidence]) => [replayMockMoves(workflowPath), evidence], + ), + ); + } + + /** + * Mock/test mutation seam for the syntax-discovery lifecycle. It keeps the + * private proof sidecar out of WorkflowInfo while exercising the same rail + * event plus revisioned graph invalidation as production add/edit/delete. + */ + replaceSourceDiscoveredWorkflows( + workflows: readonly WorkflowInfo[], + evidenceByPath: MockWorkflowIdentityEvidenceByPath, + ): void { + this.workflowIdentityEvidenceStore = { ...evidenceByPath }; + this.workflows = workflows.map((workflow) => ({ ...workflow })); + void import("./events").then(({ publishMockBusMessage }) => { + publishMockBusMessage({ type: "workflows.changed" }); + }); + } + private allocateSystemGraphRevision(workspaceKey: WorkspaceKey): number { const revision = (this.systemGraphRevision.get(workspaceKey) ?? 0) + 1; this.systemGraphRevision.set(workspaceKey, revision); @@ -1894,47 +1985,48 @@ export class MockApi implements HarnessApi { from: "agent:research", to: "agent:growth", kind: "invokes", - basis: "static", + basis: "static-invocation", mode: "blocking", }, { from: "agent:research", to: "agent:growth", kind: "invokes", - basis: "static", + basis: "static-invocation", mode: "async", }, { from: "agent:research", to: "agent:leasing", kind: "invokes", - basis: "static", + basis: "static-invocation", mode: "async", }, { from: "agent:growth", to: "agent:research", kind: "invokes", - basis: "static", + basis: "static-invocation", mode: "async", }, { from: "agent:reporting", to: "agent:leasing", kind: "invokes", - basis: "static", + basis: "static-invocation", mode: "blocking", }, ], warnings: [], }; - // Keep the original relationship-rich graph for acme-app's graph behavior + // Keep the original invocation-rich graph for acme-app's graph behavior // specs. Every other mock project is an honest inventory projection of the // agents beneath that exact root, which lets Project-axis tests prove parent // and nested projects expose the same membership as the rail. const projection = projectMockSystemGraphInventory( selectedScope.cwd, this.workflows, + this.workflowIdentityEvidence, ); const graph = samePath(selectedScope.cwd, "/Users/demo/acme-app") ? fixtureGraph @@ -2424,7 +2516,11 @@ export class MockApi implements HarnessApi { await delay(180); const refusal = refuseAgentName(name); if (refusal) - throw new ApiError(400, "POST /api/agents/scaffold \u2192 400 (mock)", refusal); + throw new ApiError( + 400, + "POST /api/agents/scaffold \u2192 400 (mock)", + refusal, + ); // THE ROOT BARRIER, and the reason it is here: the real route only writes // into a folder the rail can show, and this mock originally skipped that // guard — so `templates.spec.ts` asserted a scaffold into diff --git a/packages/harness/web/src/lib/mock-data.ts b/packages/harness/web/src/lib/mock-data.ts index 9d2e0f27f..05b938a45 100644 --- a/packages/harness/web/src/lib/mock-data.ts +++ b/packages/harness/web/src/lib/mock-data.ts @@ -3,18 +3,32 @@ * running harness server (see MockApi in ./api). */ import type { CanvasOverviewContent } from "../components/CanvasOverviewPanel"; -import type { AccountPlanView, HarnessEntry, HarnessSession, HarnessSettings, MacroDef, SessionRecord, SessionSummary, TemplateDetailView, TemplateSummary, WorkflowInfo } from "@shared/types"; +import type { + AccountPlanView, + HarnessEntry, + HarnessSession, + HarnessSettings, + MacroDef, + SessionRecord, + SessionSummary, + TemplateDetailView, + TemplateSummary, + WorkflowInfo, +} from "@shared/types"; const now = Date.now(); -const minutesAgo = (n: number): string => new Date(now - n * 60_000).toISOString(); -const daysAgo = (n: number): string => new Date(now - n * 24 * 60 * 60_000).toISOString(); +const minutesAgo = (n: number): string => + new Date(now - n * 60_000).toISOString(); +const daysAgo = (n: number): string => + new Date(now - n * 24 * 60 * 60_000).toISOString(); /** The directory the harness itself was launched from (`npx @sapiom/harness [dir]`). */ /** Demo-only canvas overview content (the real renderer emits this inside * its own document; live mode therefore renders no app-side panel). */ export const MOCK_CANVAS_OVERVIEWS: Record = { "/Users/demo/acme-app/leasing": { - description: "Handles lease applications end to end: screening, credit check, and approval routing.", + description: + "Handles lease applications end to end: screening, credit check, and approval routing.", // Counting rule shared with the Steps tab (canvas-graph's graphCounts): // pipeline steps exclude the two terminal exits, counted separately. stats: "4 steps · 2 exits · intake entry", @@ -90,7 +104,8 @@ export const MOCK_TEMPLATES: TemplateSummary[] = [ { id: "hello-agent", name: "Hello Agent", - description: "The minimal single-step agent: a smoke test for the build, deploy, run path.", + description: + "The minimal single-step agent: a smoke test for the build, deploy, run path.", tags: ["starter", "minimal"], category: "starter", cadence: "on-demand", @@ -113,7 +128,8 @@ export const MOCK_TEMPLATES: TemplateSummary[] = [ { id: "web-research-digest", name: "Web Research Digest", - description: "Search the web for a topic and return a concise, sourced digest.", + description: + "Search the web for a topic and return a concise, sourced digest.", tags: ["research", "search"], category: "data-knowledge", cadence: "on-demand", @@ -241,13 +257,25 @@ export const MOCK_TEMPLATE_GRAPHS: Record< > = { "hello-agent": { steps: [ - { name: "greet", description: "Validate the input and return a greeting.", capabilities: [], kind: "entry", sublabel: "entry" }, + { + name: "greet", + description: "Validate the input and return a greeting.", + capabilities: [], + kind: "entry", + sublabel: "entry", + }, ], transitions: [], }, "web-research-digest": { steps: [ - { name: "search", description: "Query the web for the topic.", capabilities: ["web.search"], kind: "entry", sublabel: "entry" }, + { + name: "search", + description: "Query the web for the topic.", + capabilities: ["web.search"], + kind: "entry", + sublabel: "entry", + }, { name: "summarize", description: "Condense the results into a sourced digest.", @@ -256,16 +284,48 @@ export const MOCK_TEMPLATE_GRAPHS: Record< sublabel: "terminal · success", }, ], - transitions: [{ from: "search", to: "summarize", label: null, kind: "continue" }], + transitions: [ + { from: "search", to: "summarize", label: null, kind: "continue" }, + ], }, "dependency-upgrade": { steps: [ - { name: "scan", description: "List outdated dependencies.", capabilities: ["sandbox.run"], kind: "entry", sublabel: "entry" }, - { name: "bump", description: "Apply the upgrades in a sandbox.", capabilities: [], kind: "step", sublabel: "step" }, - { name: "test", description: "Run the suite against the bumped tree.", capabilities: [], kind: "step", sublabel: "step · can also terminate" }, - { name: "open_pr", description: "Open a PR with the passing upgrade.", capabilities: [], kind: "terminal-success", sublabel: "terminal · success" }, + { + name: "scan", + description: "List outdated dependencies.", + capabilities: ["sandbox.run"], + kind: "entry", + sublabel: "entry", + }, + { + name: "bump", + description: "Apply the upgrades in a sandbox.", + capabilities: [], + kind: "step", + sublabel: "step", + }, + { + name: "test", + description: "Run the suite against the bumped tree.", + capabilities: [], + kind: "step", + sublabel: "step · can also terminate", + }, + { + name: "open_pr", + description: "Open a PR with the passing upgrade.", + capabilities: [], + kind: "terminal-success", + sublabel: "terminal · success", + }, // A fail-only sink: amber "needs attention", NOT a green success exit. - { name: "give_up", description: "Tests still failing after retries.", capabilities: [], kind: "terminal-warn", sublabel: "terminal · needs attention" }, + { + name: "give_up", + description: "Tests still failing after retries.", + capabilities: [], + kind: "terminal-warn", + sublabel: "terminal · needs attention", + }, ], transitions: [ { from: "scan", to: "bump", label: null, kind: "continue" }, @@ -276,25 +336,78 @@ export const MOCK_TEMPLATE_GRAPHS: Record< }, "approval-chain": { steps: [ - { name: "start", description: "Record the request.", capabilities: ["database.create"], kind: "entry", sublabel: "entry" }, - { name: "present", description: "Email the current gate's approver.", capabilities: ["email.send"], kind: "step", sublabel: "step" }, + { + name: "start", + description: "Record the request.", + capabilities: ["database.create"], + kind: "entry", + sublabel: "entry", + }, + { + name: "present", + description: "Email the current gate's approver.", + capabilities: ["email.send"], + kind: "step", + sublabel: "step", + }, // A pause step shows the signal it waits for. - { name: "decide", description: "Wait for the approver's answer.", capabilities: [], kind: "pause", sublabel: "pause · approval.decided" }, - { name: "finalize", description: "All gates passed.", capabilities: ["email.send"], kind: "terminal-success", sublabel: "terminal · success" }, - { name: "compensate", description: "Roll back on rejection.", capabilities: ["email.send"], kind: "terminal-warn", sublabel: "terminal · needs attention" }, + { + name: "decide", + description: "Wait for the approver's answer.", + capabilities: [], + kind: "pause", + sublabel: "pause · approval.decided", + }, + { + name: "finalize", + description: "All gates passed.", + capabilities: ["email.send"], + kind: "terminal-success", + sublabel: "terminal · success", + }, + { + name: "compensate", + description: "Roll back on rejection.", + capabilities: ["email.send"], + kind: "terminal-warn", + sublabel: "terminal · needs attention", + }, ], transitions: [ { from: "start", to: "present", label: null, kind: "continue" }, { from: "present", to: "decide", label: null, kind: "continue" }, - { from: "decide", to: "finalize", label: "approval.decided", kind: "pause" }, + { + from: "decide", + to: "finalize", + label: "approval.decided", + kind: "pause", + }, { from: "decide", to: "compensate", label: null, kind: "continue" }, ], }, "cold-outreach-engine": { steps: [ - { name: "enrich", description: "Enrich the lead list.", capabilities: ["web.search"], kind: "entry", sublabel: "entry" }, - { name: "personalize", description: "Write a first line per prospect.", capabilities: [], kind: "step", sublabel: "step" }, - { name: "send", description: "Drip the sends.", capabilities: ["email.send"], kind: "terminal-success", sublabel: "terminal · success" }, + { + name: "enrich", + description: "Enrich the lead list.", + capabilities: ["web.search"], + kind: "entry", + sublabel: "entry", + }, + { + name: "personalize", + description: "Write a first line per prospect.", + capabilities: [], + kind: "step", + sublabel: "step", + }, + { + name: "send", + description: "Drip the sends.", + capabilities: ["email.send"], + kind: "terminal-success", + sublabel: "terminal · success", + }, ], transitions: [ { from: "enrich", to: "personalize", label: null, kind: "continue" }, @@ -306,10 +419,34 @@ export const MOCK_TEMPLATE_GRAPHS: Record< // registry declares, with the one model step the Moderate band turns on. "scheduled-research-brief": { steps: [ - { name: "search", description: "Gather sources on the topic.", capabilities: ["web.search"], kind: "entry", sublabel: "entry" }, - { name: "summarize", description: "Draft the brief from the sources.", capabilities: ["models.run"], kind: "step", sublabel: "step" }, - { name: "review", description: "Check the brief covers the ask.", capabilities: [], kind: "step", sublabel: "step" }, - { name: "deliver", description: "Send the finished brief.", capabilities: [], kind: "terminal-success", sublabel: "terminal · success" }, + { + name: "search", + description: "Gather sources on the topic.", + capabilities: ["web.search"], + kind: "entry", + sublabel: "entry", + }, + { + name: "summarize", + description: "Draft the brief from the sources.", + capabilities: ["models.run"], + kind: "step", + sublabel: "step", + }, + { + name: "review", + description: "Check the brief covers the ask.", + capabilities: [], + kind: "step", + sublabel: "step", + }, + { + name: "deliver", + description: "Send the finished brief.", + capabilities: [], + kind: "terminal-success", + sublabel: "terminal · success", + }, ], transitions: [ { from: "search", to: "summarize", label: null, kind: "continue" }, @@ -464,7 +601,9 @@ const FLOOD_WORKTREES = [ */ export function isFloodRailFixture(): boolean { if (typeof window === "undefined") return false; - return new URLSearchParams(window.location.search).get("mockFixtures") === "flood"; + return ( + new URLSearchParams(window.location.search).get("mockFixtures") === "flood" + ); } /** @@ -481,7 +620,9 @@ const floodFsTree = (): Record => const segments = tree.split("/"); const base = `/Users/demo/${tree}`; return [ - ...(segments.length > 1 ? [[`/Users/demo/${segments[0]}`, [segments[1]!]] as const] : []), + ...(segments.length > 1 + ? [[`/Users/demo/${segments[0]}`, [segments[1]!]] as const] + : []), [base, ["ari"]] as const, [`${base}/ari`, ["orchestration", "brain"]] as const, [`${base}/ari/orchestration`, []] as const, @@ -490,7 +631,10 @@ const floodFsTree = (): Record => }), ["/Users/demo/team-tools", ["slack-notifier"]] as const, ["/Users/demo/other-tools", ["slack-notifier"]] as const, - ["/Users/demo/misc", Array.from({ length: 10 }, (_, i) => `pkg${i + 1}`)] as const, + [ + "/Users/demo/misc", + Array.from({ length: 10 }, (_, i) => `pkg${i + 1}`), + ] as const, ...Array.from( { length: 10 }, (_, i) => [`/Users/demo/misc/pkg${i + 1}`, ["agent"]] as const, @@ -507,7 +651,16 @@ export const MOCK_FS_TREE: Record = { "scratch", "blank-slate", ...(isFloodRailFixture() - ? ["design-eng", "design-eng-fix", "design-eng-ij", "design-eng-main", "worktrees", "misc", "team-tools", "other-tools"] + ? [ + "design-eng", + "design-eng-fix", + "design-eng-ij", + "design-eng-main", + "worktrees", + "misc", + "team-tools", + "other-tools", + ] : []), ], ...(isFloodRailFixture() ? floodFsTree() : {}), @@ -676,11 +829,16 @@ export const MOCK_SESSION_RECORDS: Record = { reconstructed: true, // Folded live from events.ndjson — nothing archived about it yet. archivedAt: null, - limitations: ["truncated-tool-output", "assistant-narration-gap", "incomplete-final-turn"], + limitations: [ + "truncated-tool-output", + "assistant-narration-gap", + "incomplete-final-turn", + ], turns: [ { index: 1, - prompt: "Add the screening step to the leasing agent and wire it to the credit check.", + prompt: + "Add the screening step to the leasing agent and wire it to the credit check.", promptAt: minutesAgo(48), toolCalls: [ { @@ -692,8 +850,10 @@ export const MOCK_SESSION_RECORDS: Record = { }, { name: "Edit", - input: '{"file_path":"/Users/demo/acme-app/leasing/index.ts","old_string":"steps: [apply]","new_string":"steps: [apply, screening]"}', - responseSummary: "Applied 1 edit to /Users/demo/acme-app/leasing/index.ts\n…[truncated 2048 chars]", + input: + '{"file_path":"/Users/demo/acme-app/leasing/index.ts","old_string":"steps: [apply]","new_string":"steps: [apply, screening]"}', + responseSummary: + "Applied 1 edit to /Users/demo/acme-app/leasing/index.ts\n…[truncated 2048 chars]", responseTruncated: true, at: minutesAgo(46), }, @@ -753,7 +913,8 @@ export const MOCK_SESSION_RECORDS: Record = { at: daysAgo(1), }, ], - assistantText: "Wired it through `applicantQueue.publish()` and added the retry policy.", + assistantText: + "Wired it through `applicantQueue.publish()` and added the retry policy.", model: "claude-opus-4-6", usage: { inputTokens: 9120, outputTokens: 340 }, completedAt: daysAgo(1), @@ -764,7 +925,8 @@ export const MOCK_SESSION_RECORDS: Record = { prompt: "Add a test for the retry path.", promptAt: daysAgo(1), toolCalls: [], - assistantText: "Added `screening.retry.test.ts` covering the 5xx-then-success path.", + assistantText: + "Added `screening.retry.test.ts` covering the 5xx-then-success path.", model: "claude-opus-4-6", usage: { inputTokens: 10240, outputTokens: 210 }, completedAt: daysAgo(1), @@ -813,7 +975,8 @@ export const MOCK_SESSION_RECORDS: Record = { at: daysAgo(3), }, ], - assistantText: "Mid tier is metered now; the annual discount still needs deciding.", + assistantText: + "Mid tier is metered now; the annual discount still needs deciding.", model: "claude-opus-4-6", usage: { inputTokens: 8210, outputTokens: 280 }, completedAt: daysAgo(3), @@ -826,7 +989,8 @@ export const MOCK_SESSION_RECORDS: Record = { toolCalls: [ { name: "Write", - input: '{"file_path":"/Users/demo/acme-app/migrations/0042-pricing.sql"}', + input: + '{"file_path":"/Users/demo/acme-app/migrations/0042-pricing.sql"}', responseSummary: "wrote 34 lines", responseTruncated: false, at: daysAgo(3), @@ -864,7 +1028,8 @@ export const MOCK_SESSION_RECORDS: Record = { { name: "shell", input: '{"command":["cat","README.md"]}', - responseSummary: "# rfq-agent\nRequest-for-quote intake and routing.", + responseSummary: + "# rfq-agent\nRequest-for-quote intake and routing.", responseTruncated: false, at: daysAgo(1), }, @@ -897,7 +1062,11 @@ export const MOCK_SESSION_RECORDS: Record = { eventCount: 61, reconstructed: true, archivedAt: daysAgo(45), - limitations: ["truncated-tool-output", "compacted-archive", "dropped-early-turns"], + limitations: [ + "truncated-tool-output", + "compacted-archive", + "dropped-early-turns", + ], turns: [ { index: 8, @@ -912,7 +1081,8 @@ export const MOCK_SESSION_RECORDS: Record = { at: daysAgo(45), }, ], - assistantText: "Backfilled 41,203 applicant rows; 12 failed validation and are listed in `backfill-errors.json`.", + assistantText: + "Backfilled 41,203 applicant rows; 12 failed validation and are listed in `backfill-errors.json`.", model: "claude-opus-4-6", usage: { inputTokens: 21400, outputTokens: 480 }, completedAt: daysAgo(45), @@ -923,7 +1093,8 @@ export const MOCK_SESSION_RECORDS: Record = { prompt: "Ship the migration and note the 12 failures in the changelog.", promptAt: daysAgo(45), toolCalls: [], - assistantText: "Shipped, with the 12 unmigrated applicants called out under Known issues.", + assistantText: + "Shipped, with the 12 unmigrated applicants called out under Known issues.", model: "claude-opus-4-6", usage: { inputTokens: 22100, outputTokens: 130 }, completedAt: daysAgo(45), @@ -991,18 +1162,30 @@ const deepAgent = ( export const MOCK_DEEP_WORKFLOWS: WorkflowInfo[] = [ deepAgent(`${DEEP_ROOT}/backend/src/agents/ads`, "ads"), - deepAgent(`${DEEP_ROOT}/backend/src/agents/outreach`, "outreach"), + // Markerless/unlinked source agent: the mock API keeps its syntax proof in a + // private accepted sidecar, so the public row retains null cloud metadata + // while the graph still resolves the canonical `outreach` key and edges. + deepAgent(`${DEEP_ROOT}/backend/src/agents/outreach`, "outreach", { + definitionSlug: null, + }), // No slug or deployment identity: this is the realistic project-relative // fallback case the graph must still navigate back to the agent row. deepAgent(`${DEEP_ROOT}/scripts/tools/rollup`, "rollup", { definitionSlug: null, }), - deepAgent(`${DEEP_ROOT}/packages/harness/web/src/components/mailer`, "mailer", { - definitionId: 7701, - activeBuildRunId: "build-mailer-ready", - activeBuildRunStatus: "ready", - }), - deepAgent(`${DEEP_ROOT}/packages/harness/web/src/components/sender`, "sender"), + deepAgent( + `${DEEP_ROOT}/packages/harness/web/src/components/mailer`, + "mailer", + { + definitionId: 7701, + activeBuildRunId: "build-mailer-ready", + activeBuildRunStatus: "ready", + }, + ), + deepAgent( + `${DEEP_ROOT}/packages/harness/web/src/components/sender`, + "sender", + ), deepAgent(`${DEEP_ROOT}/services/gateway`, "gateway"), deepAgent(`${DEEP_ROOT}/services/workers/ads`, "ads-worker"), deepAgent(`${DEEP_ROOT}/services/workers/queue`, "queue"), @@ -1050,8 +1233,14 @@ export const MOCK_FLOOD_WORKFLOWS: WorkflowInfo[] = [ deepAgent(`/Users/demo/${tree}/ari/orchestration`, "ari-grade-repo"), deepAgent(`/Users/demo/${tree}/ari/brain`, "brain-agent"), ]), - deepAgent("/Users/demo/team-tools/slack-notifier", "@sapiom/example-slack-notifier"), - deepAgent("/Users/demo/other-tools/slack-notifier", "@sapiom/example-slack-notifier"), + deepAgent( + "/Users/demo/team-tools/slack-notifier", + "@sapiom/example-slack-notifier", + ), + deepAgent( + "/Users/demo/other-tools/slack-notifier", + "@sapiom/example-slack-notifier", + ), ...Array.from({ length: 10 }, (_, i) => deepAgent(`/Users/demo/misc/pkg${i + 1}/agent`, `filler-${i + 1}`), ), @@ -1070,7 +1259,9 @@ export const MOCK_DEEP_ROOTS: string[] = [ */ export function isDeepRailFixture(): boolean { if (typeof window === "undefined") return false; - return new URLSearchParams(window.location.search).get("mockFixtures") === "deep"; + return ( + new URLSearchParams(window.location.search).get("mockFixtures") === "deep" + ); } export const MOCK_WORKFLOWS: WorkflowInfo[] = [ @@ -1169,7 +1360,8 @@ export const MOCK_SEARCH_HISTORY: Record = { agentSessionId: "search-annotate", harness: "claude-code" as const, cwd: "/Users/demo/acme-app", - title: "You are annotating an already-generated draft of the leasing docs", + title: + "You are annotating an already-generated draft of the leasing docs", lastActiveAt: daysAgo(3), source: "transcript" as const, resumeMode: "rehydrate" as const, @@ -1206,21 +1398,33 @@ export const MOCK_MACROS: MacroDef[] = [ id: "run_local", label: "Run local", icon: "Play", - action: { kind: "inject", text: "cd {{workflow.path}} && sapiom agents run --target local", submit: true }, + action: { + kind: "inject", + text: "cd {{workflow.path}} && sapiom agents run --target local", + submit: true, + }, requiresWorkflow: true, }, { id: "deploy", label: "Deploy", icon: "Cloud", - action: { kind: "inject", text: "cd {{workflow.path}} && sapiom agents deploy", submit: true }, + action: { + kind: "inject", + text: "cd {{workflow.path}} && sapiom agents deploy", + submit: true, + }, requiresWorkflow: true, }, { id: "prod_run", label: "Prod run", icon: "Zap", - action: { kind: "inject", text: "cd {{workflow.path}} && sapiom agents run --target prod", submit: true }, + action: { + kind: "inject", + text: "cd {{workflow.path}} && sapiom agents run --target prod", + submit: true, + }, requiresWorkflow: true, }, { diff --git a/packages/harness/web/src/lib/system-graph-layout.test.ts b/packages/harness/web/src/lib/system-graph-layout.test.ts index c3ff8b5b8..181e1a08b 100644 --- a/packages/harness/web/src/lib/system-graph-layout.test.ts +++ b/packages/harness/web/src/lib/system-graph-layout.test.ts @@ -19,7 +19,13 @@ const edge = ( from: string, to: string, mode: AgentInvocationMode = "blocking", -): SystemGraphEdge => ({ from, to, kind: "invokes", basis: "static", mode }); +): SystemGraphEdge => ({ + from, + to, + kind: "invokes", + basis: "static-invocation", + mode, +}); function graph(nodeIds: string[], edges: SystemGraphEdge[]): SystemGraph { return { @@ -547,7 +553,12 @@ describe("layoutSystemGraph with groups", () => { const layout = layoutSystemGraph( graph( ["a", "b", "c", "solo"], - [edge("a", "b"), edge("b", "c"), edge("c", "a", "async"), edge("a", "c")], + [ + edge("a", "b"), + edge("b", "c"), + edge("c", "a", "async"), + edge("a", "c"), + ], ), [ group("g:cyclic", "Cyclic", ["a", "b", "c"]), @@ -571,10 +582,10 @@ describe("layoutSystemGraph with groups", () => { // A group is editable, so half a detected system can be pulled out. The // edge between the halves is still real; dropping it would make the map // claim two systems never touch. - const layout = layoutSystemGraph( - graph(["a", "b"], [edge("a", "b")]), - [group("g:one", "One", ["a"]), group("g:two", "Two", ["b"])], - ); + const layout = layoutSystemGraph(graph(["a", "b"], [edge("a", "b")]), [ + group("g:one", "One", ["a"]), + group("g:two", "Two", ["b"]), + ]); expect(layout.edges).toHaveLength(1); expect(layout.edges[0]).toMatchObject({ from: "a", @@ -611,9 +622,16 @@ describe("layoutSystemGraph with groups", () => { "Ungrouped", and matching on the label would drop the cards nothing claimed inside it and move it to the end of the map. */ const layout = layoutSystemGraph(graph(["a", "b", "orphan"], []), [ - { id: "g:named", label: "Ungrouped", nodeIds: ["a", "b"], isUngrouped: false }, + { + id: "g:named", + label: "Ungrouped", + nodeIds: ["a", "b"], + isUngrouped: false, + }, + ]); + expect(layout.groups.map((candidate) => candidate.nodeCount)).toEqual([ + 2, 1, ]); - expect(layout.groups.map((candidate) => candidate.nodeCount)).toEqual([2, 1]); expect(layout.groups[0]!.id).toBe("g:named"); expect(byId(layout, "a").groupId).toBe("g:named"); expect(byId(layout, "b").groupId).toBe("g:named"); diff --git a/packages/harness/web/src/lib/system-graph.test.ts b/packages/harness/web/src/lib/system-graph.test.ts index c4a14d92e..58e65889b 100644 --- a/packages/harness/web/src/lib/system-graph.test.ts +++ b/packages/harness/web/src/lib/system-graph.test.ts @@ -20,7 +20,7 @@ const valid: SystemGraph = { from: "agent:research", to: "agent:growth", kind: "invokes", - basis: "static", + basis: "static-invocation", mode: "async", }, ], @@ -32,6 +32,15 @@ describe("parseSystemGraph", () => { expect(parseSystemGraph(valid)).toEqual(valid); }); + it("rejects the obsolete static invocation basis spelling", () => { + expect(() => + parseSystemGraph({ + ...valid, + edges: [{ ...valid.edges[0], basis: "static" }], + }), + ).toThrow("Invalid system graph response"); + }); + it("accepts scoped package display labels", () => { const graph = { ...valid, diff --git a/packages/harness/web/src/lib/system-graph.ts b/packages/harness/web/src/lib/system-graph.ts index 3b5280a63..6d120f004 100644 --- a/packages/harness/web/src/lib/system-graph.ts +++ b/packages/harness/web/src/lib/system-graph.ts @@ -124,7 +124,7 @@ function parseEdge(value: unknown): SystemGraphEdge | null { typeof value.from !== "string" || typeof value.to !== "string" || value.kind !== "invokes" || - value.basis !== "static" || + value.basis !== "static-invocation" || (value.mode !== "blocking" && value.mode !== "async") ) { return null; @@ -133,7 +133,7 @@ function parseEdge(value: unknown): SystemGraphEdge | null { from: value.from, to: value.to, kind: "invokes", - basis: "static", + basis: "static-invocation", mode: value.mode, }; } @@ -185,7 +185,7 @@ const compareIds = (left: string, right: string): number => left === right ? 0 : left < right ? -1 : 1; /** Public graph data retains one record per mode. The V0 Canvas draws one - * connector per endpoint pair so dual-mode relationships never overlap. */ + * connector per endpoint pair so dual-mode invocations never overlap. */ export function groupSystemGraphEdges( edges: readonly SystemGraphEdge[], ): VisibleSystemGraphEdge[] { diff --git a/packages/harness/web/src/lib/use-harness-state.ts b/packages/harness/web/src/lib/use-harness-state.ts index 9e1db1764..f2a603e40 100644 --- a/packages/harness/web/src/lib/use-harness-state.ts +++ b/packages/harness/web/src/lib/use-harness-state.ts @@ -52,6 +52,7 @@ import { mergeHistory } from "./history-meta"; import { createToastMessage, type ToastMessage, type ToastTone } from "./toast"; import { subscribeEvents } from "./events"; import { systemGraphLoader } from "./system-graph-loader"; +import { WorkflowProjectionOrder } from "./workflow-projection-order"; import { retainSystemGraphAnnouncements, systemGraphAnnouncementsAfterMessage, @@ -451,6 +452,12 @@ export function useHarnessState(): HarnessStateHook { */ const workflowsRef = useRef([]); workflowsRef.current = state?.workflows ?? []; + // One ordering domain for every workflow-bearing response, including the + // boot AppState fetch. Without it, an older boot/list response can resolve + // after `workflows.changed` and resurrect a removed/rekeyed rail row. + const workflowProjectionOrder = useRef( + new WorkflowProjectionOrder(), + ).current; const [loading, setLoading] = useState(true); const [error, setError] = useState(null); // Boot-error facts (HTTP status / network-throw flag), shaped for the @@ -1047,6 +1054,7 @@ export function useHarnessState(): HarnessStateHook { useEffect(() => { let cancelled = false; + const workflowRequest = workflowProjectionOrder.begin(); // A retry re-enters the loading state and clears the prior failure so the // shell shows "reconnecting", not a stale error, while the refetch runs. if (reloadSeq > 0) { @@ -1057,12 +1065,22 @@ export function useHarnessState(): HarnessStateHook { Promise.all([api.getState(), api.getSettings()]) .then(([appState, harnessSettings]) => { if (cancelled) return; - setState(appState); + const bootWorkflowsAccepted = workflowProjectionOrder.accept( + workflowRequest, + appState.workflows, + ); + // Always adopt the boot response's non-workflow fields. Its workflow + // projection participates in the same monotonic ordering as event and + // explicit refreshes; if stale, retain the newer accepted projection + // (or an empty placeholder while that newer request is still pending). + const workflows = bootWorkflowsAccepted + ? appState.workflows + : [...(workflowProjectionOrder.current() ?? [])]; + setState({ ...appState, workflows }); // Baseline the built-agents metric: everything present at load already // existed, so seed it into the seen-set and never count it as built. const seenAtLoad = (seenAgentPathsRef.current ??= new Set()); - for (const workflow of appState.workflows) - seenAtLoad.add(workflow.path); + for (const workflow of workflows) seenAtLoad.add(workflow.path); setSettings(harnessSettings); setErrorKind(null); if (appState.tasks) setTasks(appState.tasks); @@ -1070,8 +1088,7 @@ export function useHarnessState(): HarnessStateHook { (session) => session.status !== "exited", ); if (running) setActiveSessionId(running.id); - if (appState.workflows[0]) - setSelectedWorkflowPath(appState.workflows[0].path); + if (workflows[0]) setSelectedWorkflowPath(workflows[0].path); }) .catch((err: unknown) => { if (cancelled) return; @@ -1090,7 +1107,7 @@ export function useHarnessState(): HarnessStateHook { return () => { cancelled = true; }; - }, [reloadSeq]); + }, [reloadSeq, workflowProjectionOrder]); // Recovery path for a failed boot: re-run the one-shot fetch. Bumping the // seq re-fires the boot effect above (which resets loading/error itself). @@ -1107,10 +1124,17 @@ export function useHarnessState(): HarnessStateHook { }, [activeSessionId]); const refreshWorkflows = useCallback(async () => { + const request = workflowProjectionOrder.begin(); const workflows = await api.listWorkflows(); - setState((prev) => (prev ? { ...prev, workflows } : prev)); - return workflows; - }, []); + if (workflowProjectionOrder.accept(request, workflows)) { + setState((prev) => (prev ? { ...prev, workflows } : prev)); + return workflows; + } + // A stale caller still receives the current accepted projection. This + // matters for analytics/import callers: processing the stale HTTP payload + // could baseline or emit rows that the UI correctly refused to render. + return [...(workflowProjectionOrder.current() ?? workflowsRef.current)]; + }, [workflowProjectionOrder]); useEffect(() => { return subscribeEvents((message) => { @@ -1152,21 +1176,32 @@ export function useHarnessState(): HarnessStateHook { // The workspace watcher saw a sapiom.json appear/change — the one // client signal for an agent built in-app. Emit agent.created for any // path we haven't already baselined (load) or imported (scan/connect). - void refreshWorkflows().then((workflows) => { - const seen = seenAgentPathsRef.current; - if (seen === null) { - // Lost the race with the initial load — baseline, don't emit. - seenAgentPathsRef.current = new Set(workflows.map((w) => w.path)); - return; - } - for (const path of newAgentPaths(seen, workflows)) { - seen.add(path); - trackProduct("agent.created", { - workflow_slug: slugFromPath(path), - ...agentProvenance(workflows.find((w) => w.path === path)), - }); - } - }); + // Capture this at message receipt, not response settlement: boot can + // resolve while this list is in flight. Anything announced before boot + // established its baseline belongs to that baseline regardless of HTTP + // completion order. + const baselineOnly = seenAgentPathsRef.current === null; + void refreshWorkflows() + .then((workflows) => { + const seen = seenAgentPathsRef.current; + if (baselineOnly || seen === null) { + // Lost the race with the initial load — baseline, don't emit. + const baseline = (seenAgentPathsRef.current ??= new Set()); + for (const workflow of workflows) baseline.add(workflow.path); + return; + } + for (const path of newAgentPaths(seen, workflows)) { + seen.add(path); + trackProduct("agent.created", { + workflow_slug: slugFromPath(path), + ...agentProvenance(workflows.find((w) => w.path === path)), + }); + } + }) + // A bus refresh is best-effort. Keep the last successful projection + // and let the next event/auth/manual refresh retry; never create an + // unhandled rejection from the event callback. + .catch(() => undefined); } else if (message.type === "system-graph.changed") { // Invalidate even while its workspace destination is closed. The next // open must never resurrect a pre-edit process-lifetime promise. @@ -1229,7 +1264,7 @@ export function useHarnessState(): HarnessStateHook { // Definition build evidence is authenticated enrichment. Re-list on // both sign-in and sign-out so a post-boot login can enable a ready // agent and a logout cannot leave tenant metadata pinned in memory. - void refreshWorkflows(); + void refreshWorkflows().catch(() => undefined); } }); }, [refreshWorkflows, startRunPolling]); @@ -1578,17 +1613,10 @@ export function useHarnessState(): HarnessStateHook { const connectWorkflow = useCallback( async (path: string): Promise => { const workflow = await api.connectWorkflow(path); - setState((prev) => - prev - ? { - ...prev, - workflows: [ - ...prev.workflows.filter((w) => w.path !== workflow.path), - workflow, - ], - } - : prev, - ); + // The mutation response is not an inventory snapshot. Re-list through + // the shared ordering domain so it cannot resurrect a row after a newer + // deletion event, and an older in-flight list cannot erase this import. + await refreshWorkflows(); // Connecting an existing agent is an import, not a build — baseline it so // it is never counted as newly built. (seenAgentPathsRef.current ??= new Set()).add(workflow.path); @@ -1635,7 +1663,7 @@ export function useHarnessState(): HarnessStateHook { } return workflow; }, - [rememberProjectDir, reopenProjects], + [refreshWorkflows, rememberProjectDir, reopenProjects], ); const scaffoldAgent = useCallback( @@ -2012,7 +2040,7 @@ export function useHarnessState(): HarnessStateHook { ); // The link is already durable at this point. Pull its mutable // build projection so the chip can say Building, not Deployed. - void refreshWorkflows(); + void refreshWorkflows().catch(() => undefined); } }); if (terminal.phase === "ready") { diff --git a/packages/harness/web/src/lib/workflow-projection-order.test.ts b/packages/harness/web/src/lib/workflow-projection-order.test.ts new file mode 100644 index 000000000..4bc6ef212 --- /dev/null +++ b/packages/harness/web/src/lib/workflow-projection-order.test.ts @@ -0,0 +1,69 @@ +import { describe, expect, it } from "vitest"; + +import { WorkflowProjectionOrder } from "./workflow-projection-order"; + +describe("WorkflowProjectionOrder", () => { + it("rejects an older boot response after a newer event refresh commits", () => { + const order = new WorkflowProjectionOrder(); + const boot = order.begin(); + const event = order.begin(); + + expect(order.accept(event, ["new"])).toBe(true); + expect(order.accept(boot, ["old"])).toBe(false); + expect(order.current()).toEqual(["new"]); + }); + + it("keeps an older success as fallback while the newest request is pending", () => { + const order = new WorkflowProjectionOrder(); + const baseline = order.begin(); + expect(order.accept(baseline, ["baseline"])).toBe(true); + + const older = order.begin(); + const newest = order.begin(); + expect(order.accept(older, ["fallback"])).toBe(true); + expect(order.current()).toEqual(["fallback"]); + + expect(order.accept(newest, ["latest"])).toBe(true); + expect(order.current()).toEqual(["latest"]); + }); + + it("lets a later boot retry supersede an earlier successful retry", () => { + const order = new WorkflowProjectionOrder(); + const first = order.begin(); + const retry = order.begin(); + + expect(order.accept(first, ["first"])).toBe(true); + expect(order.current()).toEqual(["first"]); + expect(order.accept(retry, ["retry"])).toBe(true); + expect(order.current()).toEqual(["retry"]); + }); + + it("retains a boot success when a newer request fails without committing", () => { + const order = new WorkflowProjectionOrder(); + const boot = order.begin(); + order.begin(); // newer event request; its rejected promise never calls accept + + expect(order.accept(boot, ["boot"])).toBe(true); + expect(order.current()).toEqual(["boot"]); + }); + + it("does not let a connect re-list resurrect a newer event deletion", () => { + const order = new WorkflowProjectionOrder(); + const connectList = order.begin(); + const deletionEventList = order.begin(); + + expect(order.accept(deletionEventList, [])).toBe(true); + expect(order.accept(connectList, ["connected"])).toBe(false); + expect(order.current()).toEqual([]); + }); + + it("does not let an older event list erase a newer connect re-list", () => { + const order = new WorkflowProjectionOrder(); + const eventList = order.begin(); + const connectList = order.begin(); + + expect(order.accept(connectList, ["connected"])).toBe(true); + expect(order.accept(eventList, [])).toBe(false); + expect(order.current()).toEqual(["connected"]); + }); +}); diff --git a/packages/harness/web/src/lib/workflow-projection-order.ts b/packages/harness/web/src/lib/workflow-projection-order.ts new file mode 100644 index 000000000..352357582 --- /dev/null +++ b/packages/harness/web/src/lib/workflow-projection-order.ts @@ -0,0 +1,34 @@ +/** + * Orders every HTTP response that can replace the browser's workflow + * projection. Boot, bus-driven refreshes, auth refreshes, scans, connects, and + * deploy refreshes all share this clock, so an older response can never put a + * row back after a newer successful request observed its removal. + * + * A merely-started request does not invalidate an older success. That detail + * lets boot remain a safe fallback when a newer bus/auth refresh fails: the + * successful response with the greatest request id wins, independent of + * completion order. + */ +export class WorkflowProjectionOrder { + private issued = 0; + private acceptedRequest = 0; + private accepted: readonly T[] | null = null; + + begin(): number { + this.issued += 1; + return this.issued; + } + + /** Accepts `rows` unless a newer request already committed successfully. */ + accept(request: number, rows: readonly T[]): boolean { + if (request < this.acceptedRequest) return false; + this.acceptedRequest = request; + this.accepted = rows; + return true; + } + + /** The last monotonic projection, if any request has committed one. */ + current(): readonly T[] | null { + return this.accepted; + } +} diff --git a/scripts/agent-studio-terminology-allowlist.json b/scripts/agent-studio-terminology-allowlist.json index e295b2d1c..55ffe7b73 100644 --- a/scripts/agent-studio-terminology-allowlist.json +++ b/scripts/agent-studio-terminology-allowlist.json @@ -192,8 +192,8 @@ "id": "server-workflow-change-event", "path": "packages/harness/src/server/index.ts", "pattern": "^workflows\\.changed$", - "occurrences": 3, - "reason": "The internal WebSocket event name remains stable for existing clients, including explicit agent connections that refresh Project graphs." + "occurrences": 1, + "reason": "The internal WebSocket event name remains stable for existing clients; accepted agent discovery publishes it from the coordinator." }, { "id": "rest-session-binding-route", @@ -514,7 +514,7 @@ "id": "web-api-workflow-change-event", "path": "packages/harness/web/src/lib/api.ts", "pattern": "^workflows\\.changed$", - "occurrences": 2, - "reason": "The internal WebSocket event name remains stable for existing clients. Announced by the mock's mover and its scaffold, the two mutations that change what the rail shows." + "occurrences": 3, + "reason": "The internal WebSocket event name remains stable for existing clients. The mock announces source discovery, moves, and scaffolds through this signal." } ]