diff --git a/CHANGELOG.md b/CHANGELOG.md index 3ae67129..f5616d99 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,30 @@ # Changelog +## 0.128.0 + +### chat-transport executor: workers on a bare chat-completions transport + +A first-class `Executor` whose runtime is a plain OpenAI-compatible `/v1/chat/completions` transport — the worker IS a model conversation, not a sandboxed process (#721). +What every offline test faked through `AgentSpec.executor` is now a shipped, supported leaf: node pinning, conserved spend, settle/verdict, and the journal + edge ledger all apply to a chat worker. + +New kernel exports (`src/runtime/supervise/chat-transport-executor.ts`): + +- `chatTransportExecutor(options): Executor` — one `execute` is one conversation SHOT: seed (fresh system prompt, or a resumed session's recorded history) + the task as the next user message, then completion → host tool calls → tool messages until the model answers without a tool call (or `maxTurnsPerShot`, default 200). Settles with the final assistant text as `out`. NON-streaming by design: the streaming `UsageEvent` channel cannot mark an unmetered turn (`tokens` has no `tokensKnown: false` twin — the documented limitation in `supervise/types`), while the one-shot `Spend` carries both honesty markers. +- Metering honesty: tokens from the transport's `usage` fields (`tokensKnown: false` when a turn omits them); dollars ONLY from the response's own cost fields (`usage.cost` / `usage.cost_usd` — `usdKnown: false` when absent). Never estimated from a local price table: this executor speaks to arbitrary endpoints whose models no local table can price. +- Fail-loud: transport failures (non-2xx, network faults, malformed completions) throw `ValidationError` — the scope's INFRA settle class — never a fake success. The turns that DID run are still recorded first, because resume-after-failure is a kernel-supported path. +- `ChatTransportTool` — the optional tool table: the OpenAI function spec the model sees plus the host-side `execute`; unknown tools and malformed arguments are fed back to the model as tool messages, never thrown. +- `chatCompletionsTransport({ url, bearer })` — the one buffered wire function (also the executor's default transport), exported so a paired harness can drive two arms through the SAME instance; `ChatTransportExecutorOptions.complete` injects a scripted transport for fully-offline runs (mirrors `RouterConfig.complete`). +- **Session continuity** — the executor is the resume consumer the kernel's `continuity: 'resume'` contract calls for: `createChatSessionStore()` keeps conversation histories keyed by settled worker id, `resume: WorkerResumeContext` continues `ofWorker`'s exact recorded message list (fails loud before any spend when the store has none — the same process-local resume boundary the kernel documents), and the finished conversation is recorded under `sessionKey` for the next resume. +- `chatWorkerSeam(options): MakeWorkerAgent` — the session-owning worker seam `workerFromBackend` refuses to be: profile model/prompt (including a graph's appended delegates directive) drive each spawn, `WorkerSpawnContext.resume` re-attaches through the seam's store keyed by kernel node ids, and an optional `deliverable` gates each settle through the existing `gateOnDeliverable` (settled ⟺ delivered). + +New example + proof: `examples/graphs/user-sim-conversation.ts` — a CONVERSATION as a graph: a simulated user is a NODE (persona profile as the root), the product agent is a chat-transport worker, each dialogue turn is one ledgered `delegates` traversal with `continuity: 'resume'`, and the offline test (`tests/examples/user-sim-conversation.test.ts`) asserts the resumed message-history chain on the requests CAPTURED at the wire (turn k = turn k−1's whole message list + the prior assistant reply + the new user turn), the `fresh`/`resume`/`resume` ledger stamps, the executor-seam lineage, and the metered spend in the one conserved pool. + +P1 parity live path fixed (the three audited #710 gaps, `examples/p1-parity`): + +- Both arms' coders now share ONE substrate: the multishot arm's transport and the graph arm's `chatTransportExecutor` speak the same bare chat-completions endpoint with the same wire model, and the parity delegates edge declares `continuity: 'resume'` so the graph arm's shots continue one session exactly as `runMultishot`'s single transcript does. As previously wired the graph arm spawned a full cli-bridge harness worker against the multishot arm's bare chat calls — a substrate gap that invalidated any live parity number. +- The live graph arm now HAS a driver: the `'chat'` graph backend requires the reviewer brain's `RouterConfig` (previously the live arm shipped with neither `brain` nor `router` and could not run). +- No silent model fallback anywhere: the coder model rides its profile, the driver model is explicit per-arm substrate config (`MultishotArmBackend.driverModel` / `RouterConfig.model`), and the live entry requires `VB_CLI_BRIDGE_URL`, `VB_CLI_BRIDGE_BEARER`, `VB_PARITY_MODEL`, `VB_PARITY_ROUTER_URL`, `VB_PARITY_ROUTER_KEY`, `VB_PARITY_DRIVER_MODEL` — a missing variable fails loud; `'parity/unspecified'` is gone. + ## 0.127.0 ### Continuity is a first-class axis of delegates traversals diff --git a/docs/api/primitive-catalog.md b/docs/api/primitive-catalog.md index 522b848a..693727c6 100644 --- a/docs/api/primitive-catalog.md +++ b/docs/api/primitive-catalog.md @@ -7,7 +7,7 @@ # Primitive catalog — the never-stale anti-reinvention inventory -> **GENERATED** from `@tangle-network/agent-runtime@0.127.0` and `@tangle-network/agent-eval@0.143.0` by `scripts/gen-primitive-catalog.mjs`. Do NOT hand-edit — run `pnpm run docs:api`. This is the mechanical companion to the JUDGMENT in `canonical-api.md` (§2 decision table + §1.5 AgentProfile law): that doc says WHICH primitive to reach for and what NOT to build; this catalog proves WHAT exists. Per-symbol signatures + `file:line` live in the per-module pages under `docs/api/`. +> **GENERATED** from `@tangle-network/agent-runtime@0.128.0` and `@tangle-network/agent-eval@0.143.0` by `scripts/gen-primitive-catalog.mjs`. Do NOT hand-edit — run `pnpm run docs:api`. This is the mechanical companion to the JUDGMENT in `canonical-api.md` (§2 decision table + §1.5 AgentProfile law): that doc says WHICH primitive to reach for and what NOT to build; this catalog proves WHAT exists. Per-symbol signatures + `file:line` live in the per-module pages under `docs/api/`. ## 1. agent-runtime — own public surface @@ -510,7 +510,7 @@ Import from `@tangle-network/agent-runtime/intelligence` — 166 exports. ### Execution kernel — recursive atom, supervision, executors, round-synchronous loop -Import from `@tangle-network/agent-runtime/kernel` — 710 exports. +Import from `@tangle-network/agent-runtime/kernel` — 719 exports. | Symbol | Kind | Summary | |---|---|---| @@ -537,6 +537,9 @@ Import from `@tangle-network/agent-runtime/kernel` — 710 exports. | `canonicalFindingEvent` | function | Producer-side cleanliness for the `finding` event. The findings payload is arbitrary analyst | | `canonicalizeAuthoredProfile` | function | Lift a profile the supervisor AUTHORED into the canonical shape every executor reads. | | `captureWorkerTraceEvidence` | function | Collect and persist one executor's structured tool trace without changing its task outcome. | +| `chatCompletionsTransport` | function | The default transport: POST `${url}/chat/completions` with an optional bearer. Fail-loud on | +| `chatTransportExecutor` | function | Build the chat-transport `Executor`: one `execute` = one conversation SHOT — seed (fresh system | +| `chatWorkerSeam` | function | The `makeWorkerAgent` seam over {@link chatTransportExecutor} — the continuity consumer | | `closingWorkerNote` | function | The worker's closing commentary off a local harness run: the TAIL of its | | `collectAgentTurn` | function | Drain a `streamAgentTurn` stream (or any `RuntimeStreamEvent` stream that | | `compareCheckOutcomes` | function | The selection order: crash < ran; then official pass-fraction; authored guesses only | @@ -550,6 +553,7 @@ Import from `@tangle-network/agent-runtime/kernel` — 710 exports. | `createActivityLog` | function | Create a bounded activity ring. `limit` caps memory for a worker that runs thousands of tools. | | `createAgentEnvironmentProviderRegistry` | function | Create a registry that resolves provider names to concrete provider instances. | | `createBudgetPool` | function | Create a conserved reservation pool from a root `Budget`. `now()` is injected so the | +| `createChatSessionStore` | function | In-memory `ChatSessionStore`. Entries are detached copies — a caller mutating a saved array | | `createEventBus` | function | Create the child→parent coordination bus: one typed pipe for settled outputs, questions, and analyst findings, with a priority-ordered pull queue and a pass-through subscribe lane. | | `createExecutor` | function | The single built-in executor factory. Picks a leaf backend by data (`config.backend`), | | `createExecutorRegistry` | function | The open resolver/registry. Pre-registers the three built-ins under their | @@ -789,6 +793,8 @@ Import from `@tangle-network/agent-runtime/kernel` — 710 exports. | `BudgetPoolRestore` | interface | State recovered from a prior process before new work is admitted. `committed` is measured spend | | `BusEvent` | interface | Every bus event is a discriminated union member keyed by `type`. | | `BusRecord` | interface | A published event stamped for ordering and observability. `seq` is the monotonic publish index; | +| `ChatSessionStore` | interface | Conversation history keyed by the settled worker id — the resume substrate. The kernel owns | +| `ChatTransportTool` | interface | One entry of the caller-provided tool table: the OpenAI function spec the model sees, and the | | `CheckExecChannel` | interface | Minimal exec channel the default runner needs. `SandboxInstance` (and therefore | | `CheckOutcome` | interface | How one candidate fared against the frozen visible checks, split by check kind. | | `CheckpointCapableBox` | interface | Loop-side widening of the box's optional checkpoint method. The | @@ -992,6 +998,7 @@ Import from `@tangle-network/agent-runtime/kernel` — 710 exports. | `AuthorizeDownMessage` | type | Product decision over an exact continuation before it is durably recorded or delivered. | | `AxisScoresOf` | type | Decompose ONE record into per-axis scores (e.g. judge dimensions). When set, it REPLACES the | | `BudgetReadout` | type | Post-reservation pool readout — the shape `Scope.budget` exposes. `tokensLeft`, | +| `ChatCompletionsTransport` | type | One buffered chat-completions call: the OpenAI-shape request body in, the parsed completion | | `CombinatorShape` | type | A combinator is just a `LoopShape`: a factory `(ShapeContext) => Agent` whose `Agent.act` | | `ContinuityMode` | type | How a spawn CONTINUES a node's prior work: `'fresh'` starts a brand-new session (the default, | | `CoordinationDeliveryEvidence` | type | Durable delivery evidence retained in commit order. An attempt without a later event carrying | @@ -1071,7 +1078,7 @@ Import from `@tangle-network/agent-runtime/kernel` — 710 exports. | `WorktreeCheckRunner` | type | The single shell-command-in-worktree runner seam (replaces the per-executor copies). | | `WorktreePatchArtifact` | type | Terminal artifact of one worktree-CLI run — the canonical worktree-harness result (the captured | -**Undocumented supporting types** (add a TSDoc line at the declaration to earn a table row): `AcquireOptions`, `AgentEnvironment`, `AgentEnvironmentCapabilities`, `AgentEnvironmentEvent`, `AgentEnvironmentProvider`, `AgentEnvironmentQuery`, `AgentEnvironmentSummary`, `AgentGraph`, `AgenticOptions`, `AgenticRunResult`, `AgenticTask`, `AgenticTool`, `AgentSession`, `AgentSessionRef`, `AgentTurnInput`, `AgentTurnResult`, `AllWorkersStalledOptions`, `AnalystRegistry`, `AnytimeReport`, `AnytimeStrategySummary`, `AnytimeTaskCurve`, `ArtifactHandle`, `AuditIntentInput`, `AuditIntentOptions`, `AuthoredHarness`, `AuthoredStrategy`, `AuthorStrategyOptions`, `BenchmarkConfig`, `BenchmarkLift`, `BenchmarkStrategySummary`, `BenchmarkTaskRow`, `BudgetPool`, `BusStats`, `ChampionPick`, `CheckpointRef`, `CheckpointRequest`, `CheckRunContext`, `CliWorktreeBridgeSeam`, `CoordinationMcpHandle`, `CopyOptions`, `CorpusReadbackOptions`, `CreateAgentEnvironmentInput`, `CreateTangleSandboxExactProcessProviderOptions`, `DefinedLeaderboard`, `DispatchReport`, `Driver`, `DriverAgentOptions`, `EventBus`, `EvolutionArchiveNode`, `EvolutionAuthor`, `EvolutionBandInfo`, `EvolutionCandidate`, `EvolutionGeneration`, `EvolutionReport`, `ExecRequest`, `ExecResult`, `ForkRequest`, `GitWorkspaceOptions`, `GraphResult`, `HarvestCorpusOptions`, `HarvestFailure`, `HarvestReport`, `Inbox`, `InProcessSandboxClientOptions`, `IntentAudit`, `Iteration`, `Leaderboard`, `LeaderboardOptions`, `LocalSandboxClientOptions`, `LoopDecisionPayload`, `LoopDispatchOptions`, `LoopEndedPayload`, `LoopIterationEndedPayload`, `LoopIterationStartedPayload`, `LoopPlanDescription`, `LoopResult`, `LoopSandboxPlacement`, `LoopStartedPayload`, `LoopTraceEmitter`, `LoopWinner`, `MaterializeLocalMcpOptions`, `McpEnvironmentOptions`, `McpToolDescriptor`, `NodeSnapshot`, `NoProgressForOptions`, `Observation`, `ObserveInput`, `ObserveOptions`, `OpenSandboxRunOptions`, `PairwiseOptions`, `PatchDeliverableOptions`, `PlacementInfo`, `PlateauOptions`, `ProgressTrackerOptions`, `PromotionGateOptions`, `PromotionVerdict`, `PublishOptions`, `ReproductionCheck`, `ResolveSandboxClientOptions`, `ResourceRequest`, `RollingDispatchOptions`, `RouterChatResult`, `RouterChatToolsResult`, `RouterConfig`, `RouterToolLoopResult`, `RunAgenticOptions`, `RunAgentRoundsOptions`, `RunGraphOptions`, `SandboxRun`, `ShotSpec`, `SpawnOpts`, `StdioMcpConnection`, `StdioMcpServerSpec`, `SteerableSandboxArgs`, `Strategy`, `StrategyEvolutionConfig`, `StrategyResult`, `StreamAgentTurnOptions`, `StructuralRolloutConfig`, `SuperviseOptions`, `SuperviseSurfaceOptions`, `SupervisorAgentDeps`, `SupervisorOpts`, `SupervisorSpanOptions`, `SupervisorSpanRecorder`, `SurfaceScore`, `ToolSpec`, `ToolStepInput`, `TraceSource`, `TrajectoryAnalysis`, `UntrackedCopyStats`, `ValidationCtx`, `Validator`, `VerifierEnvironmentOptions`, `WatchTraceOptions`, `WaterfallCollector`, `WaterfallReport`, `WaterfallSpan`, `WorkerEvidenceInput`, `Workspace`, `WorkspaceRequest`, `WorkspaceRun`, `WorktreeCliExecutorOptions`, `WorktreeFanoutOptions`, `AgentEnvironmentStatus`, `AgentSessionStatus`, `ChampionPolicy`, `EdgeDeliveryOutcome`, `GraphEdge`, `LoopTraceEvent`, `MakeWorkerAgent`, `RepairStop`, `SandboxControlClient`, `WorkspaceCommit`. +**Undocumented supporting types** (add a TSDoc line at the declaration to earn a table row): `AcquireOptions`, `AgentEnvironment`, `AgentEnvironmentCapabilities`, `AgentEnvironmentEvent`, `AgentEnvironmentProvider`, `AgentEnvironmentQuery`, `AgentEnvironmentSummary`, `AgentGraph`, `AgenticOptions`, `AgenticRunResult`, `AgenticTask`, `AgenticTool`, `AgentSession`, `AgentSessionRef`, `AgentTurnInput`, `AgentTurnResult`, `AllWorkersStalledOptions`, `AnalystRegistry`, `AnytimeReport`, `AnytimeStrategySummary`, `AnytimeTaskCurve`, `ArtifactHandle`, `AuditIntentInput`, `AuditIntentOptions`, `AuthoredHarness`, `AuthoredStrategy`, `AuthorStrategyOptions`, `BenchmarkConfig`, `BenchmarkLift`, `BenchmarkStrategySummary`, `BenchmarkTaskRow`, `BudgetPool`, `BusStats`, `ChampionPick`, `ChatTransportExecutorOptions`, `ChatWorkerSeamOptions`, `CheckpointRef`, `CheckpointRequest`, `CheckRunContext`, `CliWorktreeBridgeSeam`, `CoordinationMcpHandle`, `CopyOptions`, `CorpusReadbackOptions`, `CreateAgentEnvironmentInput`, `CreateTangleSandboxExactProcessProviderOptions`, `DefinedLeaderboard`, `DispatchReport`, `Driver`, `DriverAgentOptions`, `EventBus`, `EvolutionArchiveNode`, `EvolutionAuthor`, `EvolutionBandInfo`, `EvolutionCandidate`, `EvolutionGeneration`, `EvolutionReport`, `ExecRequest`, `ExecResult`, `ForkRequest`, `GitWorkspaceOptions`, `GraphResult`, `HarvestCorpusOptions`, `HarvestFailure`, `HarvestReport`, `Inbox`, `InProcessSandboxClientOptions`, `IntentAudit`, `Iteration`, `Leaderboard`, `LeaderboardOptions`, `LocalSandboxClientOptions`, `LoopDecisionPayload`, `LoopDispatchOptions`, `LoopEndedPayload`, `LoopIterationEndedPayload`, `LoopIterationStartedPayload`, `LoopPlanDescription`, `LoopResult`, `LoopSandboxPlacement`, `LoopStartedPayload`, `LoopTraceEmitter`, `LoopWinner`, `MaterializeLocalMcpOptions`, `McpEnvironmentOptions`, `McpToolDescriptor`, `NodeSnapshot`, `NoProgressForOptions`, `Observation`, `ObserveInput`, `ObserveOptions`, `OpenSandboxRunOptions`, `PairwiseOptions`, `PatchDeliverableOptions`, `PlacementInfo`, `PlateauOptions`, `ProgressTrackerOptions`, `PromotionGateOptions`, `PromotionVerdict`, `PublishOptions`, `ReproductionCheck`, `ResolveSandboxClientOptions`, `ResourceRequest`, `RollingDispatchOptions`, `RouterChatResult`, `RouterChatToolsResult`, `RouterConfig`, `RouterToolLoopResult`, `RunAgenticOptions`, `RunAgentRoundsOptions`, `RunGraphOptions`, `SandboxRun`, `ShotSpec`, `SpawnOpts`, `StdioMcpConnection`, `StdioMcpServerSpec`, `SteerableSandboxArgs`, `Strategy`, `StrategyEvolutionConfig`, `StrategyResult`, `StreamAgentTurnOptions`, `StructuralRolloutConfig`, `SuperviseOptions`, `SuperviseSurfaceOptions`, `SupervisorAgentDeps`, `SupervisorOpts`, `SupervisorSpanOptions`, `SupervisorSpanRecorder`, `SurfaceScore`, `ToolSpec`, `ToolStepInput`, `TraceSource`, `TrajectoryAnalysis`, `UntrackedCopyStats`, `ValidationCtx`, `Validator`, `VerifierEnvironmentOptions`, `WatchTraceOptions`, `WaterfallCollector`, `WaterfallReport`, `WaterfallSpan`, `WorkerEvidenceInput`, `Workspace`, `WorkspaceRequest`, `WorkspaceRun`, `WorktreeCliExecutorOptions`, `WorktreeFanoutOptions`, `AgentEnvironmentStatus`, `AgentSessionStatus`, `ChampionPolicy`, `EdgeDeliveryOutcome`, `GraphEdge`, `LoopTraceEvent`, `MakeWorkerAgent`, `RepairStop`, `SandboxControlClient`, `WorkspaceCommit`. ### Environment provider adapters — generic sandbox/compute bridge diff --git a/docs/api/runtime.md b/docs/api/runtime.md index 9257b634..f9a256d7 100644 --- a/docs/api/runtime.md +++ b/docs/api/runtime.md @@ -663,7 +663,7 @@ FS-backed `CoordinationLog`: append-only JSONL, fsynced per record. ###### Implementation of -[`CoordinationLog`](#coordinationlog).[`load`](#load) +[`CoordinationLog`](#coordinationlog).[`load`](#load-1) *** @@ -9827,6 +9827,248 @@ Fail loud if any reservation is still open — the conserved-pool leak detector. *** +### ChatSessionStore + +Conversation history keyed by the settled worker id — the resume substrate. The kernel owns + identity, ordering, ledger truth, and spend continuity; this store owns only the message + lists a `'resume'` spawn continues (`WorkerSpawnContext.resume.ofWorker` is the load key). + PROCESS-LOCAL by the same boundary the kernel documents for resume itself: a prior process's + workers are not resume targets. + +#### Methods + +##### load() + +> **load**(`workerId`): readonly `Record`\<`string`, `unknown`\>[] \| `undefined` + +###### Parameters + +###### workerId + +`string` + +###### Returns + +readonly `Record`\<`string`, `unknown`\>[] \| `undefined` + +##### save() + +> **save**(`workerId`, `messages`): `void` + +###### Parameters + +###### workerId + +`string` + +###### messages + +readonly `Record`\<`string`, `unknown`\>[] + +###### Returns + +`void` + +*** + +### ChatTransportTool + +One entry of the caller-provided tool table: the OpenAI function spec the model sees, and the + host-side implementation run when the model calls it. + +#### Properties + +##### spec + +> `readonly` **spec**: [`ToolSpec`](#toolspec) + +##### execute + +> `readonly` **execute**: (`args`, `task`) => `Promise`\<`string`\> + +Runs ON THIS HOST; the returned string folds back as the `tool` message. A throw is fed + back as an error message for the model to correct — a bad tool call is a real outcome, not + an infra fault. + +###### Parameters + +###### args + +`Record`\<`string`, `unknown`\> + +###### task + +`unknown` + +###### Returns + +`Promise`\<`string`\> + +*** + +### ChatTransportExecutorOptions + +#### Properties + +##### url + +> **url**: `string` + +OpenAI-compatible base URL (with or without `/v1`); the executor POSTs to + `${url}/chat/completions`. Ignored when `complete` is injected. + +##### bearer? + +> `optional` **bearer?**: `string` + +Bearer token for the default transport. Omit for an unauthenticated endpoint. + +##### model + +> **model**: `string` + +The wire model id sent on every completion. + +##### system? + +> `optional` **system?**: `string` + +System prompt seeding a FRESH conversation. A resumed conversation keeps the system message + it was recorded with — a session continues; it is not re-primed. + +##### tools? + +> `optional` **tools?**: readonly [`ChatTransportTool`](#chattransporttool)[] + +Tool table. Omitted = a pure conversation (no `tools` field on the wire). + +##### temperature? + +> `optional` **temperature?**: `number` + +##### maxTokens? + +> `optional` **maxTokens?**: `number` + +Output-token ceiling for ONE completion, sent as `max_tokens` on every request when set. + Omitted = no field on the wire, so the endpoint's own default governs. A harness pairing + this executor against another sampling path (P1 parity) pins BOTH arms to one value. + +##### maxTurnsPerShot? + +> `optional` **maxTurnsPerShot?**: `number` + +Inference-turn cap for ONE shot (one `execute`). Default 200 — a runaway backstop, not a + workflow limit (mirrors `routerToolsInlineExecutor.maxTurns`). + +##### complete? + +> `optional` **complete?**: [`ChatCompletionsTransport`](#chatcompletionstransport) + +Injected buffered transport — the offline seam (mirrors `RouterConfig.complete`). When set, + `url`/`bearer` are unused and NO network is touched. + +##### sessions? + +> `optional` **sessions?**: [`ChatSessionStore`](#chatsessionstore) + +Session store backing continuity. Required to record this conversation (with `sessionKey`) + or to continue a prior one (with `resume`). + +##### sessionKey? + +> `optional` **sessionKey?**: `string` + +The id this worker's conversation is recorded under at settle — the kernel node id when + spawned through a scope, so a later `'resume'` spawn's `resume.ofWorker` finds it. + +##### resume? + +> `optional` **resume?**: [`WorkerResumeContext`](#workerresumecontext) + +The resume lineage from `WorkerSpawnContext.resume`: this shot continues `ofWorker`'s + recorded message list. Requires `sessions` holding that conversation — fails loud before + any spend when it does not. + +##### profile? + +> `optional` **profile?**: `AgentProfile` + +Profile this executor materializes, for the kernel's materialization receipt. Omitted = + the node's receipt reads `executor-did-not-report` (a direct, unsupervised use). + +##### attemptId? + +> `optional` **attemptId?**: `string` + +Kernel-minted attempt id (`ExecutorNodeContext.attemptId`) binding the receipt to this + exact spawn. + +*** + +### ChatWorkerSeamOptions + +#### Properties + +##### url + +> **url**: `string` + +OpenAI-compatible base URL every spawned worker speaks. Unused when `complete` is set. + +##### bearer? + +> `optional` **bearer?**: `string` + +##### model? + +> `optional` **model?**: `string` + +Fallback wire model when a spawned profile carries none (`profile.model.default` wins). + +##### tools? + +> `optional` **tools?**: readonly [`ChatTransportTool`](#chattransporttool)[] + +##### temperature? + +> `optional` **temperature?**: `number` + +##### maxTokens? + +> `optional` **maxTokens?**: `number` + +Per-completion `max_tokens` for every spawned worker (see + [ChatTransportExecutorOptions.maxTokens](#maxtokens-6)). + +##### maxTurnsPerShot? + +> `optional` **maxTurnsPerShot?**: `number` + +##### complete? + +> `optional` **complete?**: [`ChatCompletionsTransport`](#chatcompletionstransport) + +Injected buffered transport — the offline seam; no network is touched when set. + +##### sessions? + +> `optional` **sessions?**: [`ChatSessionStore`](#chatsessionstore) + +Session store backing continuity. Default: one fresh in-memory store PER SEAM, matching the + kernel's process-local resume boundary (one seam = one run's sessions). + +##### deliverable? + +> `optional` **deliverable?**: [`DeliverableSpec`](#deliverablespec)\<`unknown`\> + +The completion oracle: each worker settles `valid` ⟺ this check passes on its final + assistant text (`gateOnDeliverable` — settled ⟺ DELIVERED, exactly how `workerFromBackend` + composes it). Pass the graph's deliverable so a keep-best driver can pick a winner; omitted, + workers settle unverdicted and only a driver `submit_result` can win. + +*** + ### DeliverableSpec The deployable completion oracle passed to [gateOnDeliverable](#gateondeliverable): a `check` that @@ -14060,7 +14302,7 @@ Assignment identity within the parent manager; absent only for the root. ###### Inherited from -[`SupervisorNodeContext`](#supervisornodecontext).[`profile`](#profile-6) +[`SupervisorNodeContext`](#supervisornodecontext).[`profile`](#profile-7) ##### task @@ -18957,6 +19199,31 @@ Why a reservation was refused. `budget-exhausted` means the pool ran out of a ch *** +### ChatCompletionsTransport + +> **ChatCompletionsTransport** = (`body`, `signal?`) => `Promise`\<`unknown`\> + +One buffered chat-completions call: the OpenAI-shape request body in, the parsed completion + JSON out. The ONE wire function of this module — the executor's default transport is built + from it, and a harness that must prove two arms share a substrate (P1 parity) drives BOTH + through the same instance. + +#### Parameters + +##### body + +`Record`\<`string`, `unknown`\> + +##### signal? + +`AbortSignal` + +#### Returns + +`Promise`\<`unknown`\> + +*** + ### CoordinationOwnerId > **CoordinationOwnerId** = `string` @@ -23570,6 +23837,94 @@ wall-clock limit. The readout is an absolute instant, not a shrinking remainder. *** +### chatCompletionsTransport() + +> **chatCompletionsTransport**(`opts`): [`ChatCompletionsTransport`](#chatcompletionstransport) + +The default transport: POST `${url}/chat/completions` with an optional bearer. Fail-loud on + any non-2xx — the status and body head become the settle reason. + +#### Parameters + +##### opts + +###### url + +`string` + +###### bearer? + +`string` + +#### Returns + +[`ChatCompletionsTransport`](#chatcompletionstransport) + +*** + +### createChatSessionStore() + +> **createChatSessionStore**(): [`ChatSessionStore`](#chatsessionstore) + +In-memory `ChatSessionStore`. Entries are detached copies — a caller mutating a saved array + cannot corrupt a recorded session. + +#### Returns + +[`ChatSessionStore`](#chatsessionstore) + +*** + +### chatTransportExecutor() + +> **chatTransportExecutor**(`opts`): [`Executor`](index.md#executor-2)\<`string`\> + +Build the chat-transport `Executor`: one `execute` = one conversation SHOT — seed (fresh system +prompt, or the resumed session's recorded history) + the task as the next user message, then +loop completion → host tool calls → tool messages until the model answers without a tool call +(or the turn cap). Settles with the final assistant text as `out`. + +Fail-loud contract: transport failures (non-2xx, network faults, malformed completions) throw +`ValidationError`, which the scope settles as an INFRA failure (`Settled.down.infra`) — never a +fake success. The accumulated conversation is still recorded before the throw when a store is +configured, because the inference HAPPENED and a resume may continue a failed session (the +kernel deliberately allows resume-after-failure; the seam decides). + +#### Parameters + +##### opts + +[`ChatTransportExecutorOptions`](#chattransportexecutoroptions) + +#### Returns + +[`Executor`](index.md#executor-2)\<`string`\> + +*** + +### chatWorkerSeam() + +> **chatWorkerSeam**(`opts`): [`MakeWorkerAgent`](#makeworkeragent) + +The `makeWorkerAgent` seam over [chatTransportExecutor](#chattransportexecutor) — the continuity consumer +`workerFromBackend` refuses to be. Every spawn becomes one conversation shot: the spawned +profile's system prompt + instructions (which is where a graph's delegates directive lands) +seed a fresh session, and a `'resume'` spawn re-attaches by loading `resume.ofWorker`'s +recorded message list from the seam's session store. Conversations are recorded under the +kernel node id, which is exactly what a later `resume.ofWorker` names. + +#### Parameters + +##### opts + +[`ChatWorkerSeamOptions`](#chatworkerseamoptions) + +#### Returns + +[`MakeWorkerAgent`](#makeworkeragent) + +*** + ### gateOnDeliverable() > **gateOnDeliverable**\<`Out`\>(`inner`, `deliverable`): [`Executor`](index.md#executor-2)\<`Out`\> diff --git a/docs/canonical-api.md b/docs/canonical-api.md index 1c9686f5..0650d816 100644 --- a/docs/canonical-api.md +++ b/docs/canonical-api.md @@ -4,7 +4,7 @@ Generated signatures and the complete export list live in docs/api/. Run pnpm docs:freshness after editing this file. --> -> **Version 0.127.0.** +> **Version 0.128.0.** > [`docs/api/primitive-catalog.md`](./api/primitive-catalog.md) lists every export and import path. > `agent-eval` must satisfy `>=0.143.0 <0.144.0`. > `sandbox` must satisfy `>=0.17.2 <0.18.0`. @@ -135,6 +135,7 @@ A general "loop" primitive is the single most common modelling error in this rep | Pick the **execution transport a driven loop runs on** (`sandbox` box / cli-bridge / router) from a product flag | `resolveSandboxClient({ backend })`: `/kernel` | a per-product `if (backend === 'router') …` branch re-wiring `createExecutor` + `inlineSandboxClient` | | Pick the **chat backend an in-process turn runs on** (`router`/`tcloud`/`cli-bridge`/`sandbox`) from a product flag | `resolveAgentBackend({ backend })`: root `.` | the copy-pasted `backend-name → createOpenAICompatibleBackend` branch every eval product hand-rolled (the copies drift) | | Pick / register a leaf backend, or bring your own agent | `createExecutor({ backend })` / `createExecutorRegistry()` / implement `Executor`: `/kernel` | a per-vendor adapter or closed `inline\|sandbox\|cli` switch (won't report through the `UsageEvent` channel) | +| Run a worker as a **conversation on a bare `/v1/chat/completions` endpoint** (no sandbox), with session continuity for `continuity: 'resume'` graphs | `chatTransportExecutor(options)` + `chatWorkerSeam({ url, sessions?, deliverable? })` + `createChatSessionStore()`: `/kernel` | a leaf-seam fake of a chat worker, a multishot transcript loop outside the kernel (no ledger, no conserved pool), or a resume that re-primes a fresh session | | Optimize text or named components with upstream GEPA | `officialGepa({ recipe, ... })`, passed as `improve(...).method` from root `.` | a local GEPA approximation, prompt mutation loop, or silent fallback when Python is unavailable | | Optimize one text surface with Microsoft SkillOpt | `officialSkillOpt({ trainer, optimizer, ... })`, passed as `improve(...).method` from root `.` | Runtime-owned SkillOpt search or a silent local fallback | | Improve one profile coordinate | `improve(profile, { surface, executionRef, method, trainScenarios, selectionScenarios, testScenarios, judges, agent, costCeiling })` from root `.`; `executionRef` binds saved work to executable behavior, `agent` receives the exact complete candidate profile, and the total-cost option limits the whole run | an implicit per-surface optimizer, a method that sees final-test cases, an unmeasured profile mutation, or separate optimizer and final-test spend limits | diff --git a/examples/graphs/README.md b/examples/graphs/README.md index 958a05cc..fbb51fbf 100644 --- a/examples/graphs/README.md +++ b/examples/graphs/README.md @@ -1,9 +1,9 @@ # graphs — agent topologies as plain data -Six runnable topologies for `runGraph` (the agent-graph layer over `supervise()`). +Seven runnable topologies for `runGraph` (the agent-graph layer over `supervise()`). Each file's graph is a ≤25-line data literal — nodes are canonical `AgentProfile`s, edges are typed values carrying versioned registry directives — and each `main()` prints the EDGE LEDGER as the proof artifact: every traversal, its outcome (`delivered | stripped | empty | unpropagated`), its continuity stamp (`fresh | resume | steer`), its byte count, and the concrete worker it reached. -All six run offline at $0 (scripted driver brain + in-process leaf workers, in [`shared.ts`](./shared.ts) — the same seams the kernel's own graph tests use). +All seven run offline at $0 (scripted driver brain + in-process leaf workers, in [`shared.ts`](./shared.ts) — the same seams the kernel's own graph tests use; `user-sim-conversation` runs the REAL `chatTransportExecutor` through an injected scripted transport). ```bash pnpm tsx examples/graphs/collaborates-review-loop.ts @@ -11,6 +11,7 @@ pnpm tsx examples/graphs/best-of-n.ts pnpm tsx examples/graphs/watchdog-steer.ts pnpm tsx examples/graphs/shot-loop.ts pnpm tsx examples/graphs/shot-loop-resumed.ts +pnpm tsx examples/graphs/user-sim-conversation.ts pnpm tsx examples/graphs/analyst-agent-review.ts ``` @@ -21,9 +22,10 @@ pnpm tsx examples/graphs/analyst-agent-review.ts | [`watchdog-steer.ts`](./watchdog-steer.ts) | root + one builder with a live trace; `watchWorkers` passthrough runs the shipped online detector panel | Mid-run intervention: the detector's finding reaches the driver over the bus, and the corrective steer lands as the delegates edge's second delivered traversal BEFORE settle. | | [`shot-loop.ts`](./shot-loop.ts) | reviewer(root) ↔ coder; `delegates maxTraversals: 3`, `analyzes` verify → reviewer | The multishot loop as data: each shot and each verify report is one ledgered traversal, the shot budget lives on the edge, and the deliverable gates on the verdict. | | [`shot-loop-resumed.ts`](./shot-loop-resumed.ts) | reviewer(root) ↔ coder; `delegates maxTraversals: 3, continuity: 'resume'` | Continuity as data: shot 1 spawns `fresh`, shots 2+ RESUME the coder's prior settled session — the executor seam receives `resume: { ofWorker, sequence }`, the ledger stamps every hop's continuity, and all shots spend from the one conserved pool. | +| [`user-sim-conversation.ts`](./user-sim-conversation.ts) | user-sim persona (root) ↔ product-agent chat worker; `delegates continuity: 'resume'` | A CONVERSATION as a graph (#721): the simulated user is a NODE (persona profile), the product agent runs on `chatTransportExecutor` (a bare chat-completions conversation, no sandbox), each dialogue turn is one ledgered traversal, and the wire-captured requests prove one growing message history re-attached across three workers. | | [`analyst-agent-review.ts`](./analyst-agent-review.ts) | root + implementer; `analyzes` whose analyst is the `reviewer` NODE (no delegates edge to it) | The analyst as a tool-equipped AGENT: the reviewer node is spawned on the implementer's settle with directive + trace evidence as its task, its settle output IS the finding, and its spend lands in the one conserved budget. | -The offline proof for all six (exact ledger counts, outcomes, destinations) lives in `tests/examples/graph-topologies.test.ts`. +The offline proof for all seven (exact ledger counts, outcomes, destinations) lives in `tests/examples/graph-topologies.test.ts` and `tests/examples/user-sim-conversation.test.ts`. ## Three ledger semantics worth knowing diff --git a/examples/graphs/user-sim-conversation.ts b/examples/graphs/user-sim-conversation.ts new file mode 100644 index 00000000..fac36cce --- /dev/null +++ b/examples/graphs/user-sim-conversation.ts @@ -0,0 +1,161 @@ +/** + * user-sim-conversation — a CONVERSATION as a graph (#721): a simulated user is a NODE, the + * product agent is a chat-transport worker, and each dialogue turn is one ledgered traversal. + * + * The root ('user-sim') is a persona profile — an optimizable artifact, not harness code: its + * driver brain plays the user, and each of its messages crosses the delegates edge as a spawn. + * The edge declares `continuity: 'resume'`, so turn 1 spawns the product agent `fresh` and every + * later turn RESUMES the same conversation: `chatWorkerSeam` (the session-owning executor seam + * over `chatTransportExecutor`) loads the prior worker's recorded message list and the new turn + * continues it — one growing OpenAI-shape message history, exactly like a chat session, while the + * kernel keeps identity, ordering, the edge ledger, and one conserved spend pool. + * + * Fully offline: the driver brain is scripted, and the product agent runs the REAL + * `chatTransportExecutor` through an injected scripted transport (zero network, $0) that captures + * every wire request — the proof that the resumed history chain is what actually crossed the + * transport. Run: pnpm tsx examples/graphs/user-sim-conversation.ts + */ + +import { + type AgentGraph, + chatWorkerSeam, + promptHandle, + type RunGraphOptions, + runGraph, + type WorkerSpawnContext, +} from '@tangle-network/agent-runtime/kernel' +import { printLedger, scriptedBrain } from './shared' + +const brief = promptHandle('delegates/worker-brief/v1') + +/** The product agent's scripted side of the dialogue, one reply per traversal. */ +export const AGENT_REPLIES = [ + 'Happy to help — do you want the Starter or the Pro plan?', + 'Pro is $49/user/month and includes SSO. Shall I place the order for 6 seats?', + 'ORDER-CONFIRMED: Pro plan, 6 seats with SSO — receipt #881.', +] + +/** The simulated user's turns — the persona's half of the dialogue, driven by the root brain. */ +export const USER_TURNS = [ + 'Hi — I need a team plan with SSO for 6 people.', + 'Pro, if SSO is included. What does it cost?', + 'Yes — place the order.', +] + +export function userSimConversation(): { + graph: AgentGraph + opts: RunGraphOptions + /** Every OpenAI-shape request body the product agent's transport received, in order — the + * resumed message-history chain, captured at the wire. */ + requests: Array> + /** Every spawn's kernel-authored context — the continuity/resume lineage the seam received. */ + contexts: Array +} { + // ── The topology: a two-party conversation as plain data ── + const graph: AgentGraph = { + nodes: [ + { + id: 'user-sim', + profile: { + name: 'user-sim', + prompt: { + systemPrompt: + 'You are Ada, a busy founder buying a team plan. Terse. SSO is non-negotiable.', + }, + }, + }, + { + id: 'product-agent', + profile: { + name: 'product-agent', + model: { default: 'scripted/product-agent' }, + prompt: { systemPrompt: 'You are the product sales agent. Close honestly.' }, + }, + }, + ], + edges: [ + { + kind: 'delegates', + from: 'user-sim', + to: 'product-agent', + directive: brief, + maxTraversals: 3, + continuity: 'resume', + }, + ], + deliverable: { + describe: 'a confirmed order', + check: (out) => typeof out === 'string' && out.includes('ORDER-CONFIRMED'), + }, + budget: { maxIterations: 30, maxTokens: 100_000 }, + } + + // ── The seams: a scripted user brain, a REAL chat worker on a scripted transport ── + const requests: Array> = [] + let replyIndex = 0 + const seam = chatWorkerSeam({ + url: 'http://offline.invalid', + // The completion oracle: each turn's worker settles `valid` ⟺ the order confirmed, so the + // keep-best finalizer crowns the confirming turn — same deliverable the graph declares. + deliverable: graph.deliverable, + // The injected transport IS the endpoint: capture the wire body, script the reply, meter + // real usage fields (0.25 each — exact in binary — so spend continuity is assertable). + complete: async (body) => { + requests.push(structuredClone(body)) + const content = AGENT_REPLIES[Math.min(replyIndex, AGENT_REPLIES.length - 1)] + replyIndex += 1 + return { + choices: [{ message: { content } }], + usage: { prompt_tokens: 12, completion_tokens: 9, cost: 0.25 }, + } + }, + }) + const contexts: Array = [] + const opts: RunGraphOptions = { + runId: 'usim', + makeWorkerAgent: (profile, context) => { + contexts.push(context) + return seam(profile, context) + }, + brain: scriptedBrain([ + ...USER_TURNS.flatMap((turn) => [ + { + toolCalls: [ + { name: 'spawn_agent', arguments: { profile: { name: 'product-agent' }, task: turn } }, + ], + }, + { toolCalls: [{ name: 'await_event', arguments: {} }] }, + ]), + { content: 'done' }, + ]), + } + return { graph, opts, requests, contexts } +} + +export async function main(): Promise { + const { graph, opts, requests, contexts } = userSimConversation() + const res = await runGraph(graph, opts) + printLedger('user-sim-conversation', res) + console.log('SPAWN CONTINUITY (what the executor seam received):') + for (const context of contexts) { + const lineage = + context?.resume === undefined + ? '' + : ` resume.ofWorker=${context.resume.ofWorker} sequence=${context.resume.sequence}` + console.log(` ${context?.assignmentId}: ${context?.continuity ?? 'fresh'}${lineage}`) + } + console.log('MESSAGE-HISTORY CHAIN (messages per wire request):') + for (const [i, req] of requests.entries()) { + const messages = req.messages as Array<{ role: string }> + console.log( + ` turn ${i + 1}: ${messages.length} messages [${messages.map((m) => m.role).join(', ')}]`, + ) + } +} + +if (import.meta.url === `file://${process.argv[1]}`) { + main().catch((err) => { + console.error(err) + process.exit(1) + }) +} diff --git a/examples/p1-parity/arms.ts b/examples/p1-parity/arms.ts index 2523641c..82cf419f 100644 --- a/examples/p1-parity/arms.ts +++ b/examples/p1-parity/arms.ts @@ -13,15 +13,34 @@ * The record maps each arm's OWN instrumentation onto shared field names; where the two forms * genuinely differ (edge ledger, conserved pool, early stop) the difference is documented on the * field and left visible in the data, never papered over. + * + * SUBSTRATE SYMMETRY (the validity invariant, per #710/#721): live, BOTH arms' coders are a + * conversation on the SAME bare OpenAI-compatible `/v1/chat/completions` endpoint — the multishot + * arm through `runMultishot`'s transport seam, the graph arm through `chatTransportExecutor` + * (whose delegates edge declares `continuity: 'resume'`, so its shots continue ONE session + * exactly as `runMultishot`'s single transcript does). Pairing a full harness-worker coder + * against a bare-chat coder would measure the substrate gap, not the orchestration layer, so + * that shape is not expressible here: the ONLY difference between the arms is the thing P1 + * exists to measure — the orchestration form (transcript loop vs supervised graph with ledger + + * conserved pool). Models are likewise explicit everywhere; a silent fallback id could let the + * two arms drift to different models unnoticed, so a missing model always fails loud. Sampling + * is part of the substrate too: both arms pin the coder's temperature + max_tokens to + * {@link PARITY_CODER_SAMPLING} (residual documented on {@link ParityRecord}). */ -import type { MultishotMessage, MultishotTransport } from '@tangle-network/agent-eval/multishot' +import type { + MultishotMessage, + MultishotResult, + MultishotTransport, +} from '@tangle-network/agent-eval/multishot' import { runMultishot } from '@tangle-network/agent-eval/multishot' import type { AgentProfile } from '@tangle-network/agent-interface' +import type { RuntimeHooks } from '@tangle-network/agent-runtime' import { type AgentGraph, type AnalystRegistry, type Budget, + chatWorkerSeam, type EdgeTraversal, GraphEdgeCapError, type MakeWorkerAgent, @@ -33,6 +52,20 @@ import { type ToolLoopChat, } from '@tangle-network/agent-runtime/kernel' +// ── The shared coder sampling (the F1 parity pin) ────────────────────────────── + +/** + * The ONE sampling configuration BOTH arms' coder completions use. The arms must sample + * identically or a paired row measures a sampling difference and calls it an orchestration + * effect. `runMultishot` hardcodes agent temperature 0.7 and defaults its turn-initial + * `max_tokens` to 2500; the graph arm pins the SAME two values through `chatWorkerSeam` → + * `chatTransportExecutor`, and the multishot arm pins `agentMaxTokens` explicitly so an upstream + * default change cannot silently split the arms. (Temperature has no `runMultishot` option — the + * offline suite asserts the captured agent requests carry this constant, so upstream drift fails + * a test instead of skewing live rows.) + */ +export const PARITY_CODER_SAMPLING = { temperature: 0.7, maxTokens: 2500 } as const + // ── The shared cell ──────────────────────────────────────────────────────────── /** One coding cell, fed VERBATIM to both arms — the input-equivalence contract of the harness. */ @@ -70,6 +103,12 @@ export interface CellSpec { * - Multishot `spend.tokens` is metered at the transport seam (the sum of `usage` on every agent and * driver completion); graph `spend` is the run's reconciled `spentTotal` from the conserved * pool's journal. Both are that form's honest total, measured by different machinery. + * - Coder sampling is pinned identically in both arms ({@link PARITY_CODER_SAMPLING}), with ONE + * residual: `runMultishot` lowers `max_tokens` to 2000 for tool-FOLLOW-UP completions — a + * second request class the graph arm does not have (`chatTransportExecutor` sends one pinned + * `max_tokens` on every request). This harness advertises no tools, so the follow-up class + * never fires here; a future tool-carrying cell would reintroduce the asymmetry. Documented, + * never silent. */ export interface ParityRecord { /** Did any shot satisfy the completion check? Graph arm: the run settled a winner (the @@ -80,6 +119,31 @@ export interface ParityRecord { shotsUsed: number /** Total measured resource spend for the arm's whole run (driver + coder legs). */ spend: { tokens: { input: number; output: number }; usd: number } + /** Where `spend.usd` came from. `'measured'` = every counted completion carried the provider's + * own cost field (`usage.cost` / `usage.cost_usd` → the transport's `costUsd`). `'estimated'` + * = at least one completion lacked it and `runMultishot`'s price-table fallback + * (`estimateRouterCost`) filled the gap — only the multishot arm's completed runs can produce + * this. `'unknown'` = at least one completion lacked it and NOTHING estimated it: the graph + * arm never estimates (the chat executor's no-fabricated-measurement rule), and neither does + * the multishot arm's infra-death path (the loop died before its estimator could run). */ + usdSource: 'measured' | 'estimated' | 'unknown' + /** Coder shots that died to TRANSPORT/INFRA faults, not to the task. Graph arm: infra-flagged + * `down` settles of delegates-spawned workers (`Settled.down.infra`, observed on the run's own + * hook stream). Multishot arm: transport throws counted at the arm's metering seam — either + * leg, because `runMultishot` has no infra channel and dies on the FIRST one, so a nonzero + * count also means the loop ended early and the row is an infra casualty, never an ordinary + * non-convergence. (The graph arm's driver-brain death surfaces as the run's no-winner + * reason, not here.) */ + infraShots: number + /** False when ANY counted completion lacked provider token usage — `spend.tokens` is then a + * known subtotal, not the measured total. Graph arm: the conserved pool's own taint flag + * (`spentTotal.tokensKnown`). Multishot arm: metered per response at the transport seam; a + * transport throw counts as an unreported turn (work may have burned tokens the provider + * never got to report — mirroring the kernel's never-reinterpret-as-zero rule). */ + tokensKnown: boolean + /** The dollar twin of `tokensKnown`: false when ANY counted completion lacked a provider cost + * field (equivalently, `usdSource !== 'measured'`). */ + usdKnown: boolean /** Wall-clock duration of the arm call, measured identically around both arms. */ wallMs: number /** Corrective direction DELIVERED to the coder after the initial brief. Multishot arm: driver (user) @@ -100,6 +164,12 @@ export interface ParityRecord { export interface MultishotArmBackend { readonly agentTransport: MultishotTransport readonly driverTransport: MultishotTransport + /** The reviewer (driver) leg's wire model — REQUIRED, no fallback. Substrate config of this + * arm, exactly as the paired graph arm declares its driver model on `RouterConfig.model` + * (the reviewer PROFILE stays model-less: as the graph ROOT it is materialized by the driver + * brain, whose model axis lives in that substrate config). Live runs feed BOTH arms the same + * env value; the offline backend pins a scripted id. */ + readonly driverModel: string /** The shared completion check, applied to each turn-initial coder reply. MUST be the same * predicate the paired graph arm's deliverable uses, or the comparison is invalid. */ readonly shotPassed: (assistantText: string) => boolean @@ -109,7 +179,7 @@ export interface MultishotArmBackend { readonly baseUrl?: string } -/** Execution seams for the graph arm: fully-scripted (offline/CI) or the live cli-bridge. */ +/** Execution seams for the graph arm: fully-scripted (offline/CI) or the live chat transport. */ export type GraphArmBackend = | { readonly kind: 'seam' @@ -120,14 +190,17 @@ export type GraphArmBackend = readonly shotPassed: (workerOutText: string) => boolean } | { - readonly kind: 'bridge' - readonly bridgeUrl: string - readonly bridgeBearer: string - /** Fallback bridge wire id (e.g. `pi/deepseek`); the spawned profile may select its own. */ - readonly model?: string - readonly cwd?: string - /** Router substrate for the reviewer (driver) brain. */ - readonly router?: RouterConfig + /** LIVE: the coder runs on `chatTransportExecutor` against the SAME bare chat-completions + * endpoint the multishot arm's transport posts to — the substrate-symmetry contract. */ + readonly kind: 'chat' + /** OpenAI-compatible base URL both arms' coders speak (e.g. a cli-bridge `/v1`). */ + readonly url: string + readonly bearer?: string + /** The coder wire model id — the same id the paired multishot arm sends. */ + readonly model: string + /** Router substrate for the reviewer (driver) brain. REQUIRED: a live graph driver with + * neither `brain` nor `router` cannot run at all. */ + readonly router: RouterConfig readonly shotPassed: (workerOutText: string) => boolean } @@ -151,7 +224,10 @@ export function parityAnalysts(): AnalystRegistry { /** The two-node reviewer→coder topology for one cell — plain data, the shot budget on the edge. * The cell's profiles are used AS-IS (node id = `profile.name`), the task is the root task - * (`deliverable.describe`) and each spawn's payload, and `shotPassed` is the deliverable. */ + * (`deliverable.describe`) and each spawn's payload, and `shotPassed` is the deliverable. + * The delegates edge declares `continuity: 'resume'` — every shot after the first CONTINUES the + * coder's session, mirroring `runMultishot`'s one persistent transcript, so the two arms share + * the conversation shape and differ only in orchestration. */ export function buildParityGraph( cell: CellSpec, shotPassed: (workerOutText: string) => boolean, @@ -170,6 +246,7 @@ export function buildParityGraph( to: coder, directive: promptHandle('delegates/worker-brief/v1'), maxTraversals: cell.shots, + continuity: 'resume', }, { kind: 'analyzes', @@ -189,50 +266,131 @@ export function buildParityGraph( // ── Arm A: agent-eval multishot ──────────────────────────────────────────────── +/** Marker for a transport throw inside `runMultishot`: the loop has no infra channel and dies on + * the first one, so the metering seam wraps the fault and the arm settles an honest infra row + * (`infraShots > 0`) instead of crashing the whole cell run. Any OTHER `runMultishot` rejection + * (e.g. an empty-driver authoring fault) still fails loud. */ +class LoopTransportFailure extends Error { + readonly fault: unknown + constructor(fault: unknown) { + super( + `p1-parity multishot transport failed: ${fault instanceof Error ? fault.message : String(fault)}`, + ) + this.name = 'LoopTransportFailure' + this.fault = fault + } +} + export async function runMultishotArm( cell: CellSpec, backend: MultishotArmBackend, ): Promise { validateCell(cell) - const tokens = { input: 0, output: 0 } - // Meter tokens at the transport seam — the usage channel `runMultishot` exposes. + // The arm's OWN meter at the transport seam — the one channel `runMultishot` exposes. Beyond + // tokens it records the validity facts the record must state: completions that lacked usage or + // a cost field, transport deaths, and (for the infra-abort path, where the loop returns no + // transcript) each leg's successful replies. With `tools: []` every agent completion IS one + // turn-initial shot reply, so the abort-path tallies match the transcript-derived ones exactly. + const meter = { + tokens: { input: 0, output: 0 }, + usdMeasured: 0, + turnsMissingUsage: 0, + turnsMissingCost: 0, + infraShots: 0, + agentReplies: [] as string[], + driverReplies: [] as string[], + } const metered = - (transport: MultishotTransport): MultishotTransport => + (transport: MultishotTransport, leg: 'agent' | 'driver'): MultishotTransport => async (req) => { - const res = await transport(req) - tokens.input += res.usage?.prompt_tokens ?? 0 - tokens.output += res.usage?.completion_tokens ?? 0 + let res: Awaited> + try { + res = await transport(req) + } catch (fault) { + // The dead turn's spend is UNREPORTED, not zero: mark both channels unknown, exactly as + // the kernel marks an infra-thrown executor's spend. + meter.infraShots += 1 + meter.turnsMissingUsage += 1 + meter.turnsMissingCost += 1 + throw new LoopTransportFailure(fault) + } + const usage = res.usage + if ( + typeof usage?.prompt_tokens === 'number' && + typeof usage?.completion_tokens === 'number' + ) { + meter.tokens.input += usage.prompt_tokens + meter.tokens.output += usage.completion_tokens + } else { + meter.turnsMissingUsage += 1 + } + // A completion without a provider cost field makes `runMultishot` substitute its + // price-table estimate — recorded so the row states `usdSource: 'estimated'`, never + // presenting an estimate as a measurement. + if (typeof res.costUsd === 'number') meter.usdMeasured += res.costUsd + else meter.turnsMissingCost += 1 + const content = (res.message.content ?? '').trim() + if (leg === 'agent') meter.agentReplies.push(content) + else if (content.length > 0) meter.driverReplies.push(content) // empty ⇒ retried, never delivered return res } const startedAt = Date.now() - const sim = await runMultishot({ - profile: cell.coderProfile, - persona: { id: 'parity-cell' }, - shape: { - buildOpener: () => cell.task, - buildDriverSystemPrompt: () => cell.reviewerProfile.prompt?.systemPrompt ?? '', - }, - tools: [], - toolExecutors: {}, - maxTurns: cell.shots, - agentModel: cell.coderProfile.model?.default ?? 'parity/unspecified', - driverModel: cell.reviewerProfile.model?.default ?? 'parity/unspecified', - agentTransport: metered(backend.agentTransport), - driverTransport: metered(backend.driverTransport), - apiKey: backend.apiKey ?? 'unused', - baseUrl: backend.baseUrl ?? 'http://unused.invalid', - }) + let sim: MultishotResult | undefined + try { + sim = await runMultishot({ + profile: cell.coderProfile, + persona: { id: 'parity-cell' }, + shape: { + buildOpener: () => cell.task, + buildDriverSystemPrompt: () => cell.reviewerProfile.prompt?.systemPrompt ?? '', + }, + tools: [], + toolExecutors: {}, + maxTurns: cell.shots, + agentModel: requireProfileModel(cell.coderProfile, 'coderProfile'), + driverModel: requireModel(backend.driverModel, 'MultishotArmBackend.driverModel'), + // Coder sampling parity (F1): pin the turn-initial ceiling to the shared constant; the + // agent leg's temperature 0.7 is hardcoded inside `runMultishot` and asserted by test. + agentMaxTokens: PARITY_CODER_SAMPLING.maxTokens, + agentTransport: metered(backend.agentTransport, 'agent'), + driverTransport: metered(backend.driverTransport, 'driver'), + apiKey: backend.apiKey ?? 'unused', + baseUrl: backend.baseUrl ?? 'http://unused.invalid', + }) + } catch (err) { + if (!(err instanceof LoopTransportFailure)) throw err + } const wallMs = Date.now() - startedAt - const shotReplies = turnInitialAssistantReplies(sim.transcript) - const steering = sim.transcript.slice(1).filter((msg) => msg.role === 'user') + // Completed run: shots/steering read off the loop's own transcript; the infra-abort path reads + // the meter (the transcript died with the loop). The two agree by construction — see the meter + // note above. + const shotReplies = sim !== undefined ? turnInitialAssistantReplies(sim.transcript) : [] + const steering = + sim !== undefined + ? sim.transcript + .slice(1) + .filter((msg) => msg.role === 'user') + .map((msg) => msg.content) + : meter.driverReplies + const replies = sim !== undefined ? shotReplies : meter.agentReplies return { - converged: shotReplies.some((text) => backend.shotPassed(text)), - shotsUsed: shotReplies.length, - spend: { tokens: { ...tokens }, usd: sim.costUsd }, + converged: replies.some((text) => backend.shotPassed(text)), + shotsUsed: replies.length, + // A completed run's usd is the loop's own honest total (which may CONTAIN estimates — stated + // by `usdSource`); an infra-aborted run reports only what was measured, never re-estimating. + spend: { + tokens: { ...meter.tokens }, + usd: sim !== undefined ? sim.costUsd : meter.usdMeasured, + }, + usdSource: + meter.turnsMissingCost === 0 ? 'measured' : sim !== undefined ? 'estimated' : 'unknown', + infraShots: meter.infraShots, + tokensKnown: meter.turnsMissingUsage === 0, + usdKnown: meter.turnsMissingCost === 0, wallMs, steeringDelivered: { count: steering.length, - bytes: steering.reduce((sum, msg) => sum + Buffer.byteLength(msg.content, 'utf8'), 0), + bytes: steering.reduce((sum, text) => sum + Buffer.byteLength(text, 'utf8'), 0), }, // No `ledger`: `runMultishot` has no edge instrumentation, and the runner never fakes one. } @@ -256,23 +414,51 @@ function turnInitialAssistantReplies(transcript: ReadonlyArray export async function runGraphArm(cell: CellSpec, backend: GraphArmBackend): Promise { validateCell(cell) const graph = buildParityGraph(cell, backend.shotPassed) + // The infra channel: `runGraph` composes caller hooks onto the run's own event stream, and an + // infra-flagged `down` settle rides `agent.child` with its `infra` marker. Collected here, then + // intersected with the ledger's delegates-bound worker ids so ONLY coder shots count (the + // driver brain's own death surfaces as the run's no-winner reason instead). + const downInfraWorkers = new Set() + const infraHooks: RuntimeHooks = { + onEvent: (event) => { + if (event.target !== 'agent.child' || event.phase !== 'after') return + const payload = event.payload as + | { childId?: unknown; status?: unknown; infra?: unknown } + | undefined + if ( + typeof payload?.childId === 'string' && + payload.status === 'down' && + payload.infra === true + ) { + downInfraWorkers.add(payload.childId) + } + }, + } const opts: RunGraphOptions = backend.kind === 'seam' ? { makeWorkerAgent: backend.makeWorkerAgent, brain: backend.brain, analysts: backend.analysts ?? parityAnalysts(), + hooks: infraHooks, } : { - backend: { - backend: 'bridge', - bridgeUrl: backend.bridgeUrl, - bridgeBearer: backend.bridgeBearer, - ...(backend.model !== undefined ? { model: backend.model } : {}), - ...(backend.cwd !== undefined ? { cwd: backend.cwd } : {}), - }, - ...(backend.router !== undefined ? { router: backend.router } : {}), + // The coder on the SAME bare chat transport the multishot arm posts to, with the + // session-owning seam honoring the edge's `continuity: 'resume'`, and the graph's own + // deliverable as the settle gate. The reviewer brain runs on the router substrate. + makeWorkerAgent: chatWorkerSeam({ + url: backend.url, + ...(backend.bearer !== undefined ? { bearer: backend.bearer } : {}), + model: backend.model, + // Coder sampling parity (F1): the same pinned temperature + max_tokens the multishot + // arm's coder leg sends, from the one shared constant — never a per-arm choice. + temperature: PARITY_CODER_SAMPLING.temperature, + maxTokens: PARITY_CODER_SAMPLING.maxTokens, + deliverable: graph.deliverable, + }), + router: backend.router, analysts: parityAnalysts(), + hooks: infraHooks, } const startedAt = Date.now() try { @@ -282,12 +468,19 @@ export async function runGraphArm(cell: CellSpec, backend: GraphArmBackend): Pro res.result.spentTotal, res.ledger, Date.now() - startedAt, + downInfraWorkers, ) } catch (err) { if (err instanceof GraphEdgeCapError) { // The cap (the cyclic-graph backstop), not the task, ended the run: an honest // non-convergence row, with the full evidence the error carries. - return graphRecord(false, err.result.spentTotal, err.ledger, Date.now() - startedAt) + return graphRecord( + false, + err.result.spentTotal, + err.ledger, + Date.now() - startedAt, + downInfraWorkers, + ) } throw err } @@ -298,21 +491,30 @@ function graphRecord( spentTotal: Spend, ledger: ReadonlyArray, wallMs: number, + downInfraWorkers: ReadonlySet, ): ParityRecord { const delegates = ledger.filter((row) => row.kind === 'delegates') // Each live coder worker is one shot; steers re-use an existing worker id, refused rows have // none — so distinct bound worker ids count executed shots exactly. - const shotsUsed = new Set( - delegates.filter((row) => row.workerId !== undefined).map((row) => row.workerId), - ).size + const workerIds = new Set( + delegates + .map((row) => row.workerId) + .filter((workerId): workerId is string => workerId !== undefined), + ) const steering = delegates.filter((row) => row.outcome === 'delivered' && row.traversal > 1) return { converged, - shotsUsed, + shotsUsed: workerIds.size, spend: { tokens: { input: spentTotal.tokens.input, output: spentTotal.tokens.output }, usd: spentTotal.usd, }, + // The graph arm NEVER estimates: dollars come only from provider cost fields, so the pool's + // taint flag decides between fully-measured and known-subtotal — 'estimated' is unreachable. + usdSource: spentTotal.usdKnown !== false ? 'measured' : 'unknown', + infraShots: [...downInfraWorkers].filter((workerId) => workerIds.has(workerId)).length, + tokensKnown: spentTotal.tokensKnown !== false, + usdKnown: spentTotal.usdKnown !== false, wallMs, steeringDelivered: { count: steering.length, @@ -335,6 +537,24 @@ function requireProfileName(profile: AgentProfile, field: string): string { return name } +/** Fail loud on a missing model: a silent fallback id would let the two arms drift to different + * models — the exact class of hidden asymmetry this harness exists to rule out. */ +function requireModel(model: string | undefined, field: string): string { + if (typeof model !== 'string' || model.length === 0) { + throw new Error( + `p1-parity: ${field} must name the model — no 'parity/unspecified' fallback; live runs ` + + 'take it from the environment (see run-parity.ts), offline backends pin a scripted id', + ) + } + return model +} + +/** The coder's model rides its PROFILE (both arms consume it: `agentModel` here, the chat seam in + * the graph arm) — required, same no-fallback rule. */ +function requireProfileModel(profile: AgentProfile, field: string): string { + return requireModel(profile.model?.default, `${field}.model.default`) +} + function validateCell(cell: CellSpec): void { const reviewer = requireProfileName(cell.reviewerProfile, 'reviewerProfile') const coder = requireProfileName(cell.coderProfile, 'coderProfile') diff --git a/examples/p1-parity/offline.ts b/examples/p1-parity/offline.ts index 0ddee9ca..612c088c 100644 --- a/examples/p1-parity/offline.ts +++ b/examples/p1-parity/offline.ts @@ -7,10 +7,13 @@ * * Synthetic accounting: every scripted completion reports `usage {5,5}` and `$0`, mirroring the * leaf seam's per-shot spend, so the two arms' metering pipelines carry comparable numbers - * offline. The multishot arm additionally meters its scripted DRIVER completions (`runMultishot`'s - * driver is an inference leg); the graph arm's scripted brain meters nothing (a live graph - * driver would meter through `spentBreakdown.driverInference`). Real numbers arrive only with - * the live backend. + * offline. BOTH driver legs are metered: the multishot arm meters its scripted DRIVER + * completions at the transport seam (`runMultishot`'s driver is an inference leg), and the graph + * arm's scripted brain reports the same synthetic usage per turn ({@link meteredScriptedBrain}), + * metering through `spentBreakdown.driverInference` exactly as a live router brain would — so a + * completed offline run's conserved pool stays fully KNOWN (`tokensKnown` / `usdKnown` true) and + * the validity channel on a `ParityRecord` is exercised, not skipped. Real numbers arrive only + * with the live backend. */ import type { @@ -18,7 +21,7 @@ import type { MultishotTransportRequest, } from '@tangle-network/agent-eval/multishot' import type { AgentProfile } from '@tangle-network/agent-interface' -import type { MakeWorkerAgent } from '@tangle-network/agent-runtime/kernel' +import type { MakeWorkerAgent, ToolLoopChat } from '@tangle-network/agent-runtime/kernel' import { type LeafShot, leafSeam, type ScriptedTurn, scriptedBrain } from '../graphs/shared' import type { CellSpec, GraphArmBackend, MultishotArmBackend } from './arms' @@ -77,13 +80,32 @@ export function offlineMultishotBackend(script: ShotScript): { } } return { - backend: { agentTransport, driverTransport, shotPassed: offlineShotPassed }, + backend: { + agentTransport, + driverTransport, + driverModel: 'scripted/parity-reviewer', + shotPassed: offlineShotPassed, + }, capture: { agentRequests, driverRequests }, } } // ── Graph arm: scripted brain + leaf seam ────────────────────────────────────── +/** A `scriptedBrain` whose every turn reports the synthetic usage (`{5,5}` tokens, $0): the + * driver-inference meter records a KNOWN turn instead of a `tokensKnown: false` taint, matching + * how the multishot arm's scripted driver leg is metered. An unmetered brain is still expressible + * (use `scriptedBrain` directly) — that models a driver whose inference channel reports nothing, + * which honestly taints the pool. */ +export function meteredScriptedBrain(turns: ScriptedTurn[]): ToolLoopChat { + const brain = scriptedBrain(turns) + return async (messages, tools) => ({ + ...(await brain(messages, tools)), + usage: { input: 5, output: 5 }, + costUsd: 0, + }) +} + export interface GraphCapture { /** Every profile the leaf factory received (the graph-pinned coder profile, with the * delegates directive appended to its instructions), in spawn order. */ @@ -140,7 +162,7 @@ export function offlineGraphBackend( backend: { kind: 'seam', makeWorkerAgent, - brain: scriptedBrain(turns), + brain: meteredScriptedBrain(turns), shotPassed: offlineShotPassed, }, capture: { spawnedProfiles, spawnedTasks }, diff --git a/examples/p1-parity/parity.test.ts b/examples/p1-parity/parity.test.ts index b95a8d7b..a1b3a919 100644 --- a/examples/p1-parity/parity.test.ts +++ b/examples/p1-parity/parity.test.ts @@ -11,21 +11,42 @@ * 3. The non-convergence path stays honest: a script with no passing shot drives the graph * into its delegates cap (`GraphEdgeCapError`), which maps to `converged: false` with the * refusal visible in the ledger — and the multishot arm reports the same verdict. + * 4. Validity channels ride every row: `infraShots` / `tokensKnown` / `usdKnown` / `usdSource` + * are all-clear on completed offline runs, and a scripted 500 in EITHER arm produces a row + * that is visibly an infra casualty (nonzero `infraShots`, unknown-marked spend) — never an + * ordinary non-convergence, never a crash of the cell run. + * 5. Coder sampling parity: the one shared pin ({@link PARITY_CODER_SAMPLING}) is what the + * arms' coder completions actually carry, asserted at the captured wire requests. */ +import { chatWorkerSeam } from '@tangle-network/agent-runtime/kernel' import { describe, expect, it } from 'vitest' import { buildParityGraph, type CellSpec, + type GraphArmBackend, + type MultishotArmBackend, + PARITY_CODER_SAMPLING, type ParityRecord, runGraphArm, runMultishotArm, } from './arms' -import { offlineGraphBackend, offlineMultishotBackend, offlineShotPassed } from './offline' +import { + meteredScriptedBrain, + offlineGraphBackend, + offlineMultishotBackend, + offlineShotPassed, +} from './offline' const parityCell = (shots: number): CellSpec => ({ task: 'make the failing test suite pass', - coderProfile: { name: 'coder', prompt: { systemPrompt: 'Make tests pass.' } }, + // The coder model is mandatory and pinned (the arms refuse a silent fallback); the reviewer + // profile stays model-less — the driver model is each arm's substrate config. + coderProfile: { + name: 'coder', + model: { default: 'scripted/parity-coder' }, + prompt: { systemPrompt: 'Make tests pass.' }, + }, reviewerProfile: { name: 'reviewer', prompt: { systemPrompt: 'Verify.' } }, shots, budget: { maxIterations: 30, maxTokens: 100_000 }, @@ -35,6 +56,13 @@ function expectWellFormed(record: ParityRecord): void { expect(typeof record.converged).toBe('boolean') expect(Number.isInteger(record.shotsUsed)).toBe(true) expect(record.shotsUsed).toBeGreaterThanOrEqual(0) + expect(Number.isInteger(record.infraShots)).toBe(true) + expect(record.infraShots).toBeGreaterThanOrEqual(0) + expect(typeof record.tokensKnown).toBe('boolean') + expect(typeof record.usdKnown).toBe('boolean') + expect(['measured', 'estimated', 'unknown']).toContain(record.usdSource) + // The dollar twin invariant: usdKnown ⟺ every counted completion carried a cost field. + expect(record.usdKnown).toBe(record.usdSource === 'measured') for (const n of [ record.spend.tokens.input, record.spend.tokens.output, @@ -48,6 +76,15 @@ function expectWellFormed(record: ParityRecord): void { } } +/** The all-clear validity channel a COMPLETED offline run must report: nothing died to infra, + * every completion carried usage + cost, and no estimator fired. */ +function expectAllClearValidity(record: ParityRecord): void { + expect(record.infraShots).toBe(0) + expect(record.tokensKnown).toBe(true) + expect(record.usdKnown).toBe(true) + expect(record.usdSource).toBe('measured') +} + describe('p1-parity — the same cell reaches both arms and both report honestly', () => { it('input equivalence: task, profiles, and shot budget arrive at both execution seams', async () => { const cell = parityCell(2) @@ -71,6 +108,14 @@ describe('p1-parity — the same cell reaches both arms and both report honestly }) // The shot budget reached `runMultishot`: exactly `shots` coder completions were requested. expect(multishot.capture.agentRequests).toHaveLength(cell.shots) + // Coder sampling parity (F1): every agent-leg completion carries the ONE shared pin — + // `runMultishot`'s hardcoded agent temperature asserted against the constant (drift in + // either fails here, not silently in live rows) and the explicitly pinned turn-initial + // max_tokens ceiling. + for (const req of multishot.capture.agentRequests) { + expect(req.temperature).toBe(PARITY_CODER_SAMPLING.temperature) + expect(req.maxTokens).toBe(PARITY_CODER_SAMPLING.maxTokens) + } // ── Graph arm, captured at the leaf factory ── const firstSpawn = graph.capture.spawnedProfiles[0] @@ -106,6 +151,11 @@ describe('p1-parity — the same cell reaches both arms and both report honestly expect(multishotRecord.converged).toBe(true) expect(graphRecord.converged).toBe(true) + // Completed offline runs are all-clear on the validity channel: zero infra deaths, every + // completion metered (usage + cost), no estimator fired in either arm. + expectAllClearValidity(multishotRecord) + expectAllClearValidity(graphRecord) + // The measured difference P1 exists to surface: the graph settles at the passing shot; the // `runMultishot` has no deliverable check and burns the full budget. expect(graphRecord.shotsUsed).toBe(2) @@ -118,9 +168,11 @@ describe('p1-parity — the same cell reaches both arms and both report honestly expect(graphRecord.steeringDelivered.bytes).toBeGreaterThan(0) expect(multishotRecord.steeringDelivered.bytes).toBeGreaterThan(0) - // Spend flows through each arm's own metering: graph = 2 leaf shots × {5,5} from the - // conserved pool's journal; multishot = (3 agent + 2 driver) × {5,5} at the transport seam. - expect(graphRecord.spend).toEqual({ tokens: { input: 10, output: 10 }, usd: 0 }) + // Spend flows through each arm's own metering, driver leg included: graph = 2 leaf shots × + // {5,5} + 7 metered driver-brain turns × {5,5} (2 spawns + 4 awaits + the final stop), + // reconciled from the conserved pool's journal; multishot = (3 agent + 2 driver) × {5,5} at + // the transport seam. + expect(graphRecord.spend).toEqual({ tokens: { input: 45, output: 45 }, usd: 0 }) expect(multishotRecord.spend).toEqual({ tokens: { input: 25, output: 25 }, usd: 0 }) // The honest ledger asymmetry: the graph's edge ledger is present and complete (2 delivered @@ -149,10 +201,115 @@ describe('p1-parity — the same cell reaches both arms and both report honestly expect(multishotRecord.shotsUsed).toBe(2) expect(graphRecord.shotsUsed).toBe(2) + // ORDINARY non-convergence: the task failed, the infrastructure did not — the infra channel + // is zero and every completion stayed fully metered (the scripted-500 rows below differ on + // exactly these fields). + expectAllClearValidity(multishotRecord) + expectAllClearValidity(graphRecord) + // The graph's evidence: two delivered shots, then the cap REFUSED the third spawn — the // refusal is a ledger row, not a swallowed error. const last = graphRecord.ledger?.at(-1) expect(last?.outcome).toBe('unpropagated') expect(last?.reason).toContain('traversal-cap-exhausted') }) + + it('multishot arm: a scripted-500 transport death is an infra row with the measured partials, never a crash', async () => { + const cell = parityCell(3) + const scripted = offlineMultishotBackend(['fail']) + let agentCalls = 0 + const backend: MultishotArmBackend = { + ...scripted.backend, + // Shot 1 completes (fails the check); the driver re-briefs; shot 2 dies on the wire. + agentTransport: async (req) => { + agentCalls += 1 + if (agentCalls === 2) throw new Error('upstream 500: bad gateway') + return scripted.backend.agentTransport(req) + }, + } + const record = await runMultishotArm(cell, backend) + expectWellFormed(record) + expect(record.converged).toBe(false) + // Visibly an infra casualty, not an ordinary non-convergence: the infra channel is nonzero + // (`runMultishot` has no infra channel of its own and died on the throw — the arm's seam + // caught it and settled this row instead of crashing the cell run). + expect(record.infraShots).toBe(1) + expect(record.shotsUsed).toBe(1) + expect(record.steeringDelivered.count).toBe(1) + // The completed turns' spend stayed measured (shot 1 + one driver re-brief, {5,5} each)… + expect(record.spend.tokens).toEqual({ input: 10, output: 10 }) + expect(record.spend.usd).toBe(0) + // …but the dead turn's spend is UNREPORTED, not zero: both channels marked unknown, and no + // price-table estimate substituted (the loop died before its estimator could run). + expect(record.tokensKnown).toBe(false) + expect(record.usdKnown).toBe(false) + expect(record.usdSource).toBe('unknown') + expect(record.ledger).toBeUndefined() + }) + + it('graph arm: an infra-dead coder shot is counted from the down-INFRA settle, with unknown-marked spend', async () => { + const cell = parityCell(1) + // The REAL chat executor on a scripted transport that 500s on its one turn: the worker + // settles down-INFRA (the executor's wrapped ValidationError is the scope's infra class), + // the delegates cap refuses a retry spawn, and the arm still returns an honest row. + const chatRequests: Array> = [] + const backend: GraphArmBackend = { + kind: 'seam', + makeWorkerAgent: chatWorkerSeam({ + url: 'http://offline.invalid', + temperature: PARITY_CODER_SAMPLING.temperature, + maxTokens: PARITY_CODER_SAMPLING.maxTokens, + deliverable: { + describe: cell.task, + check: (out) => typeof out === 'string' && offlineShotPassed(out), + }, + complete: async (body) => { + chatRequests.push(structuredClone(body)) + throw new Error('upstream 500: bad gateway') + }, + }), + // Metered like the backend's own brain, so the unknown-marked spend asserted below is + // attributable to the INFRA DEATH alone, never to an unmetered driver turn. + brain: meteredScriptedBrain([ + { + toolCalls: [ + { name: 'spawn_agent', arguments: { profile: { name: 'coder' }, task: cell.task } }, + ], + }, + // ONE event: the down settle. (No verify report — analysts fire on `done` settles only.) + { toolCalls: [{ name: 'await_event', arguments: {} }] }, + // The retry spawn drives INTO the exhausted delegates cap — refused, ledgered. + { + toolCalls: [ + { name: 'spawn_agent', arguments: { profile: { name: 'coder' }, task: 'retry' } }, + ], + }, + { content: 'done' }, + ]), + shotPassed: offlineShotPassed, + } + const record = await runGraphArm(cell, backend) + expectWellFormed(record) + expect(record.converged).toBe(false) + // Visibly an infra casualty: the dead worker's settle carried the kernel's infra flag and + // the arm counted it against the delegates-bound shot. + expect(record.infraShots).toBe(1) + expect(record.shotsUsed).toBe(1) + expect(record.steeringDelivered.count).toBe(0) + // The kernel marks an infra-thrown executor's spend unreported — never zero, never estimated. + expect(record.tokensKnown).toBe(false) + expect(record.usdKnown).toBe(false) + expect(record.usdSource).toBe('unknown') + // The graph half of the F1 pin reached the wire before the 500. + expect(chatRequests).toHaveLength(1) + expect(chatRequests[0]?.temperature).toBe(PARITY_CODER_SAMPLING.temperature) + expect(chatRequests[0]?.max_tokens).toBe(PARITY_CODER_SAMPLING.maxTokens) + // The evidence trail: the delivered spawn bound to the dead worker, then the cap refusal. + const delegates = record.ledger?.filter((row) => row.kind === 'delegates') + expect(delegates?.map((row) => [row.traversal, row.outcome])).toEqual([ + [1, 'delivered'], + [2, 'unpropagated'], + ]) + expect(delegates?.at(-1)?.reason).toContain('traversal-cap-exhausted') + }) }) diff --git a/examples/p1-parity/run-parity.ts b/examples/p1-parity/run-parity.ts index 43bd40de..94d7ac99 100644 --- a/examples/p1-parity/run-parity.ts +++ b/examples/p1-parity/run-parity.ts @@ -3,19 +3,35 @@ * print paired records. * * pnpm tsx examples/p1-parity/run-parity.ts --backend offline --cells 2 --shots 3 + * + * VB_CLI_BRIDGE_URL=http://127.0.0.1:3344/v1 VB_CLI_BRIDGE_BEARER=... \ + * VB_PARITY_MODEL=pi/deepseek \ + * VB_PARITY_ROUTER_URL=https://router.example/v1 VB_PARITY_ROUTER_KEY=... \ + * VB_PARITY_DRIVER_MODEL=deepseek/deepseek-chat \ * pnpm tsx examples/p1-parity/run-parity.ts --backend cli-bridge --cells 1 --shots 3 * * offline — scripted seams (mirrors examples/graphs/shared.ts): zero network, zero env, $0. * Shot script per cell: fail × (shots−1), then pass — so the graph arm settles on * the final shot while the multishot arm burns its whole budget, and the paired records * show exactly that. This mode is CI-safe and is what the vitest suite exercises. - * cli-bridge — the LIVE one-command entry for later: wires the real cli-bridge backend from - * VB_CLI_BRIDGE_URL / VB_CLI_BRIDGE_BEARER (+ VB_PARITY_MODEL for the coder's - * bridge wire id). Nothing in this repo's gates ever executes it. + * cli-bridge — the LIVE entry. Nothing in this repo's gates ever executes it. + * + * LIVE SUBSTRATE SYMMETRY (the validity invariant, per #710/#721). Both arms' coders are a conversation + * on the SAME bare OpenAI-compatible `/v1/chat/completions` endpoint (`VB_CLI_BRIDGE_URL`) with + * the SAME wire model (`VB_PARITY_MODEL`): the multishot arm posts through `runMultishot`'s + * transport seam, the graph arm runs `chatTransportExecutor` via `chatWorkerSeam` — with the + * delegates edge's `continuity: 'resume'` continuing ONE session across shots exactly as + * `runMultishot`'s single transcript does. Both arms' REVIEWERS likewise share a substrate: the + * multishot driver leg posts to the router endpoint (`VB_PARITY_ROUTER_URL`) with + * `VB_PARITY_DRIVER_MODEL`, and the graph arm's driver brain is `runGraph`'s router brain on the + * same config. What remains different is exactly the treatment P1 measures: the orchestration + * form (transcript loop vs supervised graph with edge ledger + conserved pool). Every model is + * explicit — a missing env var fails loud; there is no silent fallback id. */ import { parseArgs } from 'node:util' import type { MultishotTransport } from '@tangle-network/agent-eval/multishot' +import { chatCompletionsTransport } from '@tangle-network/agent-runtime/kernel' import type { CellSpec, GraphArmBackend, MultishotArmBackend, ParityRecord } from './arms' import { runGraphArm, runMultishotArm } from './arms' import { offlineGraphBackend, offlineMultishotBackend } from './offline' @@ -51,22 +67,30 @@ function parseCli(argv: string[]): CliOptions { function parityCell(index: number, shots: number): CellSpec { return { task: `parity cell ${index + 1}: make the failing test suite pass`, - coderProfile: { name: 'coder', prompt: { systemPrompt: 'Make tests pass.' } }, + // The coder model is PINNED on its profile (the arms refuse a model-less coder — a silent + // fallback could let the two arms drift apart); offline it names the scripted transport. + // The reviewer profile stays model-less: as the graph ROOT it is materialized by the driver + // brain, and the driver model is substrate config (multishot backend / graph RouterConfig). + coderProfile: { + name: 'coder', + model: { default: 'scripted/parity-coder' }, + prompt: { systemPrompt: 'Make tests pass.' }, + }, reviewerProfile: { name: 'reviewer', prompt: { systemPrompt: 'Verify.' } }, shots, budget: { maxIterations: 30, maxTokens: 100_000 }, } } -/** The live cell: same shape, plus the coder's bridge wire id and the marker contract the live - * completion check reads (see {@link LIVE_PASS_MARKER}). */ -function liveParityCell(index: number, shots: number, model: string): CellSpec { +/** The live cell: same shape, with BOTH models from the environment and the marker contract the + * live completion check reads (see {@link LIVE_PASS_MARKER}). */ +function liveParityCell(index: number, shots: number, env: LiveEnv): CellSpec { const base = parityCell(index, shots) return { ...base, coderProfile: { ...base.coderProfile, - model: { default: model }, + model: { default: env.coderModel }, prompt: { systemPrompt: 'Make tests pass. Print the exact line ' + @@ -76,55 +100,92 @@ function liveParityCell(index: number, shots: number, model: string): CellSpec { } } -// ── Live cli-bridge wiring (NOT executed by any gate — the later live entry) ─── +// ── Live wiring (NOT executed by any gate — the live entry) ──────────────────── -interface BridgeEnv { +interface LiveEnv { + /** The bare chat-completions endpoint BOTH arms' coders speak. */ url: string bearer: string - model: string + /** The coder wire model id, identical in both arms. */ + coderModel: string + /** The router substrate BOTH arms' reviewers run on. */ + routerUrl: string + routerKey: string + driverModel: string } -function requireBridgeEnv(): BridgeEnv { - const url = process.env.VB_CLI_BRIDGE_URL - const bearer = process.env.VB_CLI_BRIDGE_BEARER - const model = process.env.VB_PARITY_MODEL - if (!url || !bearer || !model) { +function requireLiveEnv(): LiveEnv { + const read = { + VB_CLI_BRIDGE_URL: process.env.VB_CLI_BRIDGE_URL, + VB_CLI_BRIDGE_BEARER: process.env.VB_CLI_BRIDGE_BEARER, + VB_PARITY_MODEL: process.env.VB_PARITY_MODEL, + VB_PARITY_ROUTER_URL: process.env.VB_PARITY_ROUTER_URL, + VB_PARITY_ROUTER_KEY: process.env.VB_PARITY_ROUTER_KEY, + VB_PARITY_DRIVER_MODEL: process.env.VB_PARITY_DRIVER_MODEL, + } + const missing = Object.entries(read) + .filter(([, value]) => !value) + .map(([key]) => key) + if (missing.length > 0) { throw new Error( - 'cli-bridge backend needs VB_CLI_BRIDGE_URL, VB_CLI_BRIDGE_BEARER and VB_PARITY_MODEL ' + - '(the coder bridge wire id, e.g. pi/deepseek) in the environment', + `cli-bridge backend: missing ${missing.join(', ')} — the coder endpoint/bearer/model ` + + '(VB_CLI_BRIDGE_URL, VB_CLI_BRIDGE_BEARER, VB_PARITY_MODEL) and the reviewer router ' + + 'substrate (VB_PARITY_ROUTER_URL, VB_PARITY_ROUTER_KEY, VB_PARITY_DRIVER_MODEL) are all ' + + 'required; no model has a fallback', ) } - return { url, bearer, model } + return { + url: read.VB_CLI_BRIDGE_URL as string, + bearer: read.VB_CLI_BRIDGE_BEARER as string, + coderModel: read.VB_PARITY_MODEL as string, + routerUrl: read.VB_PARITY_ROUTER_URL as string, + routerKey: read.VB_PARITY_ROUTER_KEY as string, + driverModel: read.VB_PARITY_DRIVER_MODEL as string, + } } -/** A multishot transport over cli-bridge's OpenAI-compatible chat-completions surface. */ -function bridgeTransport(env: BridgeEnv): MultishotTransport { +/** A multishot transport over an OpenAI-compatible chat-completions surface — built on the SAME + * wire function (`chatCompletionsTransport`) the graph arm's `chatTransportExecutor` uses, so + * the two arms' substrate symmetry is by construction, not by parallel implementations. The + * response's own cost field (`usage.cost` / `usage.cost_usd`, the cli-bridge and OpenRouter + * conventions — the same read order `chatTransportExecutor` uses) maps to the result's + * `costUsd`, so `runMultishot` meters the MEASURED dollars instead of firing its price-table + * estimator on every live turn; a turn genuinely without one stays estimator-visible and the + * row states it (`ParityRecord.usdSource`). */ +function completionsTransport(url: string, bearer: string): MultishotTransport { + const post = chatCompletionsTransport({ url, bearer }) return async (req) => { - const res = await fetch(`${env.url.replace(/\/$/, '')}/v1/chat/completions`, { - method: 'POST', - headers: { - 'content-type': 'application/json', - authorization: `Bearer ${env.bearer}`, - }, - body: JSON.stringify({ + const body = (await post( + { model: req.model, messages: req.messages, ...(req.tools !== undefined && req.tools.length > 0 ? { tools: req.tools } : {}), ...(req.temperature !== undefined ? { temperature: req.temperature } : {}), ...(req.maxTokens !== undefined ? { max_tokens: req.maxTokens } : {}), - }), - ...(req.signal !== undefined ? { signal: req.signal } : {}), - }) - if (!res.ok) { - throw new Error(`cli-bridge completion failed: ${res.status} ${await res.text()}`) - } - const body = (await res.json()) as { + }, + req.signal, + )) as { choices?: Array<{ message?: { content?: string | null; tool_calls?: never[] } }> - usage?: { prompt_tokens?: number; completion_tokens?: number } + usage?: { + prompt_tokens?: number + completion_tokens?: number + cost?: number + cost_usd?: number + } } const message = body.choices?.[0]?.message - if (message === undefined) throw new Error('cli-bridge completion returned no message') - return { message, ...(body.usage !== undefined ? { usage: body.usage } : {}) } + if (message === undefined) throw new Error('chat completion returned no message') + const costUsd = + typeof body.usage?.cost === 'number' + ? body.usage.cost + : typeof body.usage?.cost_usd === 'number' + ? body.usage.cost_usd + : undefined + return { + message, + ...(body.usage !== undefined ? { usage: body.usage } : {}), + ...(costUsd !== undefined ? { costUsd } : {}), + } } } @@ -133,24 +194,27 @@ function bridgeTransport(env: BridgeEnv): MultishotTransport { const LIVE_PASS_MARKER = 'ALL TESTS PASS' const livePassed = (text: string): boolean => text.includes(LIVE_PASS_MARKER) -function liveBackends(env: BridgeEnv): { +function liveBackends(env: LiveEnv): { multishot: MultishotArmBackend graph: GraphArmBackend } { - const transport = bridgeTransport(env) return { multishot: { - agentTransport: transport, - driverTransport: transport, + // Coder leg on the shared coder endpoint; reviewer (driver) leg on the shared router + // substrate — each leg matching its graph-arm counterpart, including the driver model. + agentTransport: completionsTransport(env.url, env.bearer), + driverTransport: completionsTransport(env.routerUrl, env.routerKey), + driverModel: env.driverModel, shotPassed: livePassed, apiKey: env.bearer, baseUrl: env.url, }, graph: { - kind: 'bridge', - bridgeUrl: env.url, - bridgeBearer: env.bearer, - model: env.model, + kind: 'chat', + url: env.url, + bearer: env.bearer, + model: env.coderModel, + router: { routerBaseUrl: env.routerUrl, routerKey: env.routerKey, model: env.driverModel }, shotPassed: livePassed, }, } @@ -168,15 +232,19 @@ function printRecord(row: PairedRow): void { const r = row.record console.log( `cell ${row.cell} ${row.arm.padEnd(5)} converged=${r.converged} shotsUsed=${r.shotsUsed} ` + - `tokens=${r.spend.tokens.input}/${r.spend.tokens.output} usd=${r.spend.usd} ` + - `wallMs=${r.wallMs} steering=${r.steeringDelivered.count}×/${r.steeringDelivered.bytes}B ` + + `infraShots=${r.infraShots} tokens=${r.spend.tokens.input}/${r.spend.tokens.output} ` + + `tokensKnown=${r.tokensKnown} usd=${r.spend.usd} usdSource=${r.usdSource} ` + + `usdKnown=${r.usdKnown} wallMs=${r.wallMs} ` + + `steering=${r.steeringDelivered.count}×/${r.steeringDelivered.bytes}B ` + `ledger=${r.ledger === undefined ? 'none (runMultishot has no edge ledger)' : `${r.ledger.length} rows`}`, ) if (r.ledger !== undefined) { for (const t of r.ledger) { const worker = t.workerId !== undefined ? ` -> ${t.workerId}` : '' const reason = t.reason !== undefined ? ` (${t.reason})` : '' - console.log(` #${t.traversal} ${t.edge} [${t.outcome}] ${t.bytes}B${worker}${reason}`) + console.log( + ` #${t.traversal} ${t.edge} [${t.outcome}|${t.continuity}] ${t.bytes}B${worker}${reason}`, + ) } } } @@ -195,9 +263,9 @@ export async function main(): Promise { multishotBackend = offlineMultishotBackend(script).backend graphBackend = offlineGraphBackend(cell, script).backend } else { - const env = requireBridgeEnv() + const env = requireLiveEnv() const backends = liveBackends(env) - cell = liveParityCell(i, cli.shots, env.model) + cell = liveParityCell(i, cli.shots, env) multishotBackend = backends.multishot graphBackend = backends.graph } diff --git a/package.json b/package.json index ba228406..1c67a821 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@tangle-network/agent-runtime", - "version": "0.127.0", + "version": "0.128.0", "description": "Shared task-lifecycle skeleton for agents: a recursive loop kernel for chat turns, one-shot tasks, and multi-attempt loops, with trace capture and eval-gated self-improvement. Domain behavior lives in adapters; scoring and ship-gates in @tangle-network/agent-eval.", "homepage": "https://github.com/tangle-network/agent-runtime#readme", "repository": { diff --git a/src/runtime/index.ts b/src/runtime/index.ts index d3e65f5d..3701fe1a 100644 --- a/src/runtime/index.ts +++ b/src/runtime/index.ts @@ -529,6 +529,21 @@ export { type ReservationTicket, spendFromUsageEvents, } from './supervise/budget' +// The chat-transport leaf (#721): a worker that IS a model conversation on a bare +// OpenAI-compatible /v1/chat/completions endpoint — no sandbox. Ships with its session store and +// the continuity-honoring `makeWorkerAgent` seam (the resume consumer `workerFromBackend` +// refuses to be), so conversation graphs and chat-shot loops compose from data. +export { + type ChatCompletionsTransport, + type ChatSessionStore, + type ChatTransportExecutorOptions, + type ChatTransportTool, + type ChatWorkerSeamOptions, + chatCompletionsTransport, + chatTransportExecutor, + chatWorkerSeam, + createChatSessionStore, +} from './supervise/chat-transport-executor' // The completion-oracle: settled ⟺ DELIVERED. `gateOnDeliverable` wraps an executor so its // settlement `valid` reflects a deployable deliverable check (a test/judge), never self-report. export { type DeliverableSpec, gateOnDeliverable } from './supervise/completion-gate' diff --git a/src/runtime/supervise/chat-transport-executor.ts b/src/runtime/supervise/chat-transport-executor.ts new file mode 100644 index 00000000..1bbfa029 --- /dev/null +++ b/src/runtime/supervise/chat-transport-executor.ts @@ -0,0 +1,524 @@ +/** + * The chat-transport leaf executor: a worker whose runtime is a plain OpenAI-compatible + * `/v1/chat/completions` transport — the worker IS a model conversation, not a sandboxed process + * (#721). Tool calls are optional (none, or a caller-provided tool table executed on this host). + * A chat worker gets everything real workers get through the open `Executor` port: node pinning, + * conserved spend, settle/verdict, journal + edge ledger. + * + * Module home: a standalone leaf-executor module beside `worktree-cli-executor.ts` — a direct + * `(options) → Executor` constructor, NOT a `createExecutor` backend variant. The reason is + * continuity: `workerFromBackend` (the backend-as-data path every `ExecutorConfig` rides) creates + * a fresh executor per spawn with no session re-attachment and deliberately FAILS LOUD on a + * `continuity: 'resume'` spawn; the documented resume consumer is a session-owning + * `makeWorkerAgent` seam. {@link chatWorkerSeam} is that seam, and this module ships both halves + * together so no caller re-derives the resume wiring. + * + * Transport shape: NON-streaming, one buffered POST per turn — the simplest honest choice. + * A streaming executor cannot mark an unmetered turn today (`UsageEvent`'s `tokens` variant has + * no `tokensKnown: false` twin — see the documented limitation in `./types`), while the one-shot + * path returns a whole `Spend` that carries both markers. Honesty wins over liveness here. + * + * Metering: tokens come from the transport's `usage` fields; a turn without usage marks + * `tokensKnown: false`. Dollars come ONLY from the response's own cost fields (`usage.cost` / + * `usage.cost_usd`, the cli-bridge and OpenRouter conventions); a turn without one marks + * `usdKnown: false`. NEVER estimated from a local price table — this executor speaks to arbitrary + * OpenAI-compatible endpoints whose models a local table cannot price, and a silent estimate is a + * fabricated measurement. + * + * @experimental + */ + +import { randomUUID } from 'node:crypto' +import { type AgentProfile, agentProfileSchema } from '@tangle-network/agent-interface' +import { contentAddress } from '../../durable/content-address' +import { ValidationError } from '../../errors' +import type { + MakeWorkerAgent, + WorkerResumeContext, + WorkerSpawnContext, +} from '../../mcp/tools/coordination' +import type { ToolSpec } from '../router-client' +import { zeroTokenUsage } from '../util' +import { canonicalizeAuthoredProfile } from './authoring' +import { type DeliverableSpec, gateOnDeliverable } from './completion-gate' +import { attestRuntimeOwnedExecutor, newExecutionAttemptId } from './materialization' +import { concreteModelId, concreteProfileModel } from './model-policy' +import { mergeAbortSignals, taskToPrompt } from './runtime' +import type { Agent, AgentSpec, Executor, ExecutorResult, Runtime, Spend } from './types' + +// ── The transport ────────────────────────────────────────────────────────────── + +/** One buffered chat-completions call: the OpenAI-shape request body in, the parsed completion + * JSON out. The ONE wire function of this module — the executor's default transport is built + * from it, and a harness that must prove two arms share a substrate (P1 parity) drives BOTH + * through the same instance. */ +export type ChatCompletionsTransport = ( + body: Record, + signal?: AbortSignal, +) => Promise + +/** The default transport: POST `${url}/chat/completions` with an optional bearer. Fail-loud on + * any non-2xx — the status and body head become the settle reason. */ +export function chatCompletionsTransport(opts: { + url: string + bearer?: string +}): ChatCompletionsTransport { + if (typeof opts.url !== 'string' || opts.url.length === 0) { + throw new ValidationError('chatCompletionsTransport: url required') + } + const endpoint = `${opts.url.replace(/\/$/, '')}/chat/completions` + return async (body, signal) => { + const res = await fetch(endpoint, { + method: 'POST', + headers: { + 'content-type': 'application/json', + ...(opts.bearer ? { authorization: `Bearer ${opts.bearer}` } : {}), + }, + body: JSON.stringify(body), + ...(signal ? { signal } : {}), + }) + if (!res.ok) { + throw new ValidationError(`chat transport ${res.status}: ${(await res.text()).slice(0, 200)}`) + } + return res.json() + } +} + +// ── The session store (continuity substrate) ─────────────────────────────────── + +/** Conversation history keyed by the settled worker id — the resume substrate. The kernel owns + * identity, ordering, ledger truth, and spend continuity; this store owns only the message + * lists a `'resume'` spawn continues (`WorkerSpawnContext.resume.ofWorker` is the load key). + * PROCESS-LOCAL by the same boundary the kernel documents for resume itself: a prior process's + * workers are not resume targets. */ +export interface ChatSessionStore { + load(workerId: string): ReadonlyArray> | undefined + save(workerId: string, messages: ReadonlyArray>): void +} + +/** In-memory `ChatSessionStore`. Entries are detached copies — a caller mutating a saved array + * cannot corrupt a recorded session. */ +export function createChatSessionStore(): ChatSessionStore { + const sessions = new Map>>() + return { + load: (workerId) => sessions.get(workerId), + save: (workerId, messages) => { + sessions.set(workerId, structuredClone(messages) as ReadonlyArray>) + }, + } +} + +// ── The executor ─────────────────────────────────────────────────────────────── + +/** One entry of the caller-provided tool table: the OpenAI function spec the model sees, and the + * host-side implementation run when the model calls it. */ +export interface ChatTransportTool { + readonly spec: ToolSpec + /** Runs ON THIS HOST; the returned string folds back as the `tool` message. A throw is fed + * back as an error message for the model to correct — a bad tool call is a real outcome, not + * an infra fault. */ + readonly execute: (args: Record, task: unknown) => Promise +} + +export interface ChatTransportExecutorOptions { + /** OpenAI-compatible base URL (with or without `/v1`); the executor POSTs to + * `${url}/chat/completions`. Ignored when `complete` is injected. */ + url: string + /** Bearer token for the default transport. Omit for an unauthenticated endpoint. */ + bearer?: string + /** The wire model id sent on every completion. */ + model: string + /** System prompt seeding a FRESH conversation. A resumed conversation keeps the system message + * it was recorded with — a session continues; it is not re-primed. */ + system?: string + /** Tool table. Omitted = a pure conversation (no `tools` field on the wire). */ + tools?: ReadonlyArray + temperature?: number + /** Output-token ceiling for ONE completion, sent as `max_tokens` on every request when set. + * Omitted = no field on the wire, so the endpoint's own default governs. A harness pairing + * this executor against another sampling path (P1 parity) pins BOTH arms to one value. */ + maxTokens?: number + /** Inference-turn cap for ONE shot (one `execute`). Default 200 — a runaway backstop, not a + * workflow limit (mirrors `routerToolsInlineExecutor.maxTurns`). */ + maxTurnsPerShot?: number + /** Injected buffered transport — the offline seam (mirrors `RouterConfig.complete`). When set, + * `url`/`bearer` are unused and NO network is touched. */ + complete?: ChatCompletionsTransport + /** Session store backing continuity. Required to record this conversation (with `sessionKey`) + * or to continue a prior one (with `resume`). */ + sessions?: ChatSessionStore + /** The id this worker's conversation is recorded under at settle — the kernel node id when + * spawned through a scope, so a later `'resume'` spawn's `resume.ofWorker` finds it. */ + sessionKey?: string + /** The resume lineage from `WorkerSpawnContext.resume`: this shot continues `ofWorker`'s + * recorded message list. Requires `sessions` holding that conversation — fails loud before + * any spend when it does not. */ + resume?: WorkerResumeContext + /** Profile this executor materializes, for the kernel's materialization receipt. Omitted = + * the node's receipt reads `executor-did-not-report` (a direct, unsupervised use). */ + profile?: AgentProfile + /** Kernel-minted attempt id (`ExecutorNodeContext.attemptId`) binding the receipt to this + * exact spawn. */ + attemptId?: string +} + +interface ChatCompletionMessage { + content?: string | null + tool_calls?: Array<{ id?: string; function?: { name?: string; arguments?: string } }> +} + +interface ChatCompletionResponse { + choices?: Array<{ message?: ChatCompletionMessage }> + usage?: { + prompt_tokens?: number + completion_tokens?: number + cost?: number + cost_usd?: number + } +} + +const CHAT_TRANSPORT_RUNTIME: Runtime = 'chat-transport' + +/** + * Build the chat-transport `Executor`: one `execute` = one conversation SHOT — seed (fresh system + * prompt, or the resumed session's recorded history) + the task as the next user message, then + * loop completion → host tool calls → tool messages until the model answers without a tool call + * (or the turn cap). Settles with the final assistant text as `out`. + * + * Fail-loud contract: transport failures (non-2xx, network faults, malformed completions) throw + * `ValidationError`, which the scope settles as an INFRA failure (`Settled.down.infra`) — never a + * fake success. The accumulated conversation is still recorded before the throw when a store is + * configured, because the inference HAPPENED and a resume may continue a failed session (the + * kernel deliberately allows resume-after-failure; the seam decides). + */ +export function chatTransportExecutor(opts: ChatTransportExecutorOptions): Executor { + const model = concreteModelId(opts.model) + if (!model) throw new ValidationError('chatTransportExecutor: model required') + if (!opts.complete && (typeof opts.url !== 'string' || opts.url.length === 0)) { + throw new ValidationError('chatTransportExecutor: url required (or inject `complete`)') + } + for (const tool of opts.tools ?? []) { + if (typeof tool.spec?.function?.name !== 'string' || typeof tool.execute !== 'function') { + throw new ValidationError( + 'chatTransportExecutor: every tools entry needs spec.function.name + execute', + ) + } + } + const maxTurns = opts.maxTurnsPerShot ?? 200 + if (!Number.isInteger(maxTurns) || maxTurns < 1) { + throw new ValidationError('chatTransportExecutor: maxTurnsPerShot must be a positive integer') + } + if (opts.maxTokens !== undefined && (!Number.isInteger(opts.maxTokens) || opts.maxTokens < 1)) { + throw new ValidationError('chatTransportExecutor: maxTokens must be a positive integer') + } + // Resolve the seed BEFORE any spend: a resume that cannot re-attach is a configuration fault. + let seed: Array> + if (opts.resume) { + if (!opts.sessions) { + throw new ValidationError( + "chatTransportExecutor: a 'resume' spawn needs `sessions` — the store holding the " + + 'conversation this shot continues', + ) + } + const prior = opts.sessions.load(opts.resume.ofWorker) + if (prior === undefined) { + throw new ValidationError( + `chatTransportExecutor: no recorded conversation for worker '${opts.resume.ofWorker}' — ` + + 'the session store holds only conversations recorded by this process (the kernel’s ' + + 'process-local resume boundary)', + ) + } + seed = structuredClone(prior) as Array> + } else { + seed = + opts.system !== undefined && opts.system.length > 0 + ? [{ role: 'system', content: opts.system }] + : [] + } + const transport = + opts.complete ?? + chatCompletionsTransport({ url: opts.url, ...(opts.bearer ? { bearer: opts.bearer } : {}) }) + const toolSpecs = (opts.tools ?? []).map((tool) => tool.spec) + const toolByName = new Map((opts.tools ?? []).map((tool) => [tool.spec.function.name, tool])) + + const controller = new AbortController() + let artifact: ExecutorResult | undefined + let executed = false + const executionId = opts.sessionKey ?? `chat-session-${randomUUID()}` + const attemptId = opts.attemptId ?? newExecutionAttemptId(executionId) + + const executor: Executor = { + runtime: CHAT_TRANSPORT_RUNTIME, + async execute(task, signal): Promise> { + // The seed is consumed once: a second execute would replay a stale conversation and + // double-record the session. One executor instance = one shot, per the spawn contract. + if (executed) { + throw new ValidationError('chatTransportExecutor: execute() called twice on one instance') + } + executed = true + const started = Date.now() + const messages = seed + messages.push({ role: 'user', content: taskToPrompt(task) }) + const linked = mergeAbortSignals(signal, controller.signal) + const tokens = zeroTokenUsage() + let tokensKnown = true + let usd = 0 + let usdKnown = true + let turns = 0 + let lastText = '' + try { + for (let t = 0; t < maxTurns; t += 1) { + const body: Record = { + model, + messages, + ...(toolSpecs.length > 0 ? { tools: toolSpecs, tool_choice: 'auto' } : {}), + ...(opts.temperature !== undefined ? { temperature: opts.temperature } : {}), + ...(opts.maxTokens !== undefined ? { max_tokens: opts.maxTokens } : {}), + } + let raw: unknown + try { + raw = await transport(body, linked) + } catch (cause) { + // An abort is the caller's own teardown/deadline — propagate untouched so the scope + // classifies it as the abort it is, not as a transport fault of this executor. + if (cause instanceof Error && cause.name === 'AbortError') throw cause + if (cause instanceof ValidationError) throw cause + throw new ValidationError( + `chatTransportExecutor: transport failed: ${cause instanceof Error ? cause.message : String(cause)}`, + ) + } + turns += 1 + const data = raw as ChatCompletionResponse + const usage = data?.usage + if ( + usage && + typeof usage.prompt_tokens === 'number' && + typeof usage.completion_tokens === 'number' + ) { + tokens.input += usage.prompt_tokens + tokens.output += usage.completion_tokens + } else { + tokensKnown = false + } + const turnCost = + typeof usage?.cost === 'number' + ? usage.cost + : typeof usage?.cost_usd === 'number' + ? usage.cost_usd + : undefined + if (turnCost !== undefined) usd += turnCost + else usdKnown = false + const msg = data?.choices?.[0]?.message + if (msg === undefined) { + throw new ValidationError( + 'chatTransportExecutor: transport returned no choices[0].message', + ) + } + if (typeof msg.content === 'string' && msg.content.length > 0) lastText = msg.content + const toolCalls = msg.tool_calls ?? [] + if (toolCalls.length === 0 || toolSpecs.length === 0) { + // Record the terminal assistant turn so a resumed session continues from it. + messages.push({ role: 'assistant', content: msg.content ?? '' }) + break + } + // Record the assistant turn verbatim, then run each call on the host and fold the + // result back as a `tool` message for the next turn (the routerToolsInline shape). + messages.push({ + role: 'assistant', + content: msg.content ?? '', + tool_calls: toolCalls.map((tc, i) => ({ + id: tc.id ?? `call_${i}`, + type: 'function', + function: { + name: tc.function?.name ?? '', + arguments: tc.function?.arguments ?? '{}', + }, + })), + }) + for (let i = 0; i < toolCalls.length; i += 1) { + const tc = toolCalls[i] + const id = tc?.id ?? `call_${i}` + const name = tc?.function?.name ?? '' + const tool = toolByName.get(name) + if (!tool) { + messages.push({ + role: 'tool', + tool_call_id: id, + content: `error: unknown tool '${name}'`, + }) + continue + } + let args: Record + try { + args = JSON.parse(tc?.function?.arguments ?? '{}') as Record + } catch { + messages.push({ + role: 'tool', + tool_call_id: id, + content: 'error: tool arguments were not valid JSON', + }) + continue + } + let result: string + try { + result = await tool.execute(args, task) + } catch (cause) { + result = `error: ${cause instanceof Error ? cause.message : String(cause)}` + } + messages.push({ role: 'tool', tool_call_id: id, content: result }) + } + } + } finally { + // The turns that RAN are recorded even when the shot failed: the kernel deliberately + // allows resuming a failed prior worker, and the seam decides. Recording only successes + // would silently amputate a resumed session's real history. + if (opts.sessions && opts.sessionKey !== undefined) { + opts.sessions.save(opts.sessionKey, messages) + } + } + const spent: Spend = { + iterations: turns, + tokens, + ...(tokensKnown ? {} : { tokensKnown: false }), + usd, + ...(usdKnown ? {} : { usdKnown: false }), + ms: Date.now() - started, + } + artifact = { + outRef: contentAddress({ kind: 'chat-transport', model, content: lastText, turns }), + out: lastText, + spent, + } + return artifact + }, + teardown(_grace): Promise<{ destroyed: boolean }> { + controller.abort() + return Promise.resolve({ destroyed: true }) + }, + resultArtifact() { + if (!artifact) { + throw new ValidationError('chatTransportExecutor: resultArtifact() read before execute()') + } + return { ...artifact, spent: artifact.spent } + }, + } + if (opts.profile === undefined) return executor + return attestRuntimeOwnedExecutor( + executor, + { + effectiveProfile: opts.profile, + backend: 'chat-transport', + model: { status: 'known', id: model }, + execution: { kind: 'session', id: executionId }, + materializer: 'chat-transport-conversation', + plan: { + kind: 'openai-chat-conversation', + model, + maxTurnsPerShot: maxTurns, + tools: toolSpecs, + resumeOf: opts.resume?.ofWorker ?? null, + }, + }, + { + attemptId, + binding: { + endpoint: opts.complete ? 'injected-transport' : opts.url, + model, + sessionKey: opts.sessionKey ?? null, + }, + descriptor: { + kind: 'chat-transport-session', + transport: opts.complete ? 'injected' : 'http', + backend: 'chat-transport', + }, + }, + ) +} + +// ── The worker seam (the resume consumer) ────────────────────────────────────── + +export interface ChatWorkerSeamOptions { + /** OpenAI-compatible base URL every spawned worker speaks. Unused when `complete` is set. */ + url: string + bearer?: string + /** Fallback wire model when a spawned profile carries none (`profile.model.default` wins). */ + model?: string + tools?: ReadonlyArray + temperature?: number + /** Per-completion `max_tokens` for every spawned worker (see + * {@link ChatTransportExecutorOptions.maxTokens}). */ + maxTokens?: number + maxTurnsPerShot?: number + /** Injected buffered transport — the offline seam; no network is touched when set. */ + complete?: ChatCompletionsTransport + /** Session store backing continuity. Default: one fresh in-memory store PER SEAM, matching the + * kernel's process-local resume boundary (one seam = one run's sessions). */ + sessions?: ChatSessionStore + /** The completion oracle: each worker settles `valid` ⟺ this check passes on its final + * assistant text (`gateOnDeliverable` — settled ⟺ DELIVERED, exactly how `workerFromBackend` + * composes it). Pass the graph's deliverable so a keep-best driver can pick a winner; omitted, + * workers settle unverdicted and only a driver `submit_result` can win. */ + deliverable?: DeliverableSpec +} + +/** + * The `makeWorkerAgent` seam over {@link chatTransportExecutor} — the continuity consumer + * `workerFromBackend` refuses to be. Every spawn becomes one conversation shot: the spawned + * profile's system prompt + instructions (which is where a graph's delegates directive lands) + * seed a fresh session, and a `'resume'` spawn re-attaches by loading `resume.ofWorker`'s + * recorded message list from the seam's session store. Conversations are recorded under the + * kernel node id, which is exactly what a later `resume.ofWorker` names. + */ +export function chatWorkerSeam(opts: ChatWorkerSeamOptions): MakeWorkerAgent { + if (!opts.complete && (typeof opts.url !== 'string' || opts.url.length === 0)) { + throw new ValidationError('chatWorkerSeam: url required (or inject `complete`)') + } + const sessions = opts.sessions ?? createChatSessionStore() + return (rawProfile, spawnContext?: WorkerSpawnContext) => { + // The supervisor authors in the skill's flat vocabulary; lift + validate here exactly like + // `workerFromBackend` — the one other place a profile becomes a spawnable worker. + const parsed = agentProfileSchema.safeParse(canonicalizeAuthoredProfile(rawProfile)) + if (!parsed.success) { + throw new ValidationError(`chatWorkerSeam: invalid AgentProfile: ${parsed.error.message}`) + } + const profile = parsed.data + const model = concreteProfileModel(profile) ?? concreteModelId(opts.model) + if (!model) { + throw new ValidationError( + 'chatWorkerSeam: no model — set ChatWorkerSeamOptions.model or AgentProfile.model.default', + ) + } + const system = [profile.prompt?.systemPrompt, ...(profile.prompt?.instructions ?? [])] + .filter((line): line is string => typeof line === 'string' && line.trim().length > 0) + .join('\n') + const name = profile.name ?? 'chat-worker' + const spec: AgentSpec = { + profile, + harness: null, + // Per-spawn factory: built only after admission, with the kernel node context — the node id + // is the session-record key a later resume names, and the attempt id binds the receipt. + executorFactory: (executorSpec, ctx) => { + const executor = chatTransportExecutor({ + url: opts.url, + ...(opts.bearer !== undefined ? { bearer: opts.bearer } : {}), + model, + ...(system.length > 0 ? { system } : {}), + ...(opts.tools !== undefined ? { tools: opts.tools } : {}), + ...(opts.temperature !== undefined ? { temperature: opts.temperature } : {}), + ...(opts.maxTokens !== undefined ? { maxTokens: opts.maxTokens } : {}), + ...(opts.maxTurnsPerShot !== undefined ? { maxTurnsPerShot: opts.maxTurnsPerShot } : {}), + ...(opts.complete !== undefined ? { complete: opts.complete } : {}), + sessions, + ...(ctx.node?.nodeId !== undefined ? { sessionKey: ctx.node.nodeId } : {}), + ...(spawnContext?.resume !== undefined ? { resume: spawnContext.resume } : {}), + profile: executorSpec.profile, + ...(ctx.node?.attemptId !== undefined ? { attemptId: ctx.node.attemptId } : {}), + }) + return opts.deliverable ? gateOnDeliverable(executor, opts.deliverable) : executor + }, + } + return { name, act: async () => undefined, executorSpec: spec } as Agent & { + executorSpec: AgentSpec + } + } +} diff --git a/src/runtime/supervise/runtime.ts b/src/runtime/supervise/runtime.ts index cf5ea6fa..77984e01 100644 --- a/src/runtime/supervise/runtime.ts +++ b/src/runtime/supervise/runtime.ts @@ -2990,8 +2990,10 @@ function readSeam(ctx: ExecutorContext, key: string, who: string): T { } /** A leaf task is opaque (`unknown`). A string is the prompt verbatim; an object - * with a `prompt`/`content`/`task` string field uses it; otherwise it serializes. */ -function taskToPrompt(task: unknown): string { + * with a `prompt`/`content`/`task` string field uses it; otherwise it serializes. + * Module-exported (not package surface) so sibling leaf executors read a task + * identically instead of re-deriving the rule. */ +export function taskToPrompt(task: unknown): string { if (typeof task === 'string') return task if (task && typeof task === 'object') { const obj = task as Record @@ -3045,8 +3047,9 @@ function linkSignals(a: AbortSignal, b: AbortSignal): AbortSignal | undefined { } /** Combine N abort signals into one that fires when ANY does. Node-portable (no `AbortSignal.any`, - * which needs >=20.3 — the package floor is >=20). */ -function mergeAbortSignals(...signals: AbortSignal[]): AbortSignal { + * which needs >=20.3 — the package floor is >=20). Module-exported (not package surface) so + * sibling leaf executors share the one portable implementation. */ +export function mergeAbortSignals(...signals: AbortSignal[]): AbortSignal { const c = new AbortController() const onAbort = () => c.abort() for (const s of signals) { diff --git a/src/testing/fixtures/agent-improvement-proposal.json b/src/testing/fixtures/agent-improvement-proposal.json index c78fa876..42532b50 100644 --- a/src/testing/fixtures/agent-improvement-proposal.json +++ b/src/testing/fixtures/agent-improvement-proposal.json @@ -1,6 +1,6 @@ { "changedSurfaces": ["prompt"], - "digest": "sha256:3e71923bf3dedf470bb5aec4333e9fbd0906f27380c73b76adf80dc6d2ef6a07", + "digest": "sha256:5feb51aa14c16c29e4b06a4f344dec42ecc7c13f0a876c17ed743014580dc93a", "evaluation": { "decision": { "contributingChecks": [ @@ -4810,7 +4810,7 @@ ], "metadata": { "fixture": "agent-improvement-proposal", - "runtimeVersion": "0.127.0" + "runtimeVersion": "0.128.0" }, "objectives": [ { @@ -4921,8 +4921,8 @@ "baselineContentHash": "sha256:5c21ee53e513fc604cb09754e21c392b24a424da0ef37dbf8f1ee4a8a0b08f09", "candidateContentHash": "sha256:60fcbb1c728194bd51d7d19cb732d1c3f1881dce7e0a6266b41c8b98cfd65693", "kind": "agent-eval-loop", - "recordDigest": "sha256:7111287a19de2c3df46d005f7f96fc45e0d6889e31e5fd63f1bd760d5b4059e0", - "runId": "agent-runtime-0.127.0-proposal-fixture", + "recordDigest": "sha256:21aec498ac5cf1fec87690d5216cf4edb811e0db19b841f8be84c1bea8a9365e", + "runId": "agent-runtime-0.128.0-proposal-fixture", "schema": "agent-candidate-experiment" } }, @@ -4949,5 +4949,5 @@ ], "kind": "agent-improvement-proposal", "proposedAt": "2026-07-10T01:00:00.000Z", - "runId": "agent-runtime-0.127.0-proposal-fixture" + "runId": "agent-runtime-0.128.0-proposal-fixture" } diff --git a/src/testing/fixtures/agent-profile-improvement-proposal.json b/src/testing/fixtures/agent-profile-improvement-proposal.json index 4b44b5e6..a02b52f5 100644 --- a/src/testing/fixtures/agent-profile-improvement-proposal.json +++ b/src/testing/fixtures/agent-profile-improvement-proposal.json @@ -1,6 +1,6 @@ { "changedSurfaces": ["prompt", "skills"], - "digest": "sha256:7623fd80722dcb85d83bc896e592eaa4aa98e1f0a706f2a95d5218ab53ca0824", + "digest": "sha256:481b9db8d4e8e1834114e3b922876eaa0d68ac007c094ce9b2bd9aa7a14d812e", "evaluation": { "decision": { "contributingChecks": [ @@ -1715,7 +1715,7 @@ ], "metadata": { "fixture": "agent-profile-improvement-proposal", - "runtimeVersion": "0.127.0" + "runtimeVersion": "0.128.0" }, "objectives": [ { @@ -1826,7 +1826,7 @@ "baselineContentHash": "sha256:21c495a37c418c10bde64fbaa188beddeed31f1f051ea60a6a6582a9ee0db704", "candidateContentHash": "sha256:103f77bc8481601eef1ad5fe6ba84a40dffabc3a44f421f8c8559121edab84e9", "kind": "agent-eval-loop", - "recordDigest": "sha256:6e7d9cbfc9327d5ce1bde8c6de799bb4af0f5e43aef74e4e970f91469651425a", + "recordDigest": "sha256:9c0bf8c61092567c90d1d40e48ba3f44f70433c80dcd6d28cc14b14a80bcd295", "runId": "profile-improvement-1", "schema": "agent-profile-improvement-experiment" } diff --git a/tests/examples/user-sim-conversation.test.ts b/tests/examples/user-sim-conversation.test.ts new file mode 100644 index 00000000..27bf197b --- /dev/null +++ b/tests/examples/user-sim-conversation.test.ts @@ -0,0 +1,84 @@ +/** + * user-sim-conversation — the conversation-graph proof (#721), fully offline. + * + * What must hold, asserted at the seams the run ACTUALLY crossed (never on intent): + * 1. The resumed MESSAGE-HISTORY CHAIN at the wire: each turn's captured transport request is + * the previous request's whole message list + the previous assistant reply + the new user + * turn — one growing conversation, re-attached across three spawned workers. + * 2. The ledger stamps how every hop continued (`fresh`, then `resume` × 2) and binds each + * traversal to the concrete worker that ran it. + * 3. The kernel-authored resume lineage reached the executor seam (`ofWorker` chains the prior + * settled worker, `sequence` counts the chain). + * 4. Spend continuity: all three turns metered from the transport's OWN usage/cost fields into + * the run's one conserved pool. + * 5. The deliverable crowns the confirming turn as the winner. + */ + +import { describe, expect, it } from 'vitest' +import { + AGENT_REPLIES, + USER_TURNS, + userSimConversation, +} from '../../examples/graphs/user-sim-conversation' +import { runGraph } from '../../src/runtime/supervise/graph' + +describe('examples/graphs/user-sim-conversation — turns are traversals, the session is one message list', () => { + it('resumes one growing conversation across three workers, ledgered and metered', async () => { + const { graph, opts, requests, contexts } = userSimConversation() + const res = await runGraph(graph, opts) + + // ── 5. The confirming turn wins through the deliverable ── + expect(res.result.kind).toBe('winner') + if (res.result.kind === 'winner') expect(res.result.out).toBe(AGENT_REPLIES[2]) + expect(res.exhaustedEdges).toEqual([]) + + // ── 2. The ledger: one delegates edge, three delivered traversals, continuity stamped ── + expect( + res.ledger.map((row) => [row.edge, row.traversal, row.outcome, row.continuity, row.workerId]), + ).toEqual([ + ['delegates:user-sim->product-agent', 1, 'delivered', 'fresh', 'usim:s0'], + ['delegates:user-sim->product-agent', 2, 'delivered', 'resume', 'usim:s1'], + ['delegates:user-sim->product-agent', 3, 'delivered', 'resume', 'usim:s2'], + ]) + + // ── 3. The kernel-authored lineage reached the seam: each resume names the prior worker ── + expect(contexts.map((c) => [c?.continuity, c?.resume?.ofWorker, c?.resume?.sequence])).toEqual([ + ['fresh', undefined, undefined], + ['resume', 'usim:s0', 2], + ['resume', 'usim:s1', 3], + ]) + + // ── 1. THE chain, at the wire: request k = request k−1 + [assistant k−1, user k] ── + expect(requests).toHaveLength(3) + const messagesOf = (i: number) => requests[i]?.messages as Array> + // Turn 1: a fresh session — the pinned profile's system prompt (with the appended delegates + // directive) + the opener. Nothing else. + expect(messagesOf(0)).toHaveLength(2) + expect(messagesOf(0)[0]?.role).toBe('system') + expect(String(messagesOf(0)[0]?.content)).toContain( + 'You are the product sales agent. Close honestly.', + ) + expect(messagesOf(0)[1]).toEqual({ role: 'user', content: USER_TURNS[0] }) + for (const turn of [1, 2]) { + expect(messagesOf(turn)).toEqual([ + ...messagesOf(turn - 1), + { role: 'assistant', content: AGENT_REPLIES[turn - 1] }, + { role: 'user', content: USER_TURNS[turn] }, + ]) + } + // Every wire call carried the worker's model, NO tools field (a pure conversation), and NO + // sampling fields the seam never configured (nothing is silently injected). + for (const req of requests) { + expect(req.model).toBe('scripted/product-agent') + expect(req.tools).toBeUndefined() + expect(req.temperature).toBeUndefined() + expect(req.max_tokens).toBeUndefined() + } + + // ── 4. Spend continuity: 3 turns × {12, 9} tokens and $0.25, all in the one pool ── + if (res.result.kind === 'winner') { + expect(res.result.spentTotal.tokens).toEqual({ input: 36, output: 27 }) + expect(res.result.spentTotal.usd).toBe(0.75) + } + }) +}) diff --git a/tests/kernel/chat-transport-executor.test.ts b/tests/kernel/chat-transport-executor.test.ts new file mode 100644 index 00000000..9e3bd395 --- /dev/null +++ b/tests/kernel/chat-transport-executor.test.ts @@ -0,0 +1,461 @@ +/** + * chatTransportExecutor — the worker that IS a conversation on a bare chat-completions transport + * (#721). All offline (injected `complete` transport, zero network). The load-bearing cases: + * + * 1. One fresh shot: system + task seed the conversation, the loop settles with the final + * assistant text as `out`, and spend is metered from the transport's OWN usage/cost fields. + * Sampling honesty: `temperature`/`maxTokens` reach the wire on EVERY request when set (one + * request class), and neither field appears when unset. + * 2. Continuity: the conversation is recorded under the session key, and a `resume` shot + * CONTINUES that exact message list (the resumed-history chain, asserted at the transport). + * 3. Resume fails loud BEFORE any spend when the store has no recorded conversation. + * 4. Metering honesty: a turn without usage marks `tokensKnown: false`; a turn without a cost + * field marks `usdKnown: false` — never a silent estimate, never a fabricated zero. + * 5. The tool table: tool_calls run on the host and fold back as `tool` messages; unknown tools + * and malformed arguments are fed back to the model, never thrown. + * 6. Transport failures throw `ValidationError` (the scope's INFRA class) — and the turns that + * DID run are still recorded, because resume-after-failure is a kernel-supported path. + * 7. `chatWorkerSeam`: profile model/prompt win, the graph's appended directive instructions + * reach the system message, the node id keys the recorded session, and a missing model + * fails loud. + */ + +import { describe, expect, it } from 'vitest' +import { ValidationError } from '../../src/errors' +import type { WorkerSpawnContext } from '../../src/mcp/tools/coordination' +import { + type ChatCompletionsTransport, + chatTransportExecutor, + chatWorkerSeam, + createChatSessionStore, +} from '../../src/runtime/supervise/chat-transport-executor' +import type { + AgentSpec, + Budget, + Executor, + ExecutorContext, + ExecutorResult, +} from '../../src/runtime/supervise/types' + +/** The seam's agents expose their `AgentSpec` (the scope contract); narrow to it. */ +function specOf(agent: unknown): AgentSpec { + return (agent as { executorSpec: AgentSpec }).executorSpec +} + +/** Build the per-spawn executor exactly as the scope would: factory(spec, ctx). */ +function buildExecutor(spec: AgentSpec, ctx: ExecutorContext): Executor { + const factory = spec.executorFactory + if (!factory) throw new Error('expected an executorFactory on the seam spec') + return factory(spec, ctx) +} + +const never = new AbortController().signal + +/** A scripted transport: replies in order (last repeats), captures every request body. */ +function scriptedTransport(replies: Array>): { + transport: ChatCompletionsTransport + requests: Array> +} { + const requests: Array> = [] + return { + requests, + transport: async (body) => { + requests.push(structuredClone(body)) + return replies[Math.min(requests.length - 1, replies.length - 1)] + }, + } +} + +const reply = (content: string, usage?: Record) => ({ + choices: [{ message: { content } }], + ...(usage !== undefined ? { usage } : {}), +}) + +describe('chatTransportExecutor — one conversation shot on a bare transport', () => { + it('settles with the final assistant text and meters usage + response cost fields', async () => { + const { transport, requests } = scriptedTransport([ + reply('the answer', { prompt_tokens: 11, completion_tokens: 7, cost: 0.002 }), + ]) + const ex = chatTransportExecutor({ + url: 'http://unused.invalid', + model: 'test/model', + system: 'Be terse.', + complete: transport, + }) + const result = await (ex.execute('what is 2+2?', never) as Promise>) + expect(result.out).toBe('the answer') + expect(result.spent).toMatchObject({ + iterations: 1, + tokens: { input: 11, output: 7 }, + usd: 0.002, + }) + // Both channels measured — neither honesty marker is set. + expect(result.spent.tokensKnown).toBeUndefined() + expect(result.spent.usdKnown).toBeUndefined() + // The wire body: system seed + the task as the user message, model verbatim, no tools field, + // and no sampling fields the caller never set (the endpoint's own defaults govern). + expect(requests).toHaveLength(1) + expect(requests[0]?.model).toBe('test/model') + expect(requests[0]?.tools).toBeUndefined() + expect(requests[0]?.temperature).toBeUndefined() + expect(requests[0]?.max_tokens).toBeUndefined() + expect(requests[0]?.messages).toEqual([ + { role: 'system', content: 'Be terse.' }, + { role: 'user', content: 'what is 2+2?' }, + ]) + expect(ex.resultArtifact().out).toBe('the answer') + }) + + it('sends the configured temperature + maxTokens as sampling fields on EVERY request', async () => { + const { transport, requests } = scriptedTransport([ + { + choices: [ + { + message: { + content: '', + tool_calls: [{ id: 'c1', function: { name: 'step', arguments: '{}' } }], + }, + }, + ], + usage: { prompt_tokens: 1, completion_tokens: 1, cost: 0 }, + }, + reply('done', { prompt_tokens: 1, completion_tokens: 1, cost: 0 }), + ]) + await (chatTransportExecutor({ + url: 'http://unused.invalid', + model: 'test/model', + complete: transport, + temperature: 0.7, + maxTokens: 2500, + tools: [ + { + spec: { type: 'function', function: { name: 'step', parameters: {} } }, + execute: async () => 'ok', + }, + ], + }).execute('go', never) as Promise) + // Both the turn-initial and the tool-follow-up completion carry the SAME pinned sampling — + // this executor has exactly one request class (the P1 parity contract). + expect(requests).toHaveLength(2) + for (const req of requests) { + expect(req.temperature).toBe(0.7) + expect(req.max_tokens).toBe(2500) + } + }) + + it('rejects a non-positive or fractional maxTokens before any transport call', () => { + const { transport, requests } = scriptedTransport([reply('never')]) + for (const maxTokens of [0, -1, 1.5]) { + expect(() => + chatTransportExecutor({ + url: 'http://unused.invalid', + model: 'test/model', + complete: transport, + maxTokens, + }), + ).toThrow(/maxTokens must be a positive integer/) + } + expect(requests).toHaveLength(0) + }) + + it('records the conversation and a resume shot continues the exact message list', async () => { + const sessions = createChatSessionStore() + const shot1 = scriptedTransport([ + reply('draft v1', { prompt_tokens: 1, completion_tokens: 1, cost: 0 }), + ]) + await (chatTransportExecutor({ + url: 'http://unused.invalid', + model: 'test/model', + system: 'Persist.', + complete: shot1.transport, + sessions, + sessionKey: 'run:s0', + }).execute('shot 1: draft it', never) as Promise) + + const shot2 = scriptedTransport([ + reply('draft v2', { prompt_tokens: 1, completion_tokens: 1, cost: 0 }), + ]) + const result = await (chatTransportExecutor({ + url: 'http://unused.invalid', + model: 'test/model', + // A resumed session keeps its recorded system message; this one must NOT re-seed. + system: 'IGNORED ON RESUME', + complete: shot2.transport, + sessions, + sessionKey: 'run:s1', + resume: { ofWorker: 'run:s0', sequence: 2 }, + }).execute('shot 2: revise it', never) as Promise>) + expect(result.out).toBe('draft v2') + // THE resumed-history chain: shot 2's request carries shot 1's whole conversation + // (system, user, assistant) plus the new user turn — same session, one message list. + expect(shot2.requests[0]?.messages).toEqual([ + { role: 'system', content: 'Persist.' }, + { role: 'user', content: 'shot 1: draft it' }, + { role: 'assistant', content: 'draft v1' }, + { role: 'user', content: 'shot 2: revise it' }, + ]) + // And shot 2's own record extends the chain under ITS key, for shot 3. + expect(sessions.load('run:s1')?.at(-1)).toEqual({ role: 'assistant', content: 'draft v2' }) + expect(sessions.load('run:s1')).toHaveLength(5) + }) + + it('a resume with no recorded conversation fails loud before any transport call', () => { + const { transport, requests } = scriptedTransport([reply('never')]) + expect(() => + chatTransportExecutor({ + url: 'http://unused.invalid', + model: 'test/model', + complete: transport, + sessions: createChatSessionStore(), + resume: { ofWorker: 'ghost:s9', sequence: 2 }, + }), + ).toThrow(/no recorded conversation for worker 'ghost:s9'/) + expect(() => + chatTransportExecutor({ + url: 'http://unused.invalid', + model: 'test/model', + complete: transport, + resume: { ofWorker: 'ghost:s9', sequence: 2 }, + }), + ).toThrow(/needs `sessions`/) + expect(requests).toHaveLength(0) + }) + + it('marks tokensKnown/usdKnown false when the transport omits usage or cost — never estimates', async () => { + const noUsage = scriptedTransport([{ choices: [{ message: { content: 'blind turn' } }] }]) + const r1 = await (chatTransportExecutor({ + url: 'http://unused.invalid', + model: 'test/model', + complete: noUsage.transport, + }).execute('t', never) as Promise>) + expect(r1.spent.tokensKnown).toBe(false) + expect(r1.spent.usdKnown).toBe(false) + expect(r1.spent.tokens).toEqual({ input: 0, output: 0 }) + expect(r1.spent.usd).toBe(0) + + // Tokens reported, cost not: tokens stay known, dollars are explicitly unknown — the priced + // model id must NOT tempt a local estimate. + const tokensOnly = scriptedTransport([ + reply('metered tokens', { prompt_tokens: 3, completion_tokens: 4 }), + ]) + const r2 = await (chatTransportExecutor({ + url: 'http://unused.invalid', + model: 'openai/gpt-4o-mini', + complete: tokensOnly.transport, + }).execute('t', never) as Promise>) + expect(r2.spent.tokens).toEqual({ input: 3, output: 4 }) + expect(r2.spent.tokensKnown).toBeUndefined() + expect(r2.spent.usd).toBe(0) + expect(r2.spent.usdKnown).toBe(false) + }) + + it('runs the tool table on the host and feeds unknown/malformed calls back to the model', async () => { + const calls: Array<{ name: string; args: Record }> = [] + const { transport, requests } = scriptedTransport([ + { + choices: [ + { + message: { + content: '', + tool_calls: [ + { id: 'c1', function: { name: 'lookup', arguments: '{"q":"x"}' } }, + { id: 'c2', function: { name: 'nope', arguments: '{}' } }, + { id: 'c3', function: { name: 'lookup', arguments: '{broken' } }, + ], + }, + }, + ], + usage: { prompt_tokens: 1, completion_tokens: 1, cost: 0 }, + }, + reply('done with tools', { prompt_tokens: 1, completion_tokens: 1, cost: 0 }), + ]) + const result = await (chatTransportExecutor({ + url: 'http://unused.invalid', + model: 'test/model', + complete: transport, + tools: [ + { + spec: { type: 'function', function: { name: 'lookup', parameters: {} } }, + execute: async (args) => { + calls.push({ name: 'lookup', args }) + return 'result:42' + }, + }, + ], + }).execute('use the tool', never) as Promise>) + expect(result.out).toBe('done with tools') + expect(result.spent.iterations).toBe(2) + expect(calls).toEqual([{ name: 'lookup', args: { q: 'x' } }]) + // Turn 2's request folds all three outcomes back as tool messages, in call order. + const turn2 = requests[1]?.messages as Array> + expect(turn2.slice(-3)).toEqual([ + { role: 'tool', tool_call_id: 'c1', content: 'result:42' }, + { role: 'tool', tool_call_id: 'c2', content: "error: unknown tool 'nope'" }, + { role: 'tool', tool_call_id: 'c3', content: 'error: tool arguments were not valid JSON' }, + ]) + expect(requests[0]?.tools).toBeDefined() + expect(requests[0]?.tool_choice).toBe('auto') + }) + + it('throws ValidationError on transport failure — and still records the turns that ran', async () => { + const sessions = createChatSessionStore() + let turn = 0 + const ex = chatTransportExecutor({ + url: 'http://unused.invalid', + model: 'test/model', + system: 'S.', + sessions, + sessionKey: 'run:s0', + tools: [ + { + spec: { type: 'function', function: { name: 'step', parameters: {} } }, + execute: async () => 'ok', + }, + ], + complete: async () => { + turn += 1 + if (turn === 1) { + return { + choices: [ + { + message: { + content: 'working', + tool_calls: [{ id: 'c1', function: { name: 'step', arguments: '{}' } }], + }, + }, + ], + usage: { prompt_tokens: 1, completion_tokens: 1, cost: 0 }, + } + } + throw new Error('socket hang up') + }, + }) + await expect(ex.execute('go', never) as Promise).rejects.toThrow( + new ValidationError('chatTransportExecutor: transport failed: socket hang up'), + ) + // The failed shot's REAL first turn is recorded, so a resume can continue the session. + const recorded = sessions.load('run:s0') + expect(recorded?.map((m) => m.role)).toEqual(['system', 'user', 'assistant', 'tool']) + }) + + it('fails loud on a malformed completion and on a second execute of one instance', async () => { + const empty = scriptedTransport([{ choices: [] }]) + await expect( + chatTransportExecutor({ + url: 'http://unused.invalid', + model: 'test/model', + complete: empty.transport, + }).execute('t', never) as Promise, + ).rejects.toThrow(/no choices\[0\]\.message/) + + const ok = scriptedTransport([ + reply('once', { prompt_tokens: 1, completion_tokens: 1, cost: 0 }), + ]) + const ex = chatTransportExecutor({ + url: 'http://unused.invalid', + model: 'test/model', + complete: ok.transport, + }) + await (ex.execute('t', never) as Promise) + await expect(ex.execute('t', never) as Promise).rejects.toThrow( + /execute\(\) called twice/, + ) + }) +}) + +describe('chatWorkerSeam — the continuity-honoring makeWorkerAgent over the executor', () => { + const budget: Budget = { maxIterations: 10, maxTokens: 10_000 } + const nodeCtx = (nodeId: string): ExecutorContext => ({ + signal: never, + node: { rootId: 'r', parentId: 'r', nodeId, attemptId: `${nodeId}:attempt:test` }, + seams: {}, + }) + const spawnContext = (over: Partial): WorkerSpawnContext => ({ + assignmentId: 'a1', + parentNodeId: 'r', + budget, + task: 't', + label: 'chat', + ...over, + }) + + it('spawns a chat worker from the profile (model + prompt + appended directive) and records under the node id', async () => { + const { transport, requests } = scriptedTransport([ + reply('hi there', { prompt_tokens: 1, completion_tokens: 1, cost: 0 }), + ]) + const sessions = createChatSessionStore() + const seam = chatWorkerSeam({ + url: 'http://unused.invalid', + model: 'seam/fallback', + complete: transport, + sessions, + temperature: 0.7, + maxTokens: 2500, + }) + const agent = seam( + { + name: 'product-agent', + model: { default: 'profile/model' }, + // The graph pins the delegates directive by APPENDING to instructions — it must reach + // the system message. + prompt: { systemPrompt: 'Sell.', instructions: ['directive: be helpful'] }, + }, + spawnContext({}), + ) + expect(agent.name).toBe('product-agent') + const ex = buildExecutor(specOf(agent), nodeCtx('run:s0')) + const result = await (ex.execute('opening message', never) as Promise>) + expect(result.out).toBe('hi there') + expect(requests[0]?.model).toBe('profile/model') + // The seam's sampling options thread through to the executor's wire body. + expect(requests[0]?.temperature).toBe(0.7) + expect(requests[0]?.max_tokens).toBe(2500) + expect(requests[0]?.messages).toEqual([ + { role: 'system', content: 'Sell.\ndirective: be helpful' }, + { role: 'user', content: 'opening message' }, + ]) + expect(sessions.load('run:s0')).toBeDefined() + }) + + it("a 'resume' spawn continues the prior node worker's recorded conversation", async () => { + const replies = [ + reply('turn 1 reply', { prompt_tokens: 1, completion_tokens: 1, cost: 0 }), + reply('turn 2 reply', { prompt_tokens: 1, completion_tokens: 1, cost: 0 }), + ] + const { transport, requests } = scriptedTransport(replies) + const seam = chatWorkerSeam({ + url: 'http://unused.invalid', + complete: transport, + }) + const profile = { name: 'agent', model: { default: 'm' }, prompt: { systemPrompt: 'S.' } } + + const first = seam(profile, spawnContext({})) + await (buildExecutor(specOf(first), nodeCtx('g:s0')).execute( + 'user turn 1', + never, + ) as Promise) + + const second = seam( + profile, + spawnContext({ continuity: 'resume', resume: { ofWorker: 'g:s0', sequence: 2 } }), + ) + await (buildExecutor(specOf(second), nodeCtx('g:s1')).execute( + 'user turn 2', + never, + ) as Promise) + + expect(requests[1]?.messages).toEqual([ + { role: 'system', content: 'S.' }, + { role: 'user', content: 'user turn 1' }, + { role: 'assistant', content: 'turn 1 reply' }, + { role: 'user', content: 'user turn 2' }, + ]) + }) + + it('fails loud when neither the profile nor the seam names a model', () => { + const seam = chatWorkerSeam({ url: 'http://unused.invalid', complete: async () => reply('x') }) + expect(() => seam({ name: 'agent', prompt: { systemPrompt: 'S.' } }, undefined)).toThrow( + /no model/, + ) + }) +})