N01: incremental structured-output streaming parser

[sethkarten]

N01 — incremental structured output

Status: Draft. Source reviewed; final remote CPU validation is currently running. Not merge-ready.

Scope / feature

Implements incremental parsing of streamed structured tool arguments across AI providers, with streaming JSON state/parity coverage and a bounded CPU benchmark corpus.

Contracts

• Preserves streaming partial-JSON and cleanup parity across supported provider paths.
• Keeps benchmark corpus inputs bounded and uses the exact UTF-8 byte corpus.
• Includes location-independent CLI smoke coverage and parser-state tests.

No limiter

This change adds no client-side rate limiter, shared semaphore, admission queue, or synthetic local 429 behavior. Independent model requests remain independent.

Rollback

Revert this PR's commit range (or the feature commits) to return to B00B behavior; the change is isolated to the AI streaming/parser paths and their tests/benchmarks.

Evidence retained

The source branch retains review fixes and validation artifacts in commit history and test/benchmark fixtures, including evidence from known failed attempts and their subsequent corrections. Final remote CPU validation is currently running and is not represented as complete.

Exact provenance

• Base: B00B PR #1115 head branch perf/b00b-production-gate at 9d9cf28d51490ef06efba738c3fff788463acdff (not main)
• Source branch: perf/n01-incremental-structured-output
• Exact reviewed and published source head: 5835416078b7440cbf479c007e43b63d4ae55416

Note

Add incremental structured-output streaming parser to replace repeated JSON reparsing

• Introduces createStreamingJsonParseState in utils/json-parse.ts, an incremental JSON parser that processes tool call argument deltas without replaying the full prefix on each chunk.
• Replaces partialJson/partialArgs scratch buffers and repeated parseStreamingJson calls across all providers (Anthropic, Bedrock, Mistral, OpenAI completions/responses/Codex/Azure) with the new stateful parser's append/finalize lifecycle.
• Adds discardStreamingJsonParseState for cleanup on error/abort paths in all providers, and getStreamingJsonRawForProviderCheck for prefix-continuation validation in the OpenAI Responses shared stream.
• Updates CompactAssistantStreamReconstructor in compact-session-stream.ts to use per-tool parser instances and reject raw suffixes after a producer snapshot is applied.
• Includes a CPU benchmark (streaming-json-parse-cpu-bench.ts) comparing legacy vs. incremental parsers, plus unit tests for parity and strict-validation counts.
• Behavioral Change: missing parser during a tool call delta now throws instead of silently continuing; OpenAI Responses stream throws if an authoritative done payload replaces rather than extends the streamed prefix.

📊 Macroscope summarized e8b407f. 11 files reviewed, 0 issues evaluated, 0 issues filtered, 0 comments posted

🗂️ Filtered Issues

No issues evaluated.

[macroscopeapp]

🟡 Medium utils/json-parse.ts:314

On malformed-provider input like ["a\q",1], the root-close deferred repair loop writes the repaired string to the original array index even though removeValue already popped that slot and the following element shifted into it. The preview returns ["a\\q"] (losing 1) instead of the ["a\\q",1] that legacy repairJson parsing produces. The same stale-location mechanism can also overwrite a later duplicate object key. Consider capturing a stable insertion point (or re-appending popped elements) when deferring the repair so the restored value lands at the correct position.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @packages/ai/src/utils/json-parse.ts around line 314:

On malformed-provider input like `["a\q",1]`, the root-close deferred repair loop writes the repaired string to the original array index even though `removeValue` already popped that slot and the following element shifted into it. The preview returns `["a\\q"]` (losing `1`) instead of the `["a\\q",1]` that legacy `repairJson` parsing produces. The same stale-location mechanism can also overwrite a later duplicate object key. Consider capturing a stable insertion point (or re-appending popped elements) when deferring the repair so the restored value lands at the correct position.

[macroscopeapp]

🟡 Medium utils/json-parse.ts:418

When a string value ends with U+2028 or U+2029 (e.g. {"a":"x\u2028"} as complete JSON), preview() returns {a: "x"} with the separator missing, even after the full document has been streamed. The stale value can surface in the final streaming update before finalize() corrects it. In consumeString, the U+2028/U+2029 branch appends the character to token.value but skips updateStringPreview, and the closing-quote path does not flush the accumulated token value either. Call this.updateStringPreview(token) in that branch so the preview reflects the complete string.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @packages/ai/src/utils/json-parse.ts around line 418:

When a string value ends with `U+2028` or `U+2029` (e.g. `{"a":"x\u2028"}` as complete JSON), `preview()` returns `{a: "x"}` with the separator missing, even after the full document has been streamed. The stale value can surface in the final streaming update before `finalize()` corrects it. In `consumeString`, the `U+2028`/`U+2029` branch appends the character to `token.value` but skips `updateStringPreview`, and the closing-quote path does not flush the accumulated token value either. Call `this.updateStringPreview(token)` in that branch so the preview reflects the complete string.

[macroscopeapp]

🟠 High providers/mistral.ts:453

finalize() calls JSON.parse on the accumulated argument string, so it throws when a tool-call ends with an empty function.arguments string — a valid zero-argument tool call. This converts an otherwise completed streaming response into an error instead of emitting toolcall_end/done. The previous parseStreamingJson tolerated empty and incomplete argument strings by returning the best partial object. Consider making finalize() fall back to {} (or return the last partial parse) when the accumulated text is empty or incomplete rather than strictly invoking JSON.parse.

Also found in 1 other location(s)

packages/ai/src/providers/openai-completions.ts:203

finishBlock now calls strict block.parser.finalize(), whose implementation uses JSON.parse(raw). The replaced parseStreamingJson(block.partialArgs) tolerated empty, incomplete, and repairable malformed streamed arguments by returning the best partial object. If an OpenAI-compatible endpoint ends a tool call with arguments such as an empty string, an incomplete object, or a raw control/invalid escape, finalization now throws and the outer catch converts the entire otherwise completed response into an error event instead of emitting toolcall_end/done.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @packages/ai/src/providers/mistral.ts around line 453:

`finalize()` calls `JSON.parse` on the accumulated argument string, so it throws when a tool-call ends with an empty `function.arguments` string — a valid zero-argument tool call. This converts an otherwise completed streaming response into an error instead of emitting `toolcall_end`/`done`. The previous `parseStreamingJson` tolerated empty and incomplete argument strings by returning the best partial object. Consider making `finalize()` fall back to `{}` (or return the last partial parse) when the accumulated text is empty or incomplete rather than strictly invoking `JSON.parse`.

Also found in 1 other location(s):
- packages/ai/src/providers/openai-completions.ts:203 -- `finishBlock` now calls strict `block.parser.finalize()`, whose implementation uses `JSON.parse(raw)`. The replaced `parseStreamingJson(block.partialArgs)` tolerated empty, incomplete, and repairable malformed streamed arguments by returning the best partial object. If an OpenAI-compatible endpoint ends a tool call with arguments such as an empty string, an incomplete object, or a raw control/invalid escape, finalization now throws and the outer catch converts the entire otherwise completed response into an `error` event instead of emitting `toolcall_end`/`done`.

[macroscopeapp]

🟡 Medium daemon/compact-session-stream.ts:158

When seed() is called with an assistant message containing a tool call, it records that content index in toolCallSnapshots. If a later assistant message reuses the same content index, toolcall_start creates a fresh parser but leaves the stale snapshot entry in place, so every toolcall_delta with raw text returns undefined at the snapshot guard. This prevents reconstruction of the new tool call and forces an unnecessary resync. Clear the snapshot key in toolcall_start (or when the partial message is replaced) so stale entries don't block new tool calls.

Suggested change

	partial.content[event.contentIndex] = delta.contentStart;
	this.toolCallJson.set(this.toolCallKey(delta.activeSessionId, event.contentIndex), "");
	this.toolCallParsers.set(
	this.toolCallKey(delta.activeSessionId, event.contentIndex),
	createStreamingJsonParseState(),
	);
	partial.content[event.contentIndex] = delta.contentStart;
	const key = this.toolCallKey(delta.activeSessionId, event.contentIndex);
	this.toolCallSnapshots.delete(key);
	this.toolCallParsers.set(key, createStreamingJsonParseState());

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @packages/coding-agent/src/modes/daemon/compact-session-stream.ts around lines 158-162:

When `seed()` is called with an assistant message containing a tool call, it records that content index in `toolCallSnapshots`. If a later assistant message reuses the same content index, `toolcall_start` creates a fresh parser but leaves the stale snapshot entry in place, so every `toolcall_delta` with raw text returns `undefined` at the snapshot guard. This prevents reconstruction of the new tool call and forces an unnecessary resync. Clear the snapshot key in `toolcall_start` (or when the partial message is replaced) so stale entries don't block new tool calls.

[macroscopeapp]

🟠 High utils/json-parse.ts:447

beginValue assigns object members with (frame.value as Record<string, unknown>)[frame.key] = value. When the JSON key is __proto__, this invokes Object.prototype.__proto__ and mutates the prototype chain instead of creating the own data property that JSON.parse creates. A stream such as {"__proto__":{"admin":true}} therefore yields a preview whose inherited admin property is visible to consumers, with no own __proto__ property — breaking parity with JSON.parse and potentially leaking prototype-pollution state to downstream code. The same issue affects replaceCurrentValue and setValue. Use Object.defineProperty (or a null-prototype object) so __proto__ is stored as an own data property.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @packages/ai/src/utils/json-parse.ts around line 447:

`beginValue` assigns object members with `(frame.value as Record<string, unknown>)[frame.key] = value`. When the JSON key is `__proto__`, this invokes `Object.prototype.__proto__` and mutates the prototype chain instead of creating the own data property that `JSON.parse` creates. A stream such as `{"__proto__":{"admin":true}}` therefore yields a preview whose inherited `admin` property is visible to consumers, with no own `__proto__` property — breaking parity with `JSON.parse` and potentially leaking prototype-pollution state to downstream code. The same issue affects `replaceCurrentValue` and `setValue`. Use `Object.defineProperty` (or a null-prototype object) so `__proto__` is stored as an own data property.