fix: recover unclassified Responses output items

[bjoerndanz]

TL;DR

Recover Responses output items the SDK cannot classify, so an agent loop no longer replays an UNKNOWN wrapper as an input item and kill the sample on the next request.

What changed?

• recoverOutputItems rebuilds an unclassified output item from its raw payload, adds only the envelope fields the SDK requires, and revalidates the result through outputItemsFromJSON.
• A missing item id is filled with a deterministic synthetic value derived from the response id and the item index, so a repeated turn produces the same id. Role, status, content, arguments and call ids are never altered.
• consumeStream returns recovered items in output and derives text from them, so every downstream consumer and the conversation the agent loop accumulates see typed items only.
• An item that still cannot be recovered is dropped and logged with its raw type, the response id and its index, instead of being appended and failing validation later.
• The terminal-event fallback added in fix: recover unclassified Responses terminal events #28 now logs its unrecoverable branch the same way, which was flagged in review of that change.

Why?

An upstream that omits required response-envelope fields also omits them on individual output items. The SDK is forward compatible, so an output message missing id comes back as { type: "UNKNOWN", is_unknown: true, raw: { ... } } rather than raising.

The agent loop appends whatever a turn returned into the conversation without restricting item types, so that wrapper is sent as an input item on the following turn. UNKNOWN matches none of the 44 accepted input item types, so the SDK rejects the outbound request before any HTTP call with Input validation failed, and the sample is recorded as a model error. The generation that produced the item completed and was billed.

Dropping such items instead of recovering them would silently remove an assistant turn or a tool call from the transcript, which corrupts the conversation rather than fixing it, so recovery is attempted first and only genuine failures are dropped and logged.

How to test

Drive a multi-turn tool-bearing exchange against an upstream whose Responses output message items omit id. Before this change the third request fails with Input validation failed naming an input item deep in the accumulated conversation. After it, the exchange continues, the recovered item carries a synthetic-<response id>-<index> id, and the upstream accepts the replayed item.

Benchmark impact

Agentic benchmarks against such an upstream lose whole samples today, which suppresses the affected arm's completion rate without affecting the model's actual output. Recovering the items restores those samples. No dataset, solver or scorer logic changes, and behavior against an upstream that emits complete items is unchanged, since recovery only runs for items the SDK failed to classify.

Reviewer focus

• Whether a synthetic id is acceptable to place on a replayed item, and whether deriving it from the response id and index is the right stability guarantee.
• The decision to drop and log an unrecoverable item rather than fail the turn.

Checklist

• Tests cover changed behavior
• Public API or configuration changes are backward compatible, or the break is documented
• Benchmark changes document dataset provenance and licensing
• No credentials, private results, or restricted dataset contents are included
• Documentation is updated where needed

Link to Devin session: https://openrouter.devinenterprise.com/sessions/014dc0a17f314a0fb5a8741be8e66fba
Requested by: @bjoerndanz

---

[devin-ai-integration]

Original prompt from bjoern.danz

I have a suspicion that switchyard's llm client when configured with format: openai_chat does not forward/retain reasoning traces. https://github.com/NVIDIA-NeMo/Switchyard Can you investigate that suspicion?

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR that start with 'DevinAI' or '@devin'.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[devin-ai-integration]

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 3 additional findings.