Problem
Agent Eval owns comparison, scoring, and durable evidence, but it still has several ways to execute paid models itself. The remaining public paths accept provider URLs or credentials and issue provider HTTP from Eval:
createChatClient router, direct-provider, and cli-bridge transports plus the public LlmClient
multishot/router.ts and environment-key defaults in multishot/judges
- the paid probe in
integrity/preflight.ts
- reviewer and semantic-judge
LlmClientOptions
eval-campaign model transport configuration
Issue #537 removes the DSPy/public-benchmark exception. It does not close these other paths. Keeping them would let consumers bypass Runtime and make exact AgentProfile identity, retries, usage, cache accounting, and interruption safety optional.
Completion criteria
- Keep Eval execution-neutral: it accepts caller-owned
ChatClient or the existing typed model callback; it does not import Runtime and does not grow a second execution abstraction.
- Remove public router, direct-provider, and cli-bridge variants from
createChatClient. Retain only caller-owned/custom and mock forms required by Eval behavior and tests.
- Remove the public paid-network surface of
LlmClient. If request/response parsing is still needed for Eval's authenticated loopback proxy, keep only the smallest internal protocol helper and prove it cannot target a non-loopback endpoint.
- Make multishot and its judges require a caller-supplied transport. Delete environment API-key discovery and the built-in Router fetch.
- Replace the paid model probe with a caller-owned callback that returns the observed model and receipt. Provider catalog membership checks must likewise take a caller-owned request function rather than credentials.
- Make reviewer, semantic judge, and eval-campaign consume a supplied
ChatClient; remove provider URL/key configuration and default provider URLs.
- Preserve model identity, timeout, cancellation, stable per-call ID, retry count, input/output/reasoning/cache usage, billed or unknown USD, and finite execution evidence through the caller boundary.
- Do not add a deprecated alias, legacy mode, environment fallback, compatibility overload, or tests that preserve removed behavior.
- Add a source check that rejects provider credentials, default provider URLs, or outbound model endpoint requests in Eval production source. Allow the authenticated local proxy used to communicate with external optimizer processes and non-model product/telemetry HTTP clients.
- Update every in-repo caller and coordinate the matching Runtime adapter so all published examples and tests use the caller-owned path.
- Prove TypeScript, Python, package-export, official GEPA/SkillOpt, DSPy, multishot, reviewer, campaign, and packed-consumer checks.
Non-goals
Eval may still call ordinary product, telemetry, and local loopback endpoints that are not model execution. The change is specifically about provider model requests and provider credentials.
Problem
Agent Eval owns comparison, scoring, and durable evidence, but it still has several ways to execute paid models itself. The remaining public paths accept provider URLs or credentials and issue provider HTTP from Eval:
createChatClientrouter, direct-provider, and cli-bridge transports plus the publicLlmClientmultishot/router.tsand environment-key defaults in multishot/judgesintegrity/preflight.tsLlmClientOptionseval-campaignmodel transport configurationIssue #537 removes the DSPy/public-benchmark exception. It does not close these other paths. Keeping them would let consumers bypass Runtime and make exact
AgentProfileidentity, retries, usage, cache accounting, and interruption safety optional.Completion criteria
ChatClientor the existing typed model callback; it does not import Runtime and does not grow a second execution abstraction.createChatClient. Retain only caller-owned/custom and mock forms required by Eval behavior and tests.LlmClient. If request/response parsing is still needed for Eval's authenticated loopback proxy, keep only the smallest internal protocol helper and prove it cannot target a non-loopback endpoint.ChatClient; remove provider URL/key configuration and default provider URLs.Non-goals
Eval may still call ordinary product, telemetry, and local loopback endpoints that are not model execution. The change is specifically about provider model requests and provider credentials.