Skip to content

refactor(execution): remove Eval-owned paid model transports #539

Description

@drewstone

Problem

Agent Eval owns comparison, scoring, and durable evidence, but it still has several ways to execute paid models itself. The remaining public paths accept provider URLs or credentials and issue provider HTTP from Eval:

  • createChatClient router, direct-provider, and cli-bridge transports plus the public LlmClient
  • multishot/router.ts and environment-key defaults in multishot/judges
  • the paid probe in integrity/preflight.ts
  • reviewer and semantic-judge LlmClientOptions
  • eval-campaign model transport configuration

Issue #537 removes the DSPy/public-benchmark exception. It does not close these other paths. Keeping them would let consumers bypass Runtime and make exact AgentProfile identity, retries, usage, cache accounting, and interruption safety optional.

Completion criteria

  • Keep Eval execution-neutral: it accepts caller-owned ChatClient or the existing typed model callback; it does not import Runtime and does not grow a second execution abstraction.
  • Remove public router, direct-provider, and cli-bridge variants from createChatClient. Retain only caller-owned/custom and mock forms required by Eval behavior and tests.
  • Remove the public paid-network surface of LlmClient. If request/response parsing is still needed for Eval's authenticated loopback proxy, keep only the smallest internal protocol helper and prove it cannot target a non-loopback endpoint.
  • Make multishot and its judges require a caller-supplied transport. Delete environment API-key discovery and the built-in Router fetch.
  • Replace the paid model probe with a caller-owned callback that returns the observed model and receipt. Provider catalog membership checks must likewise take a caller-owned request function rather than credentials.
  • Make reviewer, semantic judge, and eval-campaign consume a supplied ChatClient; remove provider URL/key configuration and default provider URLs.
  • Preserve model identity, timeout, cancellation, stable per-call ID, retry count, input/output/reasoning/cache usage, billed or unknown USD, and finite execution evidence through the caller boundary.
  • Do not add a deprecated alias, legacy mode, environment fallback, compatibility overload, or tests that preserve removed behavior.
  • Add a source check that rejects provider credentials, default provider URLs, or outbound model endpoint requests in Eval production source. Allow the authenticated local proxy used to communicate with external optimizer processes and non-model product/telemetry HTTP clients.
  • Update every in-repo caller and coordinate the matching Runtime adapter so all published examples and tests use the caller-owned path.
  • Prove TypeScript, Python, package-export, official GEPA/SkillOpt, DSPy, multishot, reviewer, campaign, and packed-consumer checks.

Non-goals

Eval may still call ordinary product, telemetry, and local loopback endpoints that are not model execution. The change is specifically about provider model requests and provider credentials.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions