Add portable end-to-end evals for the Workshop agent

[AshishKumar4]

This adds an eval suite for the Workshop agent. An eval starts a fresh Workshop, sends a user request to the production agent, waits for the agent to finish, and calls the generated Gadget's RPC to see whether the app works. The same eval can run in local workerd or against an Access-protected preview.

A new account gets its model preference and onboarding state before its workspace opens. I also ran appointment-desk against this PR's preview, which covered Access signup, onboarding, the deployed agent, Gadget execution, and the RPC checks. GLM 5.2 passed four of five checks. One unique booking was rejected as DUPLICATE_BOOKING instead of SLOT_FULL, so the trial failed with a score of 0.8.

How an eval runs

An eval is a list of user requests and checks. We call each user request a turn. All turns in one eval use the same chat and workspace. The two-turn ledger eval works like this:

1. Ask the agent to build an expense ledger.
2. Check the ledger's RPC.
3. Ask the same agent to add budgets.
4. Check the budget feature and make sure the original ledger still works.

The agent decides how many model responses and tool calls it needs for each turn. The report records those numbers; the eval does not prescribe them.

A trial is one complete run of every turn and check in an eval with one model. WORKSHOP_EVAL_TRIALS=10 runs each eval-and-model pair ten times. With three evals and two models, that produces 60 trials.

Evals included in this PR

Eval	User requests	What it checks
appointment-desk	1	Capacity, cancellation, error codes, simultaneous bookings, and duplicate booking IDs. Five checks.
expense-ledger	2	Exact-cent splits, balances, settlement, validation, monthly totals, budgets, and preserving existing data and methods after the second request. Nine checks.
project-doc	1	A real standard Doc, useful initial content through the Docs RPC, and a full-document rewrite. Three checks.

Every eval gates the run by default. If any check fails, Vitest exits nonzero and prints the failed check names. The JSON report and report UI include the evidence returned by each check.

Writing an eval

You can define an eval by adding packages/workshop-evals/evals/<name>.eval.ts.

The task API is:

type EvalTask = {
  id: string;
  turns: readonly [EvalTurn, ...EvalTurn[]];
};

type EvalTurn = {
  prompt: string;
  verify(verifier: EvalVerifier): Promise<void>;
};

type EvalCheckOutcome = {
  pass: boolean;
  evidence?: unknown;
};

Example:

import { z } from "zod";
import { defineTaskEval } from "../src/eval.js";
import { defineEvalTask } from "../src/task.js";

interface CounterApi {
  increment(): Promise<unknown>;
}

const CounterResult = z.object({ value: z.number().int() });

const task = defineEvalTask({
  id: "counter",
  turns: [{
    prompt: `Build a Gadget named exactly "Counter" with an increment() RPC.`,
    verify: async verifier => {
      await verifier.check("increments", async () => {
        using api = await verifier.connect<CounterApi>("Counter");
        const result = CounterResult.parse(await api.increment());
        return { pass: result.value === 1, evidence: result };
      });
    },
  }],
});

defineTaskEval(task);

verifier.connect<Api>(title) opens the generated Gadget's provisional RPC. verifier.check(id, fn) records one score and keeps running the other checks if that one throws. verifier.workpieces exposes the generated workpieces when a check needs to inspect their type or output presentation.

Checks should cover behavior a user can observe. They should not require a particular tool sequence, storage layout, or synchronization technique. Parse RPC responses before scoring them. In a multi-turn eval, check earlier behavior again after each follow-up request that could break it.

Running the suite

# Defaults: every eval, both configured models, one trial
pnpm evals

# Repeat each eval-and-model pair ten times
WORKSHOP_EVAL_TRIALS=10 pnpm evals

# Choose models
WORKSHOP_EVAL_MODELS='@cf/zai-org/glm-5.2' pnpm evals

# Report scores without failing the process
WORKSHOP_EVAL_GATING_TASKS='' pnpm evals

# Gate only selected evals
WORKSHOP_EVAL_GATING_TASKS='appointment-desk,expense-ledger' pnpm evals

The manual Workshop evals workflow exposes the model list, trial count, and gating task list through workflow_dispatch. It does not run on pushes, pull requests, merges, or a schedule.

Each trial reports the fraction of checks that passed, total and per-turn duration, model responses, tool calls and tool errors, agent and provider errors, tokens and cost when available, model, target, and trial number. It also records the runner commit, deployed target commit, and taskVersion, a SHA-256 hash of the prompts. There is no LLM judge in this PR.

Local results are written to packages/workshop-evals/.wrangler/evals/results.json. Run pnpm evals:ui to inspect the transcript, scores, tool calls, errors, and evidence.

[github-actions]

Preview: pr319-evals-suite

https://pr319-evals-suite-router.cloudflare-os-previews.workers.dev

Dashboard · deleted when this PR closes

[Maximo-Guk]

Initial GPT pass:

1. P1 Gating failures leave the workflow green. .github/workflows/workshop-evals.yml:48-50,93-99 tolerates shard failures, while the pinned reporter soft-fails when publishing a Check Run. Set soft-fail: false.
2. P1 Access tokens can be sent over plaintext WebSockets. packages/integration-tests/src/rpc-client.ts:59-67 sends CF_Authorization over ws://, and packages/workshop-evals/src/target.ts:44-51 permits HTTP preview URLs. Require HTTPS whenever an Access token is provided.
3. P1 Agent turn timeouts do not bound the initiating RPC. packages/integration-tests/src/agent-session.ts:157-172 waits for newChat()/sendChatMessage() before observing the timeout promise. A stalled RPC can therefore hang indefinitely despite the advertised hard limit.
4. P1 Failed preview turns cannot delete their workspace. agent-session.ts:187-189,217-220,258-260 marks the session unusable after an error, then rejects deleteWorkspace(). The disposer in packages/workshop-evals/src/target.ts:108-113 consequently leaks the preview workspace.
5. P1 Gadget verification has no deadline. packages/workshop-evals/src/harness.ts:35-40 and src/verifier.ts:69-82 allow a generated RPC to remain pending beyond Vitest’s timeout. The async disposal scope never unwinds, leaking the session/workspace.
6. P2 The final ledger regression check passes vacuously. packages/workshop-evals/evals/expense-ledger.eval.ts:343-363 accepts empty balances() and settlement() arrays, allowing broken turn-one methods to receive a perfect score.
7. P2 Reported total tokens contain only the last model step. packages/workshop-evals/src/harness.ts:31-54 overwrites usage each turn and publishes AiChatMetadata.totalTokens, which explicitly represents only the conversation’s last step.
8. P2 The 500 ms idle debounce can finish before a callback turn starts. packages/integration-tests/src/agent-session-internals.ts:71-77 treats temporary inactivity as completion, but callback continuation performs asynchronous context loading before restoring activeAgent.

[Maximo-Guk]

Second GPT pass on efficiency:

1. High: Failed preview trials cannot delete their workspace. run() marks the session failed, while cleanup calls deleteWorkspace(), which rejects failed sessions. This leaks workspaces and may mask the original failure. packages/integration-tests/src/agent-session.ts:187-220, packages/workshop-evals/src/target.ts:108-113.
2. High: Adding .concurrent now is unsafe and inefficient. Every trial starts another full Wrangler/workerd harness, whose custom build writes shared generated files. Share one harness per file/shard, then create a fresh AgentSession per trial. packages/workshop-evals/src/target.ts:139-147, packages/workshop-backend/wrangler.jsonc:4-7.
3. Medium: These are not hermetic like the integration tests. The integration suite installs NetworkInterceptor and rejects all unexpected egress; evals allow unrestricted outbound traffic, including agent webFetch calls. Permit only required AI Gateway/inference and cost endpoints, rejecting everything else. Preview evals cannot be made hermetic this way. packages/integration-tests/src/network-interceptor.ts:1-6, packages/workshop-evals/src/target.ts:118-139.
4. Medium: The cleanup timeout reserve is not enforced. Only agent turns are timed; setup and verifier RPCs can consume the outer Vitest timeout and prevent cleanup. Use one absolute trial deadline and propagate its remaining time/signal into turns and verification. packages/workshop-evals/src/harness.ts:18-40, packages/workshop-evals/src/config.ts:9-14.
5. Medium: Scoring does not exactly match the prompts. DUPLICATE_SLOT is required but untested; the expense UI requirements are untested; the Doc structure check only verifies word presence. Conversely, the full Doc rewrite tests platform behavior not explicitly requested. appointment-desk.eval.ts:56-58, expense-ledger.eval.ts:80-86, project-doc.eval.ts:61-93.
6. Low: Every session waits for output formats although only project-doc requires them. Gateway mode also performs an unmeasured title-generation inference per trial. Make format readiness task-specific and disable/mock auxiliary title generation. packages/integration-tests/src/agent-session.ts:133-135, packages/workshop-backend/src/overseer.ts:5238-5242.

[ndisidore]

this seems like like it could leak the token - previews can be http
probably want to enforce https when this is present