v0.4.0 — batch + scale (SPEC-4)

README.md

@@ -84,6 +84,8 @@ model should have `"input": ["text", "image"]`. Other `/vision` subcommands:
 | `/vision paste-mode [hint\|auto\|off]` | Set how pasted images are handled on a text-only primary (no arg → cycle). |
 | `/vision marker-style [code\|bold\|plain]` | Set the markdown style for `[Image-#N]` markers (no arg → show current). |
 | `/vision auto-prompt [<text>\|clear]` | Set/clear the generic auto-delegation prompt (no arg → multi-line editor). |
+| `/vision preview <path>` | Open a full-screen TUI preview of an image (Kitty/iTerm2 graphics, text fallback on tmux). |
+| `/vision batch-concurrency [<1-20>]` | Max parallel image delegations in a batch (`describe_image image_paths` + paste auto mode). 1 = serial; 20 = aggressive. Default 5. |
 
 Config is stored at `~/.pi/agent/vision.json` (not `vision-tool.json`, so it
 doesn't collide with the community package during transition).
@@ -159,6 +161,66 @@ The text fallback still shows useful metadata (filename, dimensions, format,
 file size) and confirms the image was found. Real graphics require running pi
 outside tmux on a graphics-capable terminal.
 
+## Batch + scale (v0.4.0)
+
+`describe_image` accepts **multiple images** in a single call via
+`image_paths` (alongside the single `image_path` for back-compat). For
+text-only primaries (where the tool is visible), this lets the model analyze,
+compare, or cross-reference several images in one tool call instead of N
+serial round-trips:
+
+```
+describe_image(
+  image_paths: ["/tmp/before.png", "/tmp/after.png", "/tmp/diff.png"],
+  prompt: "Compare these three screenshots. What changed?"
+)
+```
+
+Delegations run **in parallel**, bounded by `batchConcurrency` (default 5,
+configurable 1–20 via `/vision batch-concurrency`). `1` = serial escape
+hatch; `20` = aggressive (rate-limit risk is yours). Each image reuses the
+v0.2.x resilience pipeline (cache/retry/fallback) independently — a cache hit
+returns 0 vision-model calls, a failed image becomes an `[error: …]` section
+rather than failing the whole batch, and `isError` is set only if **every**
+image failed. The result is one structured, order-stable text block:
+
+```
+[Batch: 3 image(s)]
+
+[Image 1] /tmp/before.png
+<vision model description>
+
+[Image 2] (cached) /tmp/after.png
+<vision model description>
+
+[Image 3] /tmp/diff.png
+[error: not_found — image not found at /tmp/diff.png]
+```
+
+A hard cap of **50 images** per call (`MAX_BATCH_IMAGES`) defends against an
+over-eager model passing an absurd array; split across calls if you need more.
+
+**Parallel auto-delegation (paste auto mode).** When `textOnlyPasteMode` is
+`"auto"` and you paste multiple image paths, delegations now run in parallel
+(one batch-level timeout = the total budget, bounded by `batchConcurrency`)
+instead of serially — so a 5-screenshot paste completes in ~`ceil(N/c)` ×
+per-call instead of `N` × per-call.
+
+**Hint mode now exposes paths.** In text-only + `"hint"` mode (the default),
+the hint line now **lists the image paths** and names the `image_paths` batch
+affordance, so the model can actually invoke `describe_image` (previously the
+hint named the tool but the path markers erased the paths, leaving the model
+unable to call it). Paste 2+ images and the model learns it can pass them all
+to `image_paths` for batch analysis.
+
+**Clipboard paste just works.** Pi binds `ctrl+v` (`alt+v` on Windows) to
+paste the system clipboard image: it reads the clipboard, writes the bytes to
+`/tmp/pi-clipboard-<uuid>.<ext>`, and inserts that path at the cursor. Our
+existing path-token pipeline detects it, renders a `[Image-#N]` marker, and
+attaches (multimodal) or delegates (text-only) — no separate clipboard code
+path needed. Multi-image clipboard = N `ctrl+v` presses = N paths = handled
+as a batch.
+
 ## How it works
 
 Two mechanisms combine to guarantee the behavior:
@@ -181,24 +243,35 @@ Two mechanisms combine to guarantee the behavior:
 
 ## Using `describe_image`
 
-The tool accepts a file path, data URL, or raw base64:
+The tool accepts a file path, data URL, or raw base64. For a **single image**:
 
 ```
 describe_image(image_path: "/tmp/screenshot.png", prompt: "What's in this image?")
 ```
 
+For **multiple images** (batch — parallel delegation, one structured result):
+
+```
+describe_image(
+  image_paths: ["/tmp/a.png", "/tmp/b.png", "/tmp/c.png"],
+  prompt: "Compare these screenshots. What changed between them?"
+)
+```
+
 Parameters:
 
 | Param | Type | Description |
 |---|---|---|
-| `image_path` | string | File path, `data:` URL, or raw base64 |
-| `prompt` | string | What to analyze or answer about the image |
-| `compress` | boolean? | Optimize the image before delegation (default `true`) |
+| `image_path` | string? | Path to a single image, `data:` URL, or raw base64. Use for one image. |
+| `image_paths` | string[]? | Multiple paths to analyze together (comparison/cross-reference). Up to 50. |
+| `prompt` | string | What to analyze or answer about the image(s). For a batch, applies to each; describe what to compare. |
+| `compress` | boolean? | Optimize the image(s) before delegation (default `true`) |
 | `reasoning` | enum? | Reasoning effort for the delegation (`off`…`xhigh`) |
 
 When caching or fallback is active, the tool result `details` include
 `cached: true` (cache hit) and `fallback: true` (result from the fallback
-model) for traceability.
+model) for traceability. For a batch, `details.batch` is an array of per-image
+results (index, path, ok, cached, fallback, errorCode) in input order.
 
 For multimodal primaries you don't call `describe_image` — just reference the
 image path in your message and the model sees it natively.

extensions/paste.ts

@@ -36,7 +36,7 @@ import { getSharedConfig, getSharedCache } from "../lib/state.ts";
 import { delegateToVisionModel, type DelegateParams } from "../lib/delegate.ts";
 import type { ReasoningLevel } from "../lib/config.ts";
 import { createComposePreviewComponent, makePreviewImage } from "../lib/preview.ts";
-import { clearSharedState } from "../lib/state.ts";
+import { mapWithConcurrency } from "../lib/batch.ts";
 
 const IMAGE_EXT_RE = /\.(png|jpe?g|gif|webp|bmp)$/i;
 // Matches path-like tokens (absolute /…, home ~/…, relative ./…/…/) ending in a
@@ -175,31 +175,31 @@ function buildResolvedMap(
 }
 
 /** Auto-delegate a single image with timeout protection. Returns the
- *  description text or undefined on failure/timeout (caller falls back to hint). */
+ *  description text or undefined on failure/timeout (caller falls back to hint).
+ *  v0.4.0: takes the shared batch `signal` (owned by the caller) instead of
+ *  creating its own AbortController — the batch owns one timeout for all
+ *  images, bounding the wall-clock of the pre-send input hook. */
 async function autoDelegateOne(
   ctx: ExtensionContext,
   config: NonNullable<ReturnType<typeof getSharedConfig>>,
   image: LoadedImage,
   cache: NonNullable<ReturnType<typeof getSharedCache>>,
+  signal: AbortSignal,
 ): Promise<{ text: string; cached: boolean } | undefined> {
-  const controller = new AbortController();
-  const timer = setTimeout(() => controller.abort(), config.autoDelegateTimeoutMs);
   try {
     const params: DelegateParams = {
       image_path: image.abs,
       prompt: config.autoDelegatePrompt,
       compress: true,
       reasoning: "off" as ReasoningLevel,
     };
-    const result = await delegateToVisionModel(ctx, config, params, controller.signal, cache);
+    const result = await delegateToVisionModel(ctx, config, params, signal, cache);
     if (result.ok) {
       return { text: result.text, cached: result.details.cached };
     }
     return undefined; // failure → caller falls back to hint
   } catch {
     return undefined; // timeout or error → hint fallback
-  } finally {
-    clearTimeout(timer);
   }
 }
 
@@ -363,37 +363,52 @@ export default function pasteExtension(_pi: ExtensionAPI): void {
 
     if (mode === "hint") {
       // Markers + hint line nudging the model to call describe_image.
-      text = `${text}\n${buildHintLine(loaded.length)}`;
+      text = `${text}\n${buildHintLine(loaded.map((l, i) => ({ token: l.token, index: resolved.get(l.token)?.index ?? i })))}`;
       return { action: "transform" as const, text };
     }
 
-    // mode === "auto": auto-delegate each image + append descriptions.
+    // mode === "auto": auto-delegate each image in PARALLEL (v0.4.0 SPEC-4 §3.2)
+    // with bounded concurrency + ONE batch-level AbortController (the budget
+    // bounds the pre-send wall-clock, not per-image). Reuses the v0.2.x delegate
+    // pipeline (cache/retry/fallback) per image. Falls back to hint on timeout/
+    // failure (all-fail → hint; per-image fail → that image gets no description).
     const cache = getSharedCache();
     const visionModel = config.provider && config.model ? `${config.provider}/${config.model}` : "(unconfigured)";
+    const hintImages = loaded.map((l, i) => ({ token: l.token, index: resolved.get(l.token)?.index ?? i }));
 
     if (!cache || !config.provider || !config.model) {
       // Can't delegate (no cache or unconfigured) → fall back to hint.
-      text = `${text}\n${buildHintLine(loaded.length)}`;
+      text = `${text}\n${buildHintLine(hintImages)}`;
       return { action: "transform" as const, text };
     }
 
+    const controller = new AbortController();
+    const timer = setTimeout(() => controller.abort(), config.autoDelegateTimeoutMs);
+    let results: Array<{ text: string; cached: boolean } | undefined>;
+    try {
+      results = await mapWithConcurrency(
+        loaded,
+        config.batchConcurrency,
+        (image) => autoDelegateOne(ctx, config, image, cache, controller.signal),
+      );
+    } finally {
+      clearTimeout(timer);
+    }
+
     const descriptions: Array<{ token: string; index: number; text: string; cached: boolean }> = [];
-    let allFailed = true;
-
-    for (const image of loaded) {
-      const result = await autoDelegateOne(ctx, config, image, cache);
-      if (result) {
-        const idx = resolved.get(image.token)?.index ?? 0;
-        descriptions.push({ token: image.token, index: idx, text: result.text, cached: result.cached });
-        allFailed = false;
+    let ok = 0;
+    for (let i = 0; i < loaded.length; i++) {
+      const r = results[i];
+      if (r) {
+        descriptions.push({ token: loaded[i]!.token, index: resolved.get(loaded[i]!.token)?.index ?? i, text: r.text, cached: r.cached });
+        ok++;
       }
-      // On undefined (failure/timeout) → that image gets no description.
-      // If ALL fail, we fall back to hint below.
+      // undefined → that image gets no description (timeout/failure mid-batch)
     }
 
-    if (allFailed) {
-      // All delegations failed → hint fallback for all images.
-      text = `${text}\n${buildHintLine(loaded.length)}`;
+    if (ok === 0) {
+      // All failed/timed out → hint fallback (with paths, §3.4).
+      text = `${text}\n${buildHintLine(hintImages)}`;
       return { action: "transform" as const, text };
     }