Adapter: compare stored Promptfoo evaluation exports

[seva9523]

User need

Promptfoo users can export a stored evaluation without rerunning it. EvalRepro should be able to snapshot two exported evaluation definitions/results and identify whether changes came from test cases, variables, prompts, assertions, metadata, ordering, or coverage.

This adapter would provide a framework-specific bridge without reading Promptfoo's internal database directly.

Proposed first input

Use the stable JSON produced by the existing command:

promptfoo export eval <evalId> --output eval.json
evalrepro snapshot promptfoo eval.json --name stored-eval -o manifest.json

CSV/HTML and direct database access are out of scope for the first PR.

Contract questions

• Which exported fields define semantic scope versus per-run provenance?
• How should provider/model identifiers, prompts, tests, vars, assertions, transforms, metadata, and metrics be represented?
• Which timestamps, generated IDs, and UI-only fields are non-semantic volatility?
• Can one canonical record represent a test-case × prompt/provider result without embedding model output in the public manifest?
• Should an adapter offer separate definition and result modes?

Minimum acceptance criteria

• Optional promptfoo adapter that reads a documented JSON export schema without importing Promptfoo internals.
• Clear schema-version detection and actionable errors for unsupported exports.
• Raw prompts, variables, outputs, and test records are hashed through the normal manifest path and never copied into the manifest.
• Generated IDs/timestamps are removed only when demonstrated to be non-semantic.
• Scope captures scoring-relevant configuration and permits intentional Promptfoo package-version comparisons.
• Network-free synthetic fixtures and mutation tests cover prompts, vars, assertions, transforms, metadata, order, and missing/added results.
• At least one real export from a non-sensitive demo evaluation is exercised in an opt-in case study.
• Documentation states whether the first release checks evaluation definition, stored results, or both.

Design status

The definition-only contract was merged in PR #23 as
420fd573.
It specifies that contract version 1 compares evaluation definitions and executed coverage, not model
responses, scores, traces, latency, cost, or token usage. Exact-head CI passed and automated review
reported no major issues. Promptfoo has not reviewed or endorsed the mapping; implementation and the
remaining acceptance criteria are still open.

Non-goals

• rerunning evaluations;
• connecting to Promptfoo Cloud;
• accessing user credentials or internal databases;
• claiming that identical configuration guarantees identical model outputs.

Collaboration request

A contributor familiar with Promptfoo's export format can begin by posting a sanitised schema outline and proposed SnapshotSource mapping. Please do not upload a private production evaluation.

[seva9523]

A source-checked definition-only adapter contract is now proposed in draft PR #23.

The design was checked against Promptfoo main at ab84555c1b0ff74eca6b03abb7936ac9a0149242 and deliberately separates:

• prompt/test/assertion/provider definition semantics;
• executed test-by-prompt coverage and order;
• generated run identity, timestamps, runtime, and provenance;
• excluded model outcomes, scores, traces, latency, cost, and token usage.

No adapter implementation or Promptfoo compatibility claim is made yet. The next gate is review of the contract, followed by strict v2/v3 synthetic fixtures and one public non-sensitive export case study.