Hi — nice framework, the Vitest-style describeTask/caseOf authoring model plus the run/eval matrix layering in packages/vieval/src/core/runner/ is a clean way to keep evals close to app code.
What I found in the source
Read packages/vieval/src/core/runner/aggregate.ts directly (not just the README):
RunResult — one scheduled task's output: { id, entryId, inferenceExecutorId, matrix, scores: readonly RunScore[] }RunScore — { kind: 'exact' | 'judge', score: number }, normalized to 0..1aggregateRunResults() rolls these up into AggregatedRunResults (per-run and per-provider exactAverage/judgeAverage/hybridAverage)
vieval run --report-out <path> writes this out as JSON report artifacts (per vieval report analyze/index/cases in the README's CLI section), which is a real structured "one eval run's graded results" object, not just a bare pass/fail count.
What EvalPort is
EvalPort is an open, Apache-2.0 interchange format for LLM eval results (TestCase/GraderResult/ResultSet JSON schema, validated by openeval.validate.validate_suite() / validate_result_set()), so eval output can move between tools.
Proposed adapter
EvalPort's existing adapters are all Python, and I'd keep this one consistent with that rather than writing a TypeScript package — so I'm proposing a standalone vieval-openeval-adapter Python package that reads vieval's exported --report-out JSON (not vieval's TS internals) and converts:
- each
RunResult → one EvalPort GraderResult (or one per RunScore, since a run can carry both an exact and a judge score) — kind → grader_id (e.g. exact_match / llm_judge), score passed through directly since both are already normalized 0..1
- the batch of
RunResults for a report → one EvalPort ResultSet
This is a low-pressure offer — mainly checking whether this is something you or your users would find useful before building it, since I'd want the report JSON shape confirmed against a real --report-out run rather than just the README. Would this be welcome, and is there a stable/documented version of that JSON artifact format I should build against?
Hi — nice framework, the Vitest-style
describeTask/caseOfauthoring model plus the run/eval matrix layering inpackages/vieval/src/core/runner/is a clean way to keep evals close to app code.What I found in the source
Read
packages/vieval/src/core/runner/aggregate.tsdirectly (not just the README):RunResult— one scheduled task's output:{ id, entryId, inferenceExecutorId, matrix, scores: readonly RunScore[] }RunScore—{ kind: 'exact' | 'judge', score: number }, normalized to0..1aggregateRunResults()rolls these up intoAggregatedRunResults(per-run and per-providerexactAverage/judgeAverage/hybridAverage)vieval run --report-out <path>writes this out as JSON report artifacts (pervieval report analyze/index/casesin the README's CLI section), which is a real structured "one eval run's graded results" object, not just a bare pass/fail count.What EvalPort is
EvalPort is an open, Apache-2.0 interchange format for LLM eval results (
TestCase/GraderResult/ResultSetJSON schema, validated byopeneval.validate.validate_suite()/validate_result_set()), so eval output can move between tools.Proposed adapter
EvalPort's existing adapters are all Python, and I'd keep this one consistent with that rather than writing a TypeScript package — so I'm proposing a standalone
vieval-openeval-adapterPython package that reads vieval's exported--report-outJSON (not vieval's TS internals) and converts:RunResult→ one EvalPortGraderResult(or one perRunScore, since a run can carry both anexactand ajudgescore) —kind→grader_id(e.g.exact_match/llm_judge),scorepassed through directly since both are already normalized0..1RunResults for a report → one EvalPortResultSetThis is a low-pressure offer — mainly checking whether this is something you or your users would find useful before building it, since I'd want the report JSON shape confirmed against a real
--report-outrun rather than just the README. Would this be welcome, and is there a stable/documented version of that JSON artifact format I should build against?