Interop: map Langtrace Dataset/Data/Evaluation to EvalPort suite.json / resultset.json

[adhabnr-ux]

Hi Langtrace team — I maintain EvalPort (https://github.com/adhabnr-ux/evalport), a small open JSON Schema spec for portable LLM eval test suites/cases/results, with Python + TypeScript SDKs that validate against it. The idea is just to let eval data move between tools without everyone writing bespoke importers/exporters.

I read through prisma/schema.prisma and your Dataset → Data → Evaluation models line up closely with EvalPort's shape:

• Data { input, output, expectedOutput, model, note, datasetId } ≈ EvalPort TestCase { input, expected_output, metadata }
• Dataset { name, description, Data[] } ≈ EvalPort EvalSuite { id, name, test_cases[] }
• Evaluation { userScore, ltUserScore, reason, testId, dataId } ≈ one entry of EvalPort ResultSet.results[] { test_case_id, passed, grader_results[] }

Concretely, a Dataset export could look like:

{
  "version": "1.0.0",
  "id": "<dataset.id>",
  "name": "<dataset.name>",
  "test_cases": [
    {
      "id": "<data.id>",
      "input": "<data.input>",
      "expected_output": "<data.expectedOutput>",
      "metadata": { "model": "<data.model>", "note": "<data.note>" }
    }
  ]
}

and Evaluation rows (which already carry userScore/reason) could round-trip as an EvalPort resultset.json. That would let a dataset built in Langtrace get run through any EvalPort-compatible runner, and conversely let a suite authored elsewhere get imported into a Langtrace Dataset/Data for scoring in your UI.

Happy to open a PR for an export/import route (maybe a script under scripts/, or an API route alongside the existing dataset endpoints) if this is useful — would want your input first on how Test.type (manual/automated) and Evaluation.type (llm/vectordb/framework/session) should map, since those don't have a direct EvalPort equivalent yet. No pressure either way — just flagging the overlap since it looked like a clean fit.

Spec: https://github.com/adhabnr-ux/evalport/blob/main/SPEC.md