Skip to content

Add a built-in llm-as-judge backend #21

Description

@anilmurty

Judge-scored benchmarks currently route through the DeepEval adapter. A lightweight built-in judge backend
(call a provider model directly with a scoring rubric) gives users a judge without the extra dependency. Add
it behind the existing Judge protocol (tjbench/judge.py) so --judge llm works.

Keep the verdict language honest: a judge reports a measured pass/fail against a rubric, never a guarantee.

Acceptance criteria: --judge llm --judge-model <spec> scores a judged benchmark via a provider model;
MockJudge path still works offline; tests added.

Metadata

Metadata

Assignees

No one assigned

    Labels

    help wantedExtra attention is needed

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions