feat: Add Agent Behavior scorer for Python and TypeScript

[Abhijeet Prasad (AbhiPrasad)]

Agent Behavior scorer

Adds Behavior, an LLM judge for evaluating an agent against an Agent Behavior spec.

Add one behavior spec at .agents/behaviors/<name>/BEHAVIOR.md, then use the scorer like any other Autoevals scorer.

TypeScript

import { Behavior } from "autoevals";
import { Eval } from "braintrust";

Eval("Support agent", {
  data: () => [
    {
      input: {
        message: "Our API is returning 401s and production is blocked.",
      },
    },
  ],
  task: async (input) => runSupportAgent(input.message),
  scores: [Behavior],
});

Python

from autoevals import Behavior
from braintrust import Eval

Eval(
    "Support agent",
    data=[
        {
            "input": {
                "message": "Our API is returning 401s and production is blocked.",
            },
        },
    ],
    task=lambda input: run_support_agent(input["message"]),
    scores=[Behavior],
)

Braintrust passes the dataset case as input, the task return value as output, and the instrumented trace thread automatically. The task can return a final answer or a structured trajectory.

When a project contains multiple behaviors, select one explicitly:

scores: [Behavior.partial({ behavior: "support-ticket-triage" })]

scores=[Behavior(behavior="support-ticket-triage")]

Callers can also select a behavior with a BEHAVIOR.md path, complete file content, or loaded behavior object. Scores are 1 for compliance, 0 for non-compliance, and null/None when the behavior is not applicable or cannot be judged.

[github-actions]

Braintrust eval report

Autoevals (HEAD-1785358511)

Score	Average	Improvements	Regressions
NumericDiff	78% (0pp)	5 🟢	7 🔴
Time_to_first_token	9.77tok (-1.37tok)	131 🟢	68 🔴
Llm_calls	1.55 (+0)	1 🟢	-
Tool_calls	0 (+0)	-	-
Errors	0 (+0)	-	-
Llm_errors	0 (+0)	-	-
Tool_errors	0 (+0)	-	-
Prompt_tokens	502.5tok (-4.74tok)	33 🟢	31 🔴
Prompt_cached_tokens	0tok (+0tok)	-	-
Prompt_cache_creation_tokens	0tok (+0tok)	-	-
Prompt_cache_creation_5m_tokens	0tok (+0tok)	-	-
Prompt_cache_creation_1h_tokens	0tok (+0tok)	-	-
Completion_tokens	452.76tok (-8.71tok)	104 🟢	98 🔴
Completion_reasoning_tokens	339.49tok (-7.31tok)	92 🟢	75 🔴
Completion_accepted_prediction_tokens	0tok (+0tok)	-	-
Completion_rejected_prediction_tokens	0tok (+0tok)	-	-
Completion_audio_tokens	0tok (+0tok)	-	-
Total_tokens	955.27tok (-13.45tok)	111 🟢	91 🔴
Estimated_cost	0$ (0$)	64 🟢	53 🔴
Duration	9.74s (-1.39s)	131 🟢	70 🔴
Llm_duration	10.39s (-1.76s)	137 🟢	61 🔴

[Ankur Goyal (ankrgyl)]

do we want to hit openai directly via http (vs. an sdk) each time we test?

[Ankur Goyal (ankrgyl)]

does behavior refer to the spec name or an individual behavior?

[Ankur Goyal (ankrgyl)]

can zod do this more concisely?