feat(atb2): handle_issue evals; run_tests.sh picks a pipeline stage; //# headers for the playground graph

[ATX24]

• fix_in_budget (PassRate 0.6): every Easy/Medium reference issue with a
  time_budget_s, DryRun; the agent must land a fix that passes the gate
  inside the budget. design_doc (PassRate 0.5): Hard reference issues;
  the design doc must match the reference at LLM-judged similarity

  = 0.7. Both are hours of wall-clock and opt in; ATB2_ISSUES narrows.
  ReferenceIssue gains time_budget_s and design_doc.

• run_tests.sh replaces setup_database.sh: resolves the two Supabase
  secrets through Infisical (and only those: an injected
  ANTHROPIC_API_KEY makes the Claude Code CLI drop its login), then asks
  which stage to test: create / organize / pr.
• //# headers on create_issue, organize_issue and gauge_issue so the
  playground's control-flow graph shows named steps.

Stack (bottom to top)

1. feat(atb2): sandbox for handle_issue: worktree lifecycle, allowlisted env, repro pre-check on canary #4634 sandbox: worktree lifecycle, allowlisted env, repro pre-check
2. feat(atb2): the gate: fmt, clippy, changed-crate tests, nextest, insta, the baml corpus; judged by exit code #4635 the gate: re-run by the pipeline, judged by exit code
3. feat(atb2): handle_issue: a design pass, then a fix pass in the sandbox; PR body; outcome; agents run on Fable #4636 handle_issue: design pass, fix pass, PR body, outcome
4. feat(atb2): handle_issue evals; run_tests.sh picks a pipeline stage; //# headers for the playground graph #4637 evals, run_tests.sh, //# headers

Each PR passes baml check / baml fmt / baml test on its own; the top of the stack reproduces the reviewed baml/feedback-part-3 tools/atb2 exactly. Supersedes baml/feedback-part-3.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
beps	Ready	Preview	Aug 31, 2026 2:16am
promptfiddle2	Ready	Preview	Aug 31, 2026 2:16am

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

⏭️ Performance benchmarks were skipped

Perf benchmarks (CodSpeed) are opt-in on pull requests — they no longer run on every push. They always run automatically after merge to canary/main.

To run them on this PR, do any of the following, then push a commit (or re-run CI):

• Add RUN_CODSPEED=1 to the PR description, or
• Include run-perf or /perf in the PR title or any commit message.