Better failure handling for stdlib similarity matrix app

[2kai2kai2]

Summary by CodeRabbit

• Bug Fixes

  • Improved report validation to reject severely incomplete results before publishing.
  • Added clearer errors when the model service is unavailable or rejects a request.
  • Corrected report-generation behavior after lookup attempts are exhausted.

• Improvements

  • Updated automated tooling to use the production environment for more reliable report processing.
  • Final report responses now present findings without exposing intermediate lookup options.
  • Added coverage for final and intermediate response formats.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
beps	Ready	Preview	Aug 10, 2026 5:57pm
promptfiddle2	Building	Preview	Aug 10, 2026 5:57pm

[github-actions]

⏭️ Performance benchmarks were skipped

Perf benchmarks (CodSpeed) are opt-in on pull requests — they no longer run on every push. They always run automatically after merge to canary/main.

To run them on this PR, do any of the following, then push a commit (or re-run CI):

• Add RUN_CODSPEED=1 to the PR description, or
• Include run-perf or /perf in the PR title or any commit message.

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 68297c16-79cb-4146-81fb-42d1269631d8

📥 Commits

Reviewing files that changed from the base of the PR and between f63c7df and 098b92b.

📒 Files selected for processing (4)

• .github/workflows/stdlib-matrix.yml
• tools/stdlib-matrix/README.md
• tools/stdlib-matrix/baml_src/align.baml
• tools/stdlib-matrix/baml_src/tests.baml

---

📝 Walkthrough

Walkthrough

The stdlib matrix now performs client preflight validation, uses an answer-only final session, and tests the distinction between intermediate and final outputs. The workflow uses the production environment and blocks publication when most TypeScript symbols remain unjudged.

Changes

Stdlib matrix validation flow

Layer / File(s)	Summary
Session finalization and client preflight tools/stdlib-matrix/baml_src/align.baml, tools/stdlib-matrix/baml_src/tests.baml	align_matrix checks client availability before starting work. Final lookup rounds use session_final without lookup schemas. Tests cover the new output format. Existing formatting changes preserve behavior.
Report publication safeguards .github/workflows/stdlib-matrix.yml, tools/stdlib-matrix/README.md	The workflow and publishing documentation use boundary-tools-prod. Publication stops when more than half of TypeScript symbols are unjudged, before first-run or deployed-report comparisons.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related PRs

• BoundaryML/baml#4337: Extends the same stdlib-matrix workflow, alignment code, and tests.
• BoundaryML/baml#4349: Builds on the same session-based stdlib matrix workflow.

Poem

A rabbit checks the client at dawn,
Then sends the final lookup on.
Findings stay; schemas leave,
Bad reports cannot deceive.
Production guards the matrix bright.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch kai/stdlib-similarity-matrix

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}