Skip to content

fix(mcp): re-land leadbay_getting_started with the 8 review findings fixed - #177

Draft
ArtyETH06 wants to merge 3 commits into
mainfrom
ArtyETH06/getting-started-codex-fixes
Draft

fix(mcp): re-land leadbay_getting_started with the 8 review findings fixed#177
ArtyETH06 wants to merge 3 commits into
mainfrom
ArtyETH06/getting-started-codex-fixes

Conversation

@ArtyETH06

Copy link
Copy Markdown
Contributor

Re-lands the walkthrough backed out by #176, with every finding from the automated review on #175 fixed.

Merge #176 first — this branch carries the revert, so after #176 lands its diff is just the feature, corrected.

The 8 findings

Finding Fix
P1 The paid reveal told the agent to pass a singular leadId — a key leadbay_enrich_titles never reads. It would be dropped, and with no leadIds the tool falls back to the top of the wishlist: the confirmed one-contact reveal could spend on several leads. Both surfaces now say leadIds: ["<id>"], and say why the array matters.
P1 execute returned the module-level manifest by reference, and the server mutates results in place to attach _meta.update_available / _meta.notifications — so per-call metadata stuck to the singleton and could replay on a later call. structuredClone per call.
P1 The live eval recorded every call as ok:true, output_len:0 and never read the tool_result. A call failing BAD_INPUT / LAST_PROMPT_REQUIRED still satisfied required_calls and reached the judge as a success. Results parsed into the evidence; required_calls / required_order read only calls that succeeded. forbidden_calls still reads every call that fired — attempting a banned call is the violation.
P1 no-unprompted-enrich-spend leaned on a missing fixture to make a paid launch fail — but this runner ignores fixtures and hits the real API. The regression it guards would have been paid for in real credits. noSpend: true blocks /leads/selection/enrichment/launch inside the spawned server, before the network. The free mode:"discover" half still runs live.
P2 Gate 4 needs leadbay_enrich_titles + leadbay_bulk_enrich_status, which a read-only deployment (LEADBAY_MCP_WRITE=0) filters out — the tour offered a button that dies on the click. Gate 4 declares requires_tools + unavailable; the agent checks its own tool set first and closes at gate 3 instead.
P2 allowed_calls was declared by scenarios and read by nobody, so a scope eval could fan out into real tools and still be decided by a judge score. Enforced as a whitelist, mechanically, before the judge.
P2 The unconditional STOP line must be the last line of the message — but ENDING B requires the 1:1 offer to be last. The agent had to violate one, reopening the displaced-offer failure. STOP scoped to gate hand-backs; ENDING B says so at the point of use.
P2 WORKFLOWS.md still required a CRM check at gate 4 and a gate 5 scheduling handoff — gates that were cut before the feature shipped, making the normative criteria impossible to satisfy. Criteria rewritten to the four gates that exist.

Version

0.28.0 is already published to npm from a different commit, so the CHANGELOG section claimed a shipped version. Renumbered to 0.29.0 across package.json, server.json, the plugin manifest and its npx pins.

Tests

pnpm -r test1351 passing (up 9), typecheck + build clean. New coverage in two new files: getting-started-codex-review-fixes.test.ts and eval-runner-evidence-and-spend-guard.test.ts.

Two pre-existing eval helpers are modified (live-session-runner.ts, live-mcp-server.ts) — the two P1 eval defects have no other correct home, and #175 already modified llm-judge-shared.ts on the same grounds. No existing *.test.ts spec is touched.

Not covered

Closes https://github.com/leadbay/product/issues/3952

ArtyETH06 and others added 3 commits August 10, 2026 16:03
…eople-getting-started"

This reverts commit 24cf7cd, reversing
changes made to 4a8653d.
…people-getting-started"

This reverts commit 2220fe8.
…rough

P1 — the paid reveal was told to pass a singular `leadId`, a key
leadbay_enrich_titles does not read; it would have been dropped and the
confirmed one-contact reveal would fall back to the wishlist default,
spending on several leads. Both surfaces now say `leadIds: [<id>]`.

P1 — the tool returned the module-level manifest BY REFERENCE, and the
server mutates results in place to attach `_meta`. Deep-clone per call.

P1 — the live eval recorded every tool call as `ok:true, output_len:0`
and never read the tool_result, so a call that errored still satisfied
`required_calls` and reached the judge as a success. Results are now
parsed into the evidence, and required/order read only successful calls.

P1 — the no-spend scenario relied on a missing fixture, but the live
runner ignores fixtures: the regression it guards would have been paid
for in real credits. The paid launch is now blocked in the spawned
server, before the network.

P2 — gate 4 needs write tools the walkthrough does not get on a
read-only deployment; it now declares `requires_tools` and degrades to
the close instead of offering a button that dies on the click.
P2 — `allowed_calls` was declared by scenarios and enforced nowhere.
P2 — the unscoped STOP line fought ENDING B's required closing offer.
P2 — WORKFLOWS.md still required the CRM and scheduling gates that were
cut before the feature shipped.

Also renumbers to 0.29.0: 0.28.0 is already published to npm from a
different commit, so the CHANGELOG section claimed a shipped version.

Co-Authored-By: Claude <noreply@anthropic.com>
@ArtyETH06 ArtyETH06 self-assigned this Aug 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant