feat(mcp): leadbay_getting_started — guided first-run walkthrough - #175
Open
ArtyETH06 wants to merge 19 commits into
Open
feat(mcp): leadbay_getting_started — guided first-run walkthrough#175ArtyETH06 wants to merge 19 commits into
ArtyETH06 wants to merge 19 commits into
Conversation
A brand-new user learns Leadbay by DOING, not by reading. Three gates, each
presenting exactly ONE option, each click running a real call:
1. "Pull today's leads" -> leadbay_pull_leads (no args)
2. "Enrich top leads" -> leadbay_enrich_titles (no titles = free)
3. "Run this every morning" -> no Leadbay tool; the host's scheduler
Ships as both a prompt (auto-emits a Claude Code skill) and a read-only
composite tool returning the step manifest, so natural phrasing ("I'm new")
reaches the tour on hosts that don't surface MCP prompts.
Two hard rules encoded in the manifest, prompt, and tests:
- Gate 2 spends NOTHING. Omitting `titles` returns mode:"discover", the free
preview. Passing titles/confirm/email/phone launches a PAID reveal, and
spending a 90-second-old account's quota to demo a feature is the worst
outcome available here.
- Gate 3 has calls:null because Leadbay exposes no scheduling API. The gate's
literal recurring wording ("every morning") is what lets the host's existing
SCHEDULED_TASK_PARAGRAPH flow fire on its own terms — server.ts is untouched,
so the daily-check-in evals are unaffected.
A warming lens (computing_wishlist/computing_scores) is explained and re-pulled
rather than reported as "no leads" — pull_leads already returns a server-built
two-option warm-up payload for that state, rendered verbatim.
The only edits to existing test files are two one-line registration appends
(the prompts/list toEqual array and TOOLS_WITH_ROUTING), matching the precedent
in 6ddbcf8 and the instruction in CLAUDE.md.
Co-Authored-By: Claude <noreply@anthropic.com>
Leadbay has NO CRM integration — no push, export, sync, or third-party OAuth exists, and report_friction's canonical missing_capability example is literally "Why can't I export to HubSpot?". But the AGENT usually does have one: users run a HubSpot/Salesforce/Pipedrive/Attio connector in the same host. So the new gate delegates rather than pretending. The walkthrough is now four gates: 1. "Pull today's leads" -> leadbay_pull_leads 2. "Enrich top leads" -> leadbay_enrich_titles (no titles = free) 3. "Add these to my CRM" -> no Leadbay tool; the agent's OWN connector 4. "Run this every morning" -> no Leadbay tool; the host's scheduler Gate 3 reuses the detection mechanism the connected-outreach-tool table in leadbay_prospecting_overview already established: read the host's installed-connector / installed-MCP inventory, else infer from conversation, else ask. It names the CAPABILITY, never a third-party tool name — a backticked `hubspot_create_company` would be the first such reference in the repo and would rot silently when the connector renames its tools. Four honesty guards, each with a test: - Never claim a CRM record was created unless the connector confirmed it. - Never write an email or phone into the CRM. Gate 2 is the FREE title preview, so no contact detail was ever revealed; writing one is fabricated PII. - With no connector: say so in one honest line, name the user's CRM, and offer leadbay_report_friction with category:'missing_capability' — the real route. - Don't hunt for a leadbay_* CRM tool; none exists. Modelled on the gate-4 scheduler delegation shipped in the previous commit — same shape, same "only the host can create one" discipline. Co-Authored-By: Claude <noreply@anthropic.com>
The tutorial now starts by proving the connection works, instead of opening with prose and jumping straight to leads. Five gates: 1. "Check my account" -> leadbay_account_status 2. "Pull today's leads" -> leadbay_pull_leads 3. "Enrich top leads" -> leadbay_enrich_titles (no titles = free) 4. "Add these to my CRM" -> no Leadbay tool; the agent's OWN connector 5. "Run this every morning" -> no Leadbay tool; the host's scheduler Gate 1 sits on top of two PINNED regressions, so its branches encode both and the tests assert them: - WORKFLOWS #30 — a brand-new org has no billing plan, so quota_status 401s. leadbay_account_status swallows that into `quota_error`. The gate must say NOTHING about quota and must NEVER suggest logging in again: the token is valid, the same response just read the account. This is the product#3761 401-hallucination bug, and a first-run tutorial is the worst possible place to reintroduce it. - WORKFLOWS #31 — account-status.ts:166 withholds the lens unless the trigger text mentions it. "Walk me through Leadbay" doesn't, so there is nothing to report; the gate must not volunteer it nor call another tool to find it. The lens appears naturally at gate 2. The underdeliver eval fixtures quota_status as a 401 on purpose, so the run proves the silence rather than assuming it. Co-Authored-By: Claude <noreply@anthropic.com>
…it fires
Live test showed the agent running the tools straight through without ever
firing a choice widget — the user watched a demo instead of taking a tutorial.
Root cause: the gates only DESCRIBED the widget in prose (gate_label /
gate_description as loose strings), leaving the agent to assemble the widget
call itself. Meanwhile leadbay_pull_leads ships a ready-made `next_steps`
{question, options[]} object, and the shared routing snippet says a next_steps
payload is the source of truth and must be mapped VERBATIM. The tour was on
the weak path.
Each step now carries that same payload shape:
next_steps: { question, options: [{label, description, kind}] } // exactly 1
explain: "what to TELL the user before firing the widget"
So the agent renders data instead of interpreting prose, and each gate is two
beats — explain, then ask, then WAIT for the click. The explain beat is the
tutorial half: gate 2 teaches what a lens is, gate 3 teaches what enrichment
is and that the preview costs nothing.
The prompt gains an explicit "NEVER run a step's tool without firing its
widget first and receiving the click" rule (with the one sane exception: the
user's own message already said to run everything), plus three failure modes
for exactly what went wrong.
The new prompt↔manifest widget-text audit caught a real bug on its first run:
markdown line-wrapping had split the widget strings across newlines, so the
shipped prompt and the manifest disagreed on the question text.
Co-Authored-By: Claude <noreply@anthropic.com>
…e clicks The buttons disappear when the tour ends. A user who was only ever shown buttons learned to click a tutorial and nothing about using Leadbay tomorrow. The manifest now carries `keep_going`: a what-you-want -> what-you-say cheat-sheet the agent renders as a small table at the finish. | Today's fresh leads | "Show me today's leads" | | Who to follow up with | "What should I follow up on" | | The story on one company | "Research <Company>" | | An email to a contact | "Draft outreach for <Contact>" | | Change who you target | "Narrow the audience to <sector>"| | Switch target audience | "Show me my lenses" | Every phrase is lifted VERBATIM from that tool's own routing.triggers, and a test enforces it: each `say` is checked against the trigger blocks in the generated tool descriptions, so a phrase that doesn't actually route fails the build. Teaching a phrase that silently does nothing is worse than teaching none, and "sounds about right" is exactly how that happens. Verified the guard bites: swapping in a plausible "Fetch my newest prospects please" fails with the offending phrase named. Skipped when the user abandons the tour early — they're already off doing what they wanted, and a tutorial summary would just interrupt. Co-Authored-By: Claude <noreply@anthropic.com>
…yllabus
The tour front-loaded text: STEP 0 asked for 2-3 sentences PLUS a preview of
all five upcoming clicks, and then gate 1 added its own explain beat on top.
The first button ended up buried under paragraphs nobody reads.
The opening is now, in one message:
1. one sentence on what Leadbay is
2. one short line naming the step — "Let's start with your account status."
3. gate 1's widget, fired immediately
and then it stops. No five-step preview, no lens explanation yet — each gate
already explains itself when its own turn comes, so saying it up front is
duplication that costs the user the thing they actually wanted: seeing it work.
Gate 1's widget text is shortened to match ("Check my Leadbay account status.")
and the gate no longer stacks a second explanation on top of the opening lines.
Two new failure modes cover the regression: opening with a wall of text, and
ending the first message without firing the widget at all.
The prompt<->manifest drift audit paid for itself again — markdown line-wrap
had split the new widget description across a newline, so prompt and manifest
disagreed on the exact string.
Co-Authored-By: Claude <noreply@anthropic.com>
…mprovised tour Live failure in Claude Desktop chat: "Walk me through Leadbay please" produced the agent's OWN product overview — a mental-model essay plus a lens table and a four-option "Where do you want to start?" widget. Nothing from leadbay_getting_started ran. The prompt was not missing. It was listed in PROMPT_CATALOG_BULLETS at ~char 10,786 of a 24.5k-char instruction block, one bullet among fifteen. A listing tells the agent the prompt EXISTS; nothing told it to CHOOSE it, so it wrote something itself. Adds a FIRST RUN routing line to the server instructions, placed before the generic start-here flow (now ~char 6,887, ahead of the catalog). It names the phrasings verbatim, says invoke `leadbay_getting_started` via `prompts/get`, and explicitly forbids improvising an overview — with the reason attached, so the prohibition doesn't read as arbitrary and get ignored the moment the agent thinks its own summary would be nicer. Deliberately NOT fixed by narrowing leadbay_prospecting_overview's very broad short_description: that regenerates its shipped snapshot and widens the blast radius. The routing line is the smaller, more direct change. New audit test pins the phrasings, the invoke verb, the anti-improvisation rule, and — load-bearing — that the line lands BEFORE the catalog listing. Co-Authored-By: Claude <noreply@anthropic.com>
Live in Claude Desktop, gate 1 came out as prose — "Let's start with your
account status — say the word and I'll check it" with a bold "-> Check my
account" line. No widget, no button.
Root cause was a contradiction the walkthrough shipped with: the prompt said
"exactly ONE option", while the shared next-steps routing snippet it INCLUDES
says "2-4 mutually-exclusive options". The model followed the shared rule and
fell back to prose. pull-leads.ts:123-127 already documented this exact failure
("a single option would make the model emit an invalid widget call, or silently
drop to prose"); the walkthrough was written against the other belief.
Every gate now carries two options: the forward action, then `I'm done for
now`. That satisfies the host contract so the widget renders, while keeping
exactly one way FORWARD — the tutorial still never asks a first-run user to
choose between paths. The exit must end the tour, never route elsewhere, or it
reintroduces the choice the rule exists to remove.
The rule is renamed ONE-FORWARD-OPTION and states the reason inline, so the
next person doesn't "fix" it back to one option.
Opening lines rewritten to lead with what the user gets ("a fresh batch of
companies worth selling to every day") and to promise something concrete
("five quick steps, and you'll have real leads by the end").
Tests now assert the two-option shape, exactly one exit, exit-ends-the-tour,
and forward-option-first.
Co-Authored-By: Claude <noreply@anthropic.com>
The user clicks a button labelled "check my account status" and got back "you're connected as X, an admin on Y". That under-delivers on the button they pressed. It was also wrong against the tool's own contract: leadbay_account_status's rendering_hint says "Report user + org, AND quota whenever readable — include quota even on a plain 'what account am I on?'", silent ONLY when quota is null, quota_error is set, or the org is unlimited. My gate told the agent to report user/org/plan and never mentioned quota at all, so it under-reported on every account where quota reads fine. Gate 1 now includes the canonical `rendering/quota-windows` snippet — the same one leadbay_account_status uses — so the tour renders what the web app renders: Daily / Weekly / Monthly, each with a ▰▱ gauge, % used, $ spent against cap, resets countdown, and the per-resource breakdown. Never raw "credits". The silence gate is preserved and widened to all three cases it actually covers (null / quota_error / unlimited_credits), not just the 401 — so WORKFLOWS #30 still holds and a plan-less org sees nothing about quota rather than an error. Two failure modes added: answering with a bare greeting when quota IS readable, and rendering credits or raw resource_type strings instead of the gauges. Co-Authored-By: Claude <noreply@anthropic.com>
The tour explained mechanics but never made the case. "Leadbay keeps a lens —
a description of who you sell to" tells a new user what the feature is; it does
not tell them why they should care, which is the actual question a first-run
user is asking.
Opening is now a short paragraph instead of two lines: what Leadbay is, how the
lens works, and what the five steps will leave them holding. Still bounded — it
must NOT walk the five steps one at a time, which is the wall-of-text version
that buried the first button two commits ago.
Every gate gains a concrete payoff, in working-life terms rather than feature
terms:
leads -> replaces the hour spent digging through directories; and the lens
sharpens from what you like, contact or skip
enrich -> ask for the operations director by title instead of pitching
whoever answers the switchboard
CRM -> no copy-pasting between tabs; a lead found here doesn't quietly
die in a chat window
daily -> prospecting is the first thing that slips on a busy week; this
removes the part that depends on remembering
Tests assert every gate carries a WHY IT'S USEFUL beat and that the opening
still teaches the lens without regressing to a syllabus, so a later trim can't
quietly strip the reasoning back out.
Co-Authored-By: Claude <noreply@anthropic.com>
Gate 1 rendered the quota gauges but left a first-run user staring at percentages and dollar figures they've never seen, with no way to tell whether they're good, bad, or something to worry about. It now follows the render with one or two plain lines: what the numbers count (the AI work Leadbay does for them — researching companies, qualifying leads — not something they spend by clicking around) and why it matters to them (it paces how many fresh leads arrive; heavy use now means a bigger batch queued next time, and it's where a smaller-than-expected batch would show its reason). Bounded deliberately: a sentence or two, no walking through every resource row, and no pricing pitch — the tool's own guidance already covers wait-vs-top-up when a window is actually exhausted. The explanation is skipped whenever the silence gate fires (quota null, quota_error, or unlimited_credits). Describing a gauge that isn't on screen is worse than saying nothing, and a test pins that. Co-Authored-By: Claude <noreply@anthropic.com>
The tour stopped at the free title preview and never revealed a contact, so a
first-run user finished the walkthrough without ever seeing the thing that
makes Leadbay useful: a name and an email to actually reach.
Gate 3 now runs in two beats:
Beat 1 — free. leadbay_enrich_titles with no titles/confirm/email/phone
returns mode:"discover", the available job titles. "Nothing spent
yet."
Beat 2 — paid, on consent. Ask them to pick 2-3 leads, state the cost BEFORE
they choose, then call again with the chosen titles, confirm:true
and email:true. Poll leadbay_bulk_enrich_status to completion and
report only the contacts that actually resolved.
The consent guarantee is the ordering, and it is explicit: the gate click
bought the free look, not the reveal. Silence is not consent, and neither is
"they clicked the gate earlier". Declining is a normal outcome — keep the
preview and move on.
It then says what it cost in one line (one credit per contact revealed), which
is the moment gate 1's quota numbers stop being abstract: they just watched
them move.
Gate 4 updated to match — it may now pass through real emails/phones, but ONLY
the ones the enrichment returned; if the user declined there are none, and
inventing one is fabrication.
The underdeliver eval keeps NO launch fixture on purpose: in that scenario the
user is never asked and never confirms, so a launch would hit an undeclared
endpoint and fail the run — the consent guarantee, enforced.
Co-Authored-By: Claude <noreply@anthropic.com>
The tour assumes an installed, signed-in connector — gate 1 is what proves it.
A user whose connector isn't installed, who can't sign in, or whose Leadbay
tools aren't appearing is upstream of every gate, and the walkthrough had
nothing for them but five buttons that would fail.
docs.leadbay.app/doc/leadbay-mcp/quickstart covers exactly that gap: connector
install, browser sign-in, the first query, and what to do when the tools don't
show up. The manifest now carries it as `docs_url`, with `docs_note` bounding
where it may appear:
1. BEFORE the tour, when the problem is SETUP rather than usage — point at
the page instead of guessing at install steps.
2. At the CLOSING, one plain link beside the keep_going cheat-sheet, for what
the five gates didn't cover (another machine, a teammate, signing back in).
Never between gates. A link mid-tour is an invitation to leave the thing the
user is in the middle of doing, which is why the audit pins the count at
exactly two rather than merely asserting the URL is present.
Both surfaces carry it — the prompt's routing branch + closing, and the tool
description — and the audit checks the prompt against the manifest's URL, so
the two can't drift the way the gate labels could.
Verified the count guard bites: injecting a third mention at gate 2 fails with
"expected the setup link twice, found 3".
Co-Authored-By: Claude <noreply@anthropic.com>
…send it to The tour ended at "here are some leads and here are some job titles." A first-run user never saw the thing that makes Leadbay worth opening: an email they could actually send. Two new beats, in the order that makes each one motivate the next. 3. "Draft the first email" -> leadbay_prepare_outreach (leadId ONLY = free) 4. "Find who to email" -> leadbay_enrich_titles, scoped to that ONE lead Gate 3 takes the top-scoring lead from gate 2 and drafts the opener through message_compose_v1. `recommended_contact` comes back with a null email at this point — that is not a gap to apologise for, it IS the hook: they now have an email written and nobody to send it to. Gate 4 answers exactly that question, which turns the paid reveal from an abstract "see who to contact" into the obvious next move, at one contact for one credit instead of two or three. Consent and honesty guards, each with a test: - Gate 3 spends NOTHING. `enrich:true` would launch a paid reveal off the back of a DRAFT click, so it is in forbidden_args with the reason attached. - The draft is addressed to the job TITLE. No contact name exists before gate 4 and inventing one is the single thing that would make the whole draft untrustworthy. - Gate 4 keeps the two-beat consent shape, now scoped to the drafted lead. The IRON LAW is NARROWED, deliberately: it said "never draft or send outreach", which the new gate would contradict outright. It now reads never SEND — drafting is the point and nothing leaves the chat, while sending, offering to send, and leadbay_report_outreach all stay forbidden. Gate labels are short because AskUserQuestion caps them at ~5 words; the audit caught "Draft an email to the top lead" at seven. Sentences live in `description`, where they belong. Also corrects drift the two-option and paid-reveal commits left behind: WORKFLOWS #50/#51 and the eval scenario still specified "EXACTLY ONE option, no 'Skip' sibling" — the normative eval contract asserting the opposite of shipped behaviour — and index.ts still called it "the three-gate script". Co-Authored-By: Claude <noreply@anthropic.com>
…rough Picking "I'm done for now" ended the tour in silence. That is the one moment a 1:1 offer is welcome rather than pushy: they have seen enough to know what Leadbay is, and stopped right before the setup work a call actually helps with — tuning the lens to their market, wiring the CRM push into their own stack, automating the daily run. The manifest gains `calendly_url` + `exit_offer`, and the prompt an "IF THEY TAKE THE EXIT" section. One short line and the link, then stop. Bounded on purpose, because the failure mode here is turning goodwill into a pitch: - ONE sentence and the link. They just said they were done. - Never re-open the walkthrough, never re-fire the declined gate, never argue for finishing the tour. - EXIT CLICK ONLY. If they left by TYPING a different request, skip it entirely and serve what they asked — a booking link on top of their real question is precisely the interruption they were avoiding. The URL is a PLACEHOLDER (calendly.com/leadbay-zoe/PLACEHOLDER) and must be swapped for Zoe's real link before this ships, or a new user gets a 404 at the friendliest moment in the tour. It is greppable by that literal word, and the audit pins the prompt's copy against the manifest so the two can't drift. Co-Authored-By: Claude <noreply@anthropic.com>
Replaces the placeholder that would have handed a new user a 404 at the friendliest moment in the tour. Verified the link resolves (HTTP 200). The `?month=2026-08` query param is stripped on purpose. It only pins which month Calendly's picker opens on, and this URL ships inside the product — a link carrying month=2026-08 opens on a stale calendar for everyone who clicks it after August. The bare URL opens on the current month, always. Co-Authored-By: Claude <noreply@anthropic.com>
…aimed the end
Live in Claude Desktop: stopping at gate 2 produced "No problem — we'll stop
there", the keep_going cheat-sheet and the docs link — and nothing about Zoe.
The offer never appeared.
Root cause was structural, not wording. The prompt shipped TWO sections that
both described the end of the tour: "IF THEY TAKE THE EXIT" and "CLOSING — HAND
THEM THE PHRASES". The agent reached the end, ran CLOSING, felt finished, and
stopped. The exit section it never applied. Worse, CLOSING's own skip-rule only
named the typed-off-script case, so an exit CLICK fell straight through to
"render the cheat-sheet" with nothing pointing back at the offer.
Merged into ONE section that owns how the tour stops, stating so explicitly
("this is the ONLY place that says what to do when the walkthrough stops"),
with three mutually-exclusive endings the agent picks between:
A finished all six gates -> cheat-sheet + setup link
B picked "I'm done for now" -> stop line + cheat-sheet + setup link
+ the 1:1 offer, REQUIRED and LAST
C typed something off-script -> none of it; serve what they asked
Beat 3 of ending B is called out as the one that gets dropped, with the reason
attached, because "an agent that renders the cheat-sheet feels finished" is
exactly what happened and a rule without its reason gets optimised away again.
The manifest's exit_offer now carries the same three-beat ordering, so both
surfaces describe one close rather than two.
Tests pin the structure a future edit would undo: the three endings exist and
are named where the agent decides, the offer is marked required-and-last, and
ending C explicitly gets no cheat-sheet, no link and no offer.
Co-Authored-By: Claude <noreply@anthropic.com>
`pnpm test:gate` failed at startup with "Could not resolve vitest.eval.config.ts". Not a failing eval — the runner could not START, and had not been able to since #71. That commit landed the live framework's parts (live-session-runner, live-mcp-server, mission-match-judge, eval-collector) and deleted the old fixture-based entry points, but never shipped replacements. The helpers were orphans: nothing imported them but each other, and all four package.json eval scripts pointed at files that do not exist on main. Adds the two missing pieces: - packages/mcp/vitest.eval.config.ts — serial (live sessions must not race), 15-min test timeout, includes only *.eval.ts so it can never overlap the unit config's *.test.ts. - packages/mcp/test/eval/scenarios.eval.ts — ONE runner over every *.scenario.ts, rather than the per-prompt boilerplate that rotted last time. Mechanical invariants first (required/forbidden calls, required_order as a subsequence), then the LLM judge, then the four floors. Three latent bugs surfaced the moment something finally imported the helpers, each found by running it rather than reading it: 1. mission-match-judge imports @leadbay/promptforge, whose exports point at dist/ — but promptforge's `build` script runs the prompt GENERATOR, and the actual tsc lives in a `compile` script nothing calls. dist/ has never existed. Aliased to source in the eval config. 2. callClaudeCLI built a shell command with JSON.stringify(prompt) — JSON escaping handed to /bin/sh, which applies its own. Died with "Unterminated quoted string" on the first real judge call, three times over. Now execFileSync with the prompt on stdin: no shell, no escaping. 3. The judge's 90s timeout could not read a 20-turn transcript. Raised, and made overridable via EVAL_JUDGE_TIMEOUT_MS. The getting-started scenarios were also single-turn, which cannot work for a GATED walkthrough: every gate stops and waits for a click, so the tour never advanced past gate 1 and the harness reported "required call never fired" — measuring the feature working correctly and calling it a failure. Both are now multi-turn, one turn per click, ending on the exit. Live proof, this branch: getting-started-completes-five-gates — MM 5 / IA 5 / NF 5 / TSF 5 19 assistant turns, 6 user turns, 217.8s, all 20 criteria passed That run exercises the whole walkthrough against the real Leadbay API, including the exit close that shipped broken earlier today. `pnpm -r test` is untouched (1351 passing): without EVAL=1 the eval file skips, so it can never fire in CI. Co-Authored-By: Claude <noreply@anthropic.com>
…ule gates Gates 5 and 6 both carried calls:null. Leadbay has no CRM integration and no scheduling API, so each handed off to a capability the HOST might have — which made the last third of a first-run tutorial a tour of somebody else's product, and pushed the payoff further from the click that earned it. Four gates now, every one calling a real Leadbay tool: 1. Check my account -> leadbay_account_status 2. Pull today's leads -> leadbay_pull_leads 3. Draft the first email -> leadbay_prepare_outreach (leadId only = free) 4. Find who to email -> leadbay_enrich_titles (free, then consented) The tour ends where Leadbay's own value ends: a contact revealed, an email already written to send them. The close is unchanged — cheat-sheet, setup guide, and Zoe's 1:1 on the exit. Removed with them: the CRM/scheduler failure modes, their WORKFLOWS #50/#51 criteria, the CHANGELOG bullet, and the three unit + six audit tests that existed only to pin gates that no longer ship. The scenario file is renamed off its stale gate count. Re-ran the eval live after the cut — MM 5 / IA 4 / NF 5 / TSF 5, 19 assistant turns over 6 user turns, so the shorter tour still lands the draft, the consented reveal and the exit offer. Co-Authored-By: Claude <noreply@anthropic.com>
ArtyETH06
marked this pull request as ready for review
August 10, 2026 13:25
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A brand-new user learns Leadbay by doing. Four gates, each presenting one way forward plus an exit, and every one calls a real Leadbay tool.
Check my accountleadbay_account_statusPull today's leadsleadbay_pull_leadsDraft the first emailleadbay_prepare_outreach—leadIdonly, freeFind who to emailleadbay_enrich_titles— free preview, then a consented paid revealThe tour ends where Leadbay's own value ends, with a contact revealed. A CRM push and a daily-schedule gate were built and then removed: both had
calls: nulland delegated to capabilities Leadbay doesn't have, which made the last third of a first-run tutorial somebody else's product.Ships as both a prompt (auto-emits a Claude Code skill) and a read-only composite tool returning the step manifest, so natural phrasing ("I'm new") reaches the tour on hosts that don't surface MCP prompts as slash commands. A
FIRST RUNrouting line in the server instructions stops the agent improvising its own overview instead.Why gate 3 sits before gate 4
leadbay_prepare_outreachreturnsrecommended_contactwithemailnull before enrichment. That isn't a gap to apologise for — it's the hook. The user ends gate 3 holding an email addressed to a job title with nobody to send it to, and gate 4 answers exactly that question. It also drops the first paid spend from "pick 2–3 leads" to one contact, one credit.Guarantees, each with a test
enrich:truewould launch a paid reveal off a draft click — it's inforbidden_argswith the reason attached.leadbay_report_outreachstays forbidden.quota_erroris set (WORKFLOWS Fill MCPB 0.6.2 SHA #30), never volunteers the lens (Fix async MCP output schemas #31).How the tour ends
One section owns this, with three endings — A finished, B exit click, C typed off-script. B is stop line → cheat-sheet → setup guide → the 1:1 offer with Zoe's Calendly, last and required. C gets none of it.
This shipped broken and was caught live: two sections both described the end, so the agent rendered the cheat-sheet, felt finished, and dropped the offer.
The eval runner is restored in this PR
pnpm test:gatecould not start —vitest.eval.config.tsandtest/eval/scripts/were deleted by #71 without replacements, so no eval has run on any PR since. Adds the config plus a single runner over all 13 scenarios, and fixes three latent bugs in the orphaned helpers (@leadbay/promptforgenever resolvable, the judge's shell-escaping crash, a 90s judge timeout too short for a 20-turn transcript). The getting-started scenarios were also single-turn, which can't work for gated flows — both are now multi-turn, one turn per click.Eval — latest run, live against the real API
WORKFLOWS #50 (walkthrough) and #51 (over-claim guard).
#51 fails, and the finding is real. The consent gate itself held —
enrich_titlesreturnedmode:"discover"withbulk_id: null, nothing spent. But the agent then calledleadbay_research_lead_by_id, a tool the walkthrough never mentions, obtained contact names around gate 4's confirm, and told the user it had "a working phone number and email" — neither of which appears anywhere in that response. Two defects: gate 4 has a side door, and it over-claimed a channel it never received. Not yet fixed.Not covered
packages/mcp/package.jsonis0.28.0and 0.28.0 is already published to npm from a different commit. The CHANGELOG section here claims that number. Wants renumbering before merge.Closes https://github.com/leadbay/product/issues/3952