Harden toolkit docs generation and indexing

[jottakka]

Reviewer guide

Open the standalone reviewer handoff. It separates new behavior from unchanged behavior, identifies meaningful commits, and explains which generated commits to skip.

Summary

• fail closed on empty, incomplete, colliding, unsafe, or unwritable toolkit generation inputs while preserving last-known-good output
• replace the redundant full-site generator workflow build with focused type-checks/tests and one consistent Engine snapshot
• publish generated toolkit routes through sitemap and llms.txt, with idempotent artifact generation that cannot commit-loop
• align route, sidebar, Markdown, slug, index, and rich-subpage behavior across build-time and runtime consumers

Scope

• no MCP server or MCP tool implementation changed
• generated JSON remains sourced from existing Engine and design-system metadata
• Algolia remains externally managed; no ranking or credential configuration changed

Test plan

• pnpm run toolkit-docs:check (46 files, 581 tests)
• pnpm test (58 files, 774 tests)
• pnpm lint
• pnpm build
• mock CLI generation and strict output verification
• strict and non-strict provider failures exit nonzero without writing output
• failed all-toolkit merges preserve prior JSON byte-for-byte
• read-only output simulation preserves prior JSON and returns nonzero
• sidebar sync dry run reports 114 aligned toolkits
• live preview crawl: 114 pages, 114 JSON APIs, sitemap, and llms.txt aligned
• representative Markdown API checks include custom Datadog documentation chunks
• manual LLMSTXT workflow dispatch passes with no diff or extra PR
• Luna and Terra review/adjudication loops clean
• local Bugbot final pass clean; GitHub Bugbot skipped the duplicate review
• all PR checks pass

---

Note

Medium Risk
Touches sitemap, llms.txt, slug/route resolution, and generator fail-closed behavior—incorrect slug filtering or snapshot handling could hide integration pages or block doc updates, but changes are covered by extensive tests and preserve prior JSON on merge errors.

Overview
This PR tightens the toolkit docs pipeline and how generated integration pages are discovered, exported, and searched.

Generator & CI: The generate workflow now runs toolkit-docs:check instead of a separate generator build, uses a single cached Engine /v1/tool_metadata snapshot for skip-unchanged runs, and refuses to wipe all toolkits on an empty API response. Writes are atomic, toolkit IDs are validated for collisions/reserved names, merge failures surface as nonzero exits while keeping last-known-good JSON when possible, and optional pr_branch supports isolated verification PRs.

Site routing & exports: Toolkit slugs are validated (getToolkitSlug can return null); routes, the integrations index, sitemap, and sidebar sync skip or drop invalid entries and sort routes deterministically. Copy-page Markdown now includes toolkit/tool documentation chunks (replace/before/after) via shared sortDocumentationChunks. llms.txt expands real toolkit URLs (not [toolkitId] templates), groups them by integration category, avoids metadata-only commit loops, and the workflow skips fork PRs with full git history.

Search: Algolia UI adds debounced query, loading/error/empty states, and shared config helpers (with tests).

Docs copy: Agentic-development and README text reflect llms.txt in CI rather than automatic markdown negotiation on every URL.

^{Reviewed by Cursor Bugbot for commit 63c423c. Bugbot is set up for automated code reviews on this repo. Configure here.}

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
docs	Ready	Preview, Comment	Jul 13, 2026 10:36pm

[jottakka]

@cursor review

[cursor]

Current version of PR was reviewed by /review-bugbot with additional instructions on Jul 12, 22:44 GMT-3. It flagged 0 findings.

^{Bugbot on commit b0a39fa is skipped.}

[jottakka]

@cursor review

[jottakka]

@cursor review

[jottakka]

Algolia search regression verification

Reproduced on the preview before the fix:

• Algolia returned <em> highlights because the request omitted InstantSearch marker tags.
• Stale results and Searching… alternated repeatedly while typing.
• The dialog performed 13 DOM state transitions for github, exposing literal HTML tags and causing visible jank.

Fixed by requesting __ais-highlight__ markers, suppressing stale-query results, and debouncing input by 150 ms.

Final generated preview measurement:

• 1 Algolia request for typing github.
• 2 states only: Searching… → results.
• 0 literal-tag renders.
• 0 console errors.
• 0 search-time long tasks.

[jottakka]

@cursor review

[jottakka]

@cursor review

[jottakka]

@cursor review

[jottakka]

@cursor review

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 63c423c. Configure here.}