Preview domain canonical URLs drop the custom domain's base path ("Host at /docs"), advertising 404 URLs to AI crawlers

[vishnukool]

Setup

• Project: agent37 (preview: agent37.mintlify.app)
• Custom domain: www.agent37.com with Host at /docs enabled in Domain setup (status: Connected)
• Standard subpath rewrites per your Vercel guide (/docs/:match* → agent37.mintlify.site/docs/:match*) — the custom domain works correctly

Bug

Pages served on the custom domain render the correct canonical (base path included). The same pages served on the .mintlify.app preview domain render a canonical pointing at the custom domain without the configured base path, i.e. a URL that 404s:

$ curl -s https://agent37.mintlify.app/models/how-to-setup-openclaw-grok-api | grep canonical
<link rel="canonical" href="https://www.agent37.com/models/how-to-setup-openclaw-grok-api"/>

Expected: https://www.agent37.com/docs/models/how-to-setup-openclaw-grok-api (the live page, 200).
Actual: https://www.agent37.com/models/how-to-setup-openclaw-grok-api → 404 on our domain.

Impact

The preview domain is noindex for classic search engines, but its robots.txt advertises Content-Signal: ai-train=yes, search=yes, ai-input=yes, so AI crawlers ingest these pages and take the declared canonical as the page's real address. We now observe a sustained crawler fleet (currently ~450 requests/day and growing, distinct proxy IPs, headless-Chrome fingerprint) requesting the bare non-/docs URLs on our domain — all 404s — and assistants that learned the wrong canonical can quote dead links to real users.

Ask

Preview-domain canonical generation should respect the configured "Host at" base path (or the preview should canonicalize to itself).