Allow Adding Knowledgebase Only MCP Server

src/CrestApps.Core.Docs/docs/changelog/1.1.0.md

@@ -19,7 +19,61 @@ page will be updated as changes land after 1.0.0.
 
 - upgrades the framework's dependency baseline, including YesSql 6.0 (new `ISession.SaveAsync` signature), the Model Context Protocol 2.0 packages, the GitHub Copilot SDK 1.0.8 (new `PermissionsApi.SetAllowAllAsync` mode-based API), Anthropic 12.39.0, OllamaSharp 5.4.30, the .NET 10.0.10 runtime/extension packages, and the `Microsoft.Extensions.AI` 10.8.3 packages
 
+## Breaking Changes
+
+- **MCP servers no longer expose tools by default.** In 1.0.0 a server registered with
+  `WithCrestAppsHandlers()` listed and invoked registered tools by default. In 1.1.0 tool exposure is
+  opt-in through the new `McpServerOptions` site settings: nothing is listed or callable until you either
+  add the tool or tool instance name to `McpServerOptions.Tools` or set `McpServerOptions.ExposeAllTools`
+  to `true`. Hosts upgrading from 1.0.0 that relied on the previous "expose everything" behavior must set
+  `ExposeAllTools = true` (or populate `Tools`) to keep exposing their tools. Prompts and resources are
+  unaffected and are still always registered
+
+## MCP server tool exposure
+
+- reworks tool exposure into an opt-in allow-list driven by the `McpServerOptions` site settings. Nothing
+  is exposed by default: an MCP server lists and invokes only the tools and configured
+  [tool instances](../core/tool-instances.md) named in `McpServerOptions.Tools`, or every tool
+  and instance when `McpServerOptions.ExposeAllTools` is `true`. The allow-list is enforced by both the
+  list and call handlers, so a tool that is not exposed can neither be discovered nor invoked. Because
+  `McpServerOptions` is backed by site settings, operators choose which tools to expose from the admin
+  settings UI without redeploying. The MVC and Blazor sample hosts add an "Exposed tools" editor to their
+  MCP server settings
+
+## Documentation search tool instances
+
+- adds documentation search [tool instance sources](../core/tool-instances.md), so a host exposes one
+  callable search function per documentation site it configures. These are ordinary tool instance sources
+  registered on the tool instances builder with `AddDocumentationSearchSources()` (or the individual
+  `AddSitemapDocumentationSource()`, `AddSearchIndexDocumentationSource()`, and
+  `AddAlgoliaDocumentationSource()` methods), so they can be used with or without the MCP server. Each
+  configured instance binds one site and surfaces as a distinct function the AI model can call, and the
+  instances are managed and persisted through the existing tool instance store (YesSql or Entity Framework
+  Core) and UI. The MVC and Blazor sample-host tool instance editors add source-specific field groups so
+  operators can configure a documentation site (base URL, sitemap or index URL, Algolia
+  application/index/search-only key, and per-instance result limits) directly from the create and edit
+  forms
+- ships three documentation search strategies, each as its own source and settings model: the sitemap
+  source crawls a site through its `sitemap.xml` (for example a public Docusaurus site such as
+  `core.crestapps.com`), the search-index source downloads a prebuilt JSON search index (for example a
+  MkDocs Material `search_index.json`) and ranks it locally, and the Algolia source forwards queries to
+  the hosted Algolia DocSearch API. A singleton materializer caches the crawled corpus or downloaded index
+  per instance and rebuilds it only when the instance changes, so the corpus is reused across calls
+- exposes documentation search functions through the MCP server the same way as any other tool instance:
+  add the instance name to `McpServerOptions.Tools` (or enable `ExposeAllTools`) to make it discoverable
+  and callable
+
 ## Fixes
 
+- makes the MCP tool allow-list respond to site-settings changes at runtime. The list and call handlers
+  now read `McpServerOptions` through `IOptionsMonitor` instead of the cached `IOptions`, so exposing or
+  removing a tool from the admin settings page takes effect without restarting the host
+- lets an MCP client invoke a code tool advertised under a function name that differs from its
+  registration key. The call handler resolves the tool by the published function name (mirroring the list
+  handler) so a listed tool can always be called
+- rethrows cancellation from the documentation search function instead of reporting it as a successful
+  error result, and returns a stable, non-leaking failure message for other errors
+- updates the MVC and Blazor sample hosts to load `@crestapps/bootstrap-select` 1.2.3 for Bootstrap
+  select styling
 - fixes post-session processing endlessly retrying and eventually failing when the AI returned a successful (HTTP 200) response that could not be parsed into structured task results. The no-tools structured output path now records a `Failed` result with a diagnostic message instead of silently returning no result, so these responses no longer exhaust all retry attempts. The unparseable-response case is now logged at `Warning` (including a preview of the raw AI response) instead of only at `Debug`, and the recorded task error message now explains that the AI produced no parseable result or there was no content to evaluate.
 - avoids issuing a post-session AI request when there is no meaningful user content to evaluate. Sessions whose user prompts are empty, whitespace-only, or system-generated are skipped, so a real AI call is only made when there is something to analyze.

src/CrestApps.Core.Docs/docs/mcp/server.md

@@ -299,12 +299,105 @@ services.Configure<McpServerOptions>(options =>
 
 ## Tool Exposure
 
-When your application acts as an MCP server, registered AI tools are exposed to external clients. The server endpoint's `WithListToolsHandler` and `WithCallToolHandler` callbacks delegate to your tool registry:
+When your application acts as an MCP server, tool exposure is **opt-in**. Nothing is listed or callable by default: the server only exposes the tools and configured [tool instances](../core/tool-instances.md) that you explicitly allow through the `McpServerOptions` site settings.
 
-1. **List tools** — Returns metadata (name, description, JSON schema) for all registered tools
-2. **Call tool** — Resolves the tool by name from the registry and invokes it with the provided arguments
+```csharp
+services.Configure<McpServerOptions>(options =>
+{
+    // Expose specific tools and tool instances by name.
+    options.Tools = ["crestapps-docs", "weather"];
+
+    // Or expose every tool and tool instance.
+    // When set to true, the Tools allow-list above is ignored.
+    options.ExposeAllTools = true;
+});
+```
+
+| Property | Effect |
+|----------|--------|
+| `Tools` | An allow-list of tool and tool instance names to expose. Matching is case-insensitive. |
+| `ExposeAllTools` | When `true`, every tool and tool instance is exposed and the allow-list is ignored. |
+
+Because `McpServerOptions` is backed by site settings, an operator can choose which tools to expose from the admin **Settings → MCP server** page without redeploying. The allow-list is enforced by **both** the list and call handlers, so a tool that is not exposed can neither be discovered nor invoked.
+
+Both code-registered tools (from `AddCoreAITool<T>()`, see [Custom Tools](../core/tools.md)) and stored tool instances (from any registered [tool instance source](../core/tool-instances.md), such as the [documentation search sources](#exposing-a-documentation-knowledge-base)) participate in the same allow-list.
+
+## Exposing a documentation knowledge base
+
+A common reason to run an MCP server is to answer questions from product or framework documentation that lives on a public site (such as a Docusaurus or MkDocs site). Instead of indexing that content into a vector store, the built-in documentation search [tool instance sources](../core/tool-instances.md) let an operator declare a documentation site as a **tool instance** and scan it on demand.
+
+These sources are ordinary tool instance sources registered on the tool instances builder, so they are usable with or without the MCP server. Each configured instance binds one site and surfaces as its own callable function, so a host can offer "search the CrestApps docs" and "search the Orchard Core docs" as two distinct tools. Instances are persisted and managed through the standard tool instance store and UI, and are exposed to MCP clients through the [allow-list](#tool-exposure) above — nothing is exposed until you opt in.
+
+### Registering the sources
+
+```csharp
+builder.Services.AddCrestAppsCore(crestApps => crestApps
+    .AddAISuite(ai => ai
+        .AddOpenAI()
+        .AddToolInstances(toolInstances => toolInstances
+            .AddDocumentationSearchSources()
+            .AddYesSqlStores()
+        )
+        .AddMcpServer(mcpServer => mcpServer
+            .AddYesSqlStores()
+        )
+    )
+    .AddYesSqlDataStore(configuration => configuration
+        .UseSqLite("Data Source=app.db;Cache=Shared")
+    )
+);
+```
+
+`AddDocumentationSearchSources()` registers all three built-in sources. To register only the ones you need, call the individual `AddSitemapDocumentationSource()`, `AddSearchIndexDocumentationSource()`, and `AddAlgoliaDocumentationSource()` methods instead. Once the sources are registered, operators create configured instances (each bound to one site) through the tool instances UI or store, and each instance becomes a callable function the AI model can invoke.
+
+### Search strategies
+
+A documentation site can be indexed in different ways depending on what the generator publishes. Each strategy is its own source with its own settings model, so you pick the one that matches the site.
+
+| Source | Registration | Best for | How it works |
+|--------|--------------|----------|--------------|
+| Sitemap crawl | `AddSitemapDocumentationSource()` | Any site that publishes `sitemap.xml` (Docusaurus, MkDocs, and most static sites). | Crawls pages, strips HTML, and ranks locally with keyword scoring. |
+| Search index | `AddSearchIndexDocumentationSource()` | MkDocs Material and other sites that publish a fetchable `search_index.json`. | Downloads the prebuilt index once and ranks its entries locally. |
+| Algolia DocSearch | `AddAlgoliaDocumentationSource()` | Docusaurus sites (and others) wired to hosted Algolia DocSearch. | Forwards the query to Algolia, which performs the ranking. |
+
+The registered source names are defined by `DocumentationToolConstants` (`sitemap-documentation`, `search-index-documentation`, `algolia-documentation`), and all three carry the `Knowledgebase` category.
+
+Each strategy binds a settings model:
+
+- **`SitemapDocumentationToolSettings`** — `BaseUrl` (site root, for example `https://core.crestapps.com`), optional `SitemapUrl` (defaults to `{BaseUrl}/sitemap.xml`), optional `MaxResults`, and optional `MaxPages`.
+- **`SearchIndexDocumentationToolSettings`** — `BaseUrl` (used to resolve relative locations and the default index URL), optional `IndexUrl` (defaults to `{BaseUrl}/search/search_index.json`), and optional `MaxResults`. This targets the MkDocs Material `search_index.json` schema (`{ "docs": [ { "location", "title", "text" } ] }`).
+- **`AlgoliaDocumentationToolSettings`** — `ApplicationId`, `ApiKey` (Algolia **search-only** key, never a write key), `IndexName`, and optional `MaxResults`.
+
+### Example: a public Docusaurus site
+
+A public Docusaurus site that requires no authentication — such as [core.crestapps.com](https://core.crestapps.com) — only needs the sitemap crawl source. Docusaurus publishes a standard `sitemap.xml` at the site root, so the crawler discovers `{BaseUrl}/sitemap.xml` automatically.
+
+1. Register the sitemap source (or all sources) as shown above.
+2. Create a tool instance from the **Documentation search (sitemap)** source with a **Name** (for example `crestapps-docs`, the name you expose to MCP clients), a clear **Description**, and a **Base URL** of `https://core.crestapps.com`.
+3. Expose the instance through the MCP server by adding its name to the allow-list:
+
+```csharp
+services.Configure<McpServerOptions>(options =>
+{
+    options.Tools = ["crestapps-docs"];
+});
+```
+
+Because the site is public, no headers, API keys, or credentials are involved — the crawler issues plain anonymous `GET` requests. The first search crawls the site and caches the corpus; later searches reuse the cache.
+
+### Corpus caching
+
+The runtime documentation source (the crawled corpus or downloaded index) is built lazily and cached by a singleton `IDocumentationSourceMaterializer`, keyed by the instance identifier. The cache is rebuilt only when the instance changes, so an edit to a site's settings is picked up on the next search while an unchanged instance reuses its corpus across calls.
+
+### Adding a new documentation source
+
+To support a documentation site that none of the built-in strategies cover, implement a new [tool instance source](../core/tool-instances.md):
+
+1. Create a settings model for the user-provided configuration.
+2. Implement `IAIToolInstanceSource.CreateTool(AIToolInstance)` to read the settings and return a `DocumentationSearchToolFunction` (or your own `AIFunction`) bound to a concrete `IDocumentationSource`.
+3. Register the source on the tool instances builder with `AddSource<TSource>(name, configure)`.
 
-Tools registered via `AddCoreAITool<T>()` (see [Custom Tools](../core/tools.md)) are automatically available to MCP clients unless they are marked with `.Hidden()`. Hidden tools remain available to explicitly configured profiles and agents, but the shared MCP handlers do not list or invoke them directly.
+Operators then create instances of your new source exactly like the built-in ones, and expose them through the same MCP allow-list.
 
 ## Server Metadata