ScrapeGoat: Your AI's Documentation GOAT!

ScrapeGoat is a documentation indexer for AI coding tools. It crawls web docs, imports local documentation, stores searchable content in PostgreSQL with pgvector, and exposes the result through a Web UI, HTTP API, and MCP Streamable HTTP endpoint.

What It Does

• Indexes documentation as libraries, versions, and revisioned sources.
• Searches with PostgreSQL full-text search and optional vector embeddings.
• Serves MCP tools for AI clients such as Codex, Claude Code, Cursor, and OpenCode.
• Provides a Web UI for browsing, importing, refreshing, deleting, renaming, and rolling back documentation.
• Supports local file and archive uploads from the Web UI.
• Tracks scrape, refresh, source, rollback, restore, favicon, and backup jobs.
• Uses API keys with Guest, User, and Admin access levels.

Architecture

ScrapeGoat is Docker-first. Host npm and node commands are for development, tests, and builds; production services should run with Docker Compose.

Service	Default Port	Purpose
postgres	5432	PostgreSQL with pgvector when using the bundled stack.
worker	8080	Pipeline worker and trusted internal tRPC API.
mcp	6280	MCP Streamable HTTP server at /mcp.
web	6281	Web dashboard.
favicon-api	internal	Favicon discovery helper.

In the multi-container stack, Web and MCP connect to the worker with:

http://worker:8080/api

Quick Start

Clone and configure:

export SCRAPEGOAT_REPOSITORY_URL="<repository-clone-url>"
git clone "$SCRAPEGOAT_REPOSITORY_URL"
cd scrapegoat
cp .env.example .env

Edit .env. At minimum, configure an embedding provider if you want semantic search. OpenAI-compatible example:

OPENAI_API_KEY=sk-...
SCRAPEGOAT_EMBEDDING_MODEL=text-embedding-3-small
SCRAPEGOAT_DB_VECTOR_SIZE=1024

Build the ScrapeGoat image and start the full local stack with bundled PostgreSQL:

docker compose -f docker-compose.build-only.yml build scrapegoat
docker compose -f docker-compose.postgres.yml up -d --no-build

Read the generated first-start API keys:

docker compose -f docker-compose.postgres.yml exec worker \
  sh -lc 'cat /tmp/.scrapegoat-apikeys'

Open:

Web UI: http://localhost:6281/
MCP:    http://localhost:6280/mcp

Use Authorization: Bearer <api-key> for MCP clients. Use the settings button in the Web UI header to enter a User or Admin key.

Compose Files

File	Use
docker-compose.build-only.yml	Builds the single ScrapeGoat image used by worker, web, MCP, and backup jobs.
docker-compose.build.yml	Compatibility build file for the same image.
docker-compose.postgres.yml	Full local/single-host stack with PostgreSQL and pgvector. Recommended for most installs.
docker-compose.yml	Runtime stack without PostgreSQL. Requires SCRAPEGOAT_DB_URL.

Common commands:

# Build
docker compose -f docker-compose.build-only.yml build scrapegoat

# Start full bundled stack
docker compose -f docker-compose.postgres.yml up -d --no-build

# Restart without rebuilding
docker compose -f docker-compose.postgres.yml up -d --no-build

# Stop services
docker compose -f docker-compose.postgres.yml down

# Stop services and remove bundled PostgreSQL data
docker compose -f docker-compose.postgres.yml down -v

External PostgreSQL

Use docker-compose.yml when PostgreSQL is managed outside Compose.

Requirements:

• Docker and Docker Compose.
• PostgreSQL 16 or newer.
• pgvector installed in the target database.
• A database user with access to the target database.

Database setup example:

CREATE DATABASE scrapegoat;
\c scrapegoat
CREATE EXTENSION IF NOT EXISTS vector;

.env example:

SCRAPEGOAT_DB_URL=postgresql://scrapegoat:<password>@postgres.example.com:5432/scrapegoat
SCRAPEGOAT_DB_VECTOR_SIZE=1024
OPENAI_API_KEY=sk-...
SCRAPEGOAT_EMBEDDING_MODEL=text-embedding-3-small

Start the runtime stack:

docker compose -f docker-compose.build-only.yml build scrapegoat
docker compose -f docker-compose.yml up -d --no-build

For multi-container deployments, set shared keys explicitly:

SCRAPEGOAT_APIKEY_ADMIN=<admin-key>
SCRAPEGOAT_APIKEY_USER=<user-key>
SCRAPEGOAT_REMOTE_APIKEY=<same-admin-key-used-by-web-and-mcp>
SCRAPEGOAT_INTERNAL_RPC_SECRET=<random-private-internal-secret>

Generate private values with:

openssl rand -hex 32

Do not expose SCRAPEGOAT_INTERNAL_RPC_SECRET to browsers, MCP clients, or public networks.

Access Control

ScrapeGoat always uses API keys for MCP and mutating Web UI actions.

Role	Access
Guest	Web UI read-only browsing. No MCP access.
User	Web UI read-only access plus MCP search/list/read tools.
Admin	Full Web UI and MCP write access.

On first startup, ScrapeGoat generates Admin and User keys, stores them in PostgreSQL metadata, and writes them once inside the worker container:

/tmp/.scrapegoat-apikeys

Seed keys before first startup:

SCRAPEGOAT_APIKEY_ADMIN=sg_admin_value
SCRAPEGOAT_APIKEY_USER=sg_user_value
SCRAPEGOAT_REMOTE_APIKEY=sg_admin_value

Rotate generated or seeded keys once on startup:

SCRAPEGOAT_MASTER_RESET_ALL_APIKEYS=true

Unset the reset flag after the reset has been consumed.

MCP

Endpoint:

http://localhost:6280/mcp

Authentication:

Authorization: Bearer <api-key>

Legacy /sse and /messages routes are not part of ScrapeGoat. Use Streamable HTTP /mcp.

MCP Tools

Tool	Role	Description
search_docs	User/Admin	Search indexed documentation.
list_libraries	User/Admin	List indexed libraries.
find_version	User/Admin	Resolve the best version for a library.
fetch_url	User/Admin	Fetch one URL and return Markdown.
scrape_docs	Admin	Enqueue documentation indexing.
refresh_version	Admin	Enqueue an ETag-aware version refresh.
add_source	Admin	Add a source to an existing version.
refresh_source	Admin	Enqueue source refresh.
delete_source	Admin	Enqueue source deletion.
rollback_version	Admin	Roll back to an archived revision.
remove_docs	Admin	Remove documentation.
list_jobs	Admin	List jobs.
get_job_info	Admin	Get one job.
cancel_job	Admin	Cancel a queued or running job.
clear_completed_jobs	Admin	Clear completed, failed, or cancelled jobs.

Client Examples

Codex ~/.codex/config.toml:

[mcp_servers.scrapegoat]
url = "http://localhost:6280/mcp"

[mcp_servers.scrapegoat.headers]
Authorization = "Bearer <api-key>"

Claude Code .claude/settings.json or user .claude.json:

{
  "mcpServers": {
    "scrapegoat": {
      "type": "url",
      "url": "http://localhost:6280/mcp",
      "headers": {
        "Authorization": "Bearer <api-key>"
      }
    }
  }
}

Cursor .cursor/mcp.json:

{
  "mcpServers": {
    "scrapegoat": {
      "type": "streamableHttp",
      "url": "http://localhost:6280/mcp",
      "headers": {
        "Authorization": "Bearer <api-key>"
      }
    }
  }
}

OpenCode opencode.json or opencode.jsonc:

{
  "mcp": {
    "scrapegoat": {
      "type": "remote",
      "url": "http://localhost:6280/mcp",
      "headers": {
        "Authorization": "Bearer <api-key>"
      },
      "enabled": true
    }
  }
}

If a client cannot store HTTP headers for remote MCP servers, use a local proxy that injects the Authorization header.

Web UI

Open:

http://localhost:6281/

Guest users can browse indexed documentation. Admin users can:

• Add documentation from a URL.
• Upload local files and archives.
• Refresh versions or individual sources.
• Delete versions or individual sources.
• Rename libraries and versions.
• Roll back to archived revisions.
• Replace, refresh, or remove library icons.
• Inspect and clear jobs.
• Run backup/restore helper jobs.
• Manage User API key access.

Local Uploads

The Web UI can import local documentation files and archives. Upload paths are sanitized before staging, archives are expanded into an import tree, and only accepted files are committed to the indexing pipeline.

Important limits are controlled by environment variables documented in .env.example, including:

• SCRAPEGOAT_WEB_IMPORT_MAX_FILES
• SCRAPEGOAT_WEB_IMPORT_MAX_TOTAL_SIZE_BYTES
• SCRAPEGOAT_WEB_IMPORT_MAX_FILE_SIZE_BYTES
• SCRAPEGOAT_WEB_IMPORT_MAX_ARCHIVE_ENTRIES
• SCRAPEGOAT_WEB_IMPORT_MAX_ARCHIVE_UNCOMPRESSED_BYTES
• SCRAPEGOAT_WEB_IMPORT_MAX_PATH_LENGTH

Configuration

Copy .env.example to .env and change only the values you need.

High-value settings:

Area	Variables
Database	SCRAPEGOAT_DB_URL, SCRAPEGOAT_DB_VECTOR_SIZE, SCRAPEGOAT_DB_USER, SCRAPEGOAT_DB_PASSWORD, SCRAPEGOAT_DB_NAME
Container image	IMAGE_REPOSITORY, SCRAPEGOAT_IMAGE_TAG
Service ports	SCRAPEGOAT_WEB_HOST_PORT, SCRAPEGOAT_MCP_HOST_PORT, SCRAPEGOAT_WORKER_HOST_PORT
API keys	SCRAPEGOAT_APIKEY_ADMIN, SCRAPEGOAT_APIKEY_USER, SCRAPEGOAT_REMOTE_APIKEY, SCRAPEGOAT_INTERNAL_RPC_SECRET
Embeddings	SCRAPEGOAT_EMBEDDING_MODEL, OPENAI_API_KEY, OPENAI_API_BASE, GOOGLE_API_KEY, GOOGLE_APPLICATION_CREDENTIALS, AWS_REGION, AZURE_OPENAI_API_KEY
Favicons	SCRAPEGOAT_FAVICON_API_URL, SCRAPEGOAT_FAVICON_API_REPLICAS, SCRAPEGOAT_FAVICON_API_IMAGE_TAG
Scraping	SCRAPEGOAT_SCRAPER_MAX_CONCURRENCY, SCRAPEGOAT_SCRAPER_DOCUMENT_MAX_SIZE
Search	SCRAPEGOAT_SEARCH_WEIGHT_VEC, SCRAPEGOAT_SEARCH_WEIGHT_FTS, SCRAPEGOAT_SEARCH_RRF_K
Uploads	SCRAPEGOAT_WEB_IMPORT_*, SCRAPEGOAT_WEBUI_IMPORT_*

For the complete list, read .env.example.

Set SCRAPEGOAT_FAVICON_API_REPLICAS=0 to stop Compose from running the bundled favicon-api container. Worker and web containers keep starting because the favicon service is an optional dependency; favicon refreshes can still use an external service when SCRAPEGOAT_FAVICON_API_URL points at one.

Backup and Restore

The bundled PostgreSQL Compose file includes scrapegoat-v3-backup under the backup and restore profiles.

Manual backup:

docker compose -f docker-compose.postgres.yml run --rm \
  --profile backup scrapegoat-v3-backup \
  backup backup --output /backups

Set backup output path:

SCRAPEGOAT_BACKUP_PATH=/backups

Reverse Proxy

Route / to the Web UI and /mcp to the MCP service. Disable buffering for MCP and allow large request bodies if you use Web UI uploads.

Minimal nginx shape:

upstream scrapegoat_web {
    server 127.0.0.1:6281;
}

upstream scrapegoat_mcp {
    server 127.0.0.1:6280;
}

server {
    listen 443 ssl;
    http2 on;
    server_name scrapegoat.example.com;

    client_max_body_size 1024M;

    location /mcp {
        proxy_buffering off;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_pass http://scrapegoat_mcp;
    }

    location / {
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_pass http://scrapegoat_web;
    }
}

See docs/infrastructure/nginx-reverse-proxy.conf for a fuller example.

Development

Install dependencies:

npm install

Run checks:

npm run typecheck
npm run lint
npm test

Build:

npm run build

Host Node starts are blocked for runtime path. Use Docker Compose for running services. Development commands remain available for tests, builds, and local inspection.

Useful docs:

• docs/ARCHITECTURE.md
• docs/setup/configuration.md
• docs/setup/installation.md
• docs/guides/basic-usage.md
• docs/guides/mcp-clients.md
• docs/guides/embedding-models.md
• docs/infrastructure/authentication.md
• docs/infrastructure/deployment-modes.md

License

MIT