webcrawler

CLI web crawler with SPA/AJAX support powered by Crawl4AI.

Crawls web pages (including JavaScript-heavy SPAs) and produces LLM-optimized Markdown — clean, focused content with noise (nav bars, ads, footers) automatically removed.

Features

• LLM-optimized Markdown — uses PruningContentFilter by default to strip boilerplate and produce clean fit_markdown; optional BM25 query-based filtering for topic-focused extraction
• Recursive site crawling — site command follows internal links with configurable depth, page limits, and path include/exclude patterns
• URL discovery — map command rapidly discovers all internal URLs on a site without saving content
• Token-optimized output — --compact flag compresses Markdown for minimal LLM token usage
• Content cleanup — automatically merges split-character text animations (GSAP SplitText, Framer Motion) and excludes cookie consent banners
• SPA/AJAX support — full-page scrolling, JavaScript execution, wait conditions for dynamic content
• Multiple output formats — filtered Markdown (primary), citations with references, raw HTML, cleaned HTML, screenshots, PDFs, link extraction
• Batch crawling — crawl multiple explicit URLs concurrently with batch command
• Anti-bot capabilities — simulate user interactions, magic mode, overlay removal
• Configurable browser — headless/headed, proxy, custom user agent, viewport size

Installation

# Create virtual environment
python3 -m venv .venv
source .venv/bin/activate

# Install the package
pip install -e .

# Install browser dependencies (required on first run)
crawl4ai-setup

Usage

Single URL crawl

# Basic crawl (saves Markdown to a URL-derived folder)
webcrawler crawl https://example.com

# Crawl a SPA with full-page scroll and screenshot
webcrawler crawl https://spa-app.com --scan-full-page --screenshot

# Wait for dynamic content before capturing
webcrawler crawl https://spa-app.com --wait-for "css:.content-loaded"

# Execute JavaScript after page load
webcrawler crawl https://example.com --js "document.querySelector('.load-more')?.click()"

# Custom output directory with all formats
webcrawler crawl https://example.com -o my_output --html --cleaned-html --screenshot --pdf --links

# Use a visible browser for debugging
webcrawler crawl https://example.com --no-headless

# Anti-bot mode with proxy
webcrawler crawl https://example.com --magic --simulate-user --proxy http://user:pass@host:port

# Focus extraction on a specific topic (BM25 filtering)
webcrawler crawl https://example.com --filter-query "machine learning"

# Stricter pruning, no links in markdown output
webcrawler crawl https://example.com --prune-threshold 0.6 --ignore-links

# Save both filtered and raw markdown for comparison
webcrawler crawl https://example.com --raw-markdown

Site crawl (recursive)

# Crawl an entire site (default: depth 2, max 50 pages)
webcrawler site https://example.com

# Deep crawl with limits
webcrawler site https://docs.example.com --max-depth 3 --max-pages 100

# Only crawl specific paths
webcrawler site https://example.com --include-path "/blog/*" --exclude-path "/blog/drafts/*"

# Token-optimized output for LLM consumption
webcrawler site https://example.com --compact --ignore-links --ignore-images

URL discovery (map)

# List all internal URLs found on a page
webcrawler map https://example.com

# Follow links to discover deeper URLs
webcrawler map https://example.com --max-depth 2

# Output as JSON, save to file
webcrawler map https://docs.example.com --include-path "/api/*" --json -o urls.json

Batch crawl (multiple explicit URLs)

webcrawler batch https://example.com https://other.com
webcrawler batch https://example.com https://other.com -o results --screenshot

CLI reference

webcrawler crawl --help
webcrawler site --help
webcrawler map --help
webcrawler batch --help

Output Files

By default, each crawl produces LLM-optimized output:

File	Description
content.md	Primary — filtered Markdown with boilerplate removed (fit_markdown)
content_citations.md	Markdown with reference-style link citations at the bottom
references.md	Link reference index
content_raw.md	Unfiltered raw Markdown (only with --raw-markdown)
metadata.json	Crawl metadata (URL, status, errors)

Optional outputs with flags: screenshot.png, page.pdf, page.html, cleaned.html, links.json

The site command additionally produces a sitemap.json at the top level with crawl results for all discovered pages.

Key CLI Options

Content Filtering (LLM quality)

Option	Description
--filter-query TEXT	BM25 query to focus on a specific topic (overrides pruning)
--prune-threshold FLOAT	Pruning filter strictness, 0.0-1.0 (default: 0.48)
--ignore-links	Remove hyperlinks from Markdown
--ignore-images	Remove image references from Markdown
--raw-markdown	Also save unfiltered raw Markdown
--word-threshold INT	Min word count for content blocks (default: 10)

Content Cleanup

Option	Description
--compact	Compact markdown for minimal LLM token usage (collapse whitespace)
--merge-split-chars / --no-merge-split-chars	Merge single-character text animation spans (e.g. GSAP SplitText, Framer Motion) into readable text (default: enabled)
--exclude-cookie-banners / --no-exclude-cookie-banners	Exclude cookie consent container elements from extraction (default: enabled)

Site Crawl Options (site command)

Option	Description
--max-depth INT	Maximum link-following depth (default: 2)
--max-pages INT	Maximum pages to crawl (default: 50)
--include-path PATTERN	Glob pattern for URL paths to include (repeatable)
--exclude-path PATTERN	Glob pattern for URL paths to exclude (repeatable)

Map Options (map command)

Option	Description
--max-depth INT	Levels of links to follow for discovery (default: 0 = seed only)
--include-path PATTERN	Glob pattern for URL paths to include (repeatable)
--exclude-path PATTERN	Glob pattern for URL paths to exclude (repeatable)
--json	Output as JSON instead of plain list
-o, --output FILE	Save URL list to file instead of stdout

Browser & Navigation

Option	Description
-o, --output	Output directory (default: URL-derived name)
--headless / --no-headless	Headless browser mode (default: headless)
--browser	Browser engine: chromium, firefox, webkit
--scan-full-page	Scroll full page for lazy-loaded content
--js	JavaScript to run after load (repeatable)
--wait-for	Wait condition: css:<selector> or js:<expression>
--screenshot	Capture page screenshot
--pdf	Capture page as PDF
--html	Save raw HTML
--links	Save extracted links as JSON
--magic	Enable anti-bot bypass
--simulate-user	Simulate human-like interaction
--proxy	HTTP proxy URL
--check-robots-txt	Respect robots.txt
-v, --verbose	Verbose output

Project Structure

webcrawler/
├── src/webcrawler/
│   ├── __init__.py
│   ├── cli.py          # Click CLI commands
│   ├── crawler.py      # Core crawl logic (Crawl4AI wrapper)
│   └── utils.py        # URL helpers, path matching, filesystem utilities
├── pyproject.toml
└── README.md

Development

pip install -e ".[dev]"
ruff check src/
pytest

License

MIT