docparser

Terminal-menu SDK documentation extractor.

Pipeline: fetch → extract → emit

Fetches pages from documentation sites, strips navigation noise, converts HTML to clean markdown, and emits timestamped .md files ready to paste into Claude or any LLM context window.

---

Quick start

uv sync                        # creates .venv and installs all dependencies
uv run playwright install chromium   # only needed for js_render receipts

uv run docparser.py

Flags:

--receipts PATH      Path to receipts JSON file   (default: ./receipts.json)
--output-dir PATH    Directory for .md output     (default: ./doc_output)
--no-color           Disable ANSI colour output

---

Receipts

A receipt is a named profile that tells docparser how to fetch and extract a particular documentation site. Receipts live in receipts.json — never in the Python source.

Bundled examples

The repo ships with two prefilled receipts in receipts.json: Memvid (memvid-python) and LM Studio (lmstudio-python) — two of my favourite projects from other creators, picked as real-world examples of what docparser can cover:

• memvid-python — static Mintlify docs; shows the -probe workflow end-to-end (the probe confirmed #content as the winning selector).
• lmstudio-python — JS-rendered Next.js site; beyond what a static -probe can see, so it needed "js_render": true plus "markdown_passthrough": true for its Copy-as-Markdown button.

Delete them or keep them — they're just regular receipts.

Edit directly

Open receipts.json in any editor, add or modify a receipt, then run -reload in the REPL to pick up the changes without restarting.

Edit interactively

Command	What it does
-list	List all receipts (key, language, last-fetched)
-show	Pretty-print a single receipt's full JSON
-add	Build a new receipt step-by-step
-edit	Patch one field of an existing receipt
-delete	Remove a receipt after confirmation
-reload	Re-read receipts.json from disk
-export	Dump all receipts to a shareable file
-import	Merge receipts from an external file

Receipt schema

{
  "my-sdk-python": {
    // Required
    "name":      "My SDK (Python)",          // human label
    "language":  "python",                   // used in code blocks and header
    "urls":      ["https://docs.example.com/overview", "…"], // fetched in order
    "selectors": ["#content", ".prose", "article", "main", "body"],

    // Optional
    "strip_tags":           ["nav", "footer", "header", "script", "style"],
    "section":              null,            // restrict to content after this H2 text
    "js_render":            false,           // use Playwright instead of requests
    "markdown_passthrough": false,           // treat extracted <pre> as raw markdown
    "notes":                "",              // free-text notes about the site

    // Managed automatically
    "last_fetched": null,                    // ISO date of last successful parse
    "last_output":  null                     // filename of last emitted .md
  }
}

---

Commands

─── Receipt management ───────────────────────────────────────────
-list           List all registered receipts
-show           Pretty-print a single receipt's full JSON
-add            Add a new receipt interactively
-edit           Edit a single field of an existing receipt
-delete         Delete a receipt after confirmation
-reload         Re-read receipts.json from disk
-export         Export all receipts to a file for sharing
-import         Merge receipts from an external file

─── Parsing ──────────────────────────────────────────────────────
-parse          Select a receipt and parse it (prompts for key)
-parse:<key>    Parse receipt <key> directly
-url            Fetch a custom URL using an existing receipt as template

─── Discovery ────────────────────────────────────────────────────
-probe          Probe a URL: find matching selectors and H2 structure
                Prompts to save findings as a new receipt
-probe-js       Probe a JS-rendered URL with headless Chromium
                Same report as -probe, plus Copy-as-Markdown detection

─── Utility ──────────────────────────────────────────────────────
-out            Show output directory and list emitted .md files
-help           Show full command reference
-exit           Quit

---

Workflow for a new site

1. -probe    → paste the docs URL, see which selectors match and what H2s exist
               answer "y" to save as a draft receipt
               (nothing matched? the site is probably JS-rendered — retry with -probe-js)
2. -edit     → refine urls, strip_tags, section filter if needed
3. -parse    → generates the .md file in doc_output/

---

JS-rendered sites

Some documentation sites (Next.js, React SPAs) return an empty shell to plain requests. Set "js_render": true in the receipt to use headless Chromium via Playwright.

To discover selectors on such a page, use -probe-js instead of -probe: it renders the page in Chromium first, runs the same selector report over the rendered DOM, and detects whether a Copy-as-Markdown button exists (a saved receipt then pre-fills js_render and markdown_passthrough for you).

If the site has a "Copy as Markdown" button (some Mintlify-hosted docs have one), also set "markdown_passthrough": true — docparser will click the button and read the clipboard directly, producing the cleanest possible output.

---

Output

Files are written to doc_output/ (or --output-dir) as:

{receipt_name}_{YYYYMMDD_HHMMSS}.md

Each file contains:

• A header block with language, fetch timestamp, and source URLs
• One <!-- SOURCE: url --> section per fetched page
• The receipt's notes field if set

---

Dependencies

Package	Purpose
requests	Static page fetching
beautifulsoup4	HTML parsing
markdownify	HTML → Markdown conversion
playwright	JS-rendered pages (optional)

Managed with uv — pinned in pyproject.toml, locked in uv.lock.

---

License

MIT © ChristofMilius