fix(crawler): crawl llms.txt index links instead of ingesting them as content - #2
Merged
Merged
Conversation
… content
An llms.txt INDEX (`/llms.txt`) is a curated list of links to the real docs,
whereas `/llms-full.txt` is the concatenated documentation prose. The crawler
treated both identically: `discover()` falls back from the full file to the
index, and `split_llms_full()` was run on whatever it got. The only guard
(`_looks_like_llms_txt`) just checks for a leading `# `, which an index also
has. Result: for any source whose site serves only `/llms.txt` — or whose
`/llms-full.txt` exceeds discover()'s 10MB cap and falls back to the index — the
corpus became a list of links with one-line descriptions instead of actual
documentation content ("the crawler only returns metadata").
Reproduced against docs.anthropic.com: its /llms-full.txt is 24MB (rejected by
the 10MB cap), so discovery fell back to the 56KB /llms.txt index and ingested
the link list as content.
Fix — distinguish the two file types and handle the index as a discovery source:
- llms_txt.looks_like_index(): an index is dominated by markdown link bullets
(>=3 bullets AND a majority of content lines), a full file is prose/code.
- llms_txt.parse_llms_index(): extract the linked page URLs (absolute + relative
resolved, deduped, code-fenced links ignored).
- crawler.crawl(): when discovery returns an index, extract its URLs and crawl
each as a normal HTML page (filtered same-host + include/exclude prefixes,
capped to max_pages, still re-validated per-URL in _visit), so pages run
through extract.extract and yield real body text. Full-content files keep the
existing section-splitting fast path. "only" mode crawls an index's links
rather than yielding nothing.
Also gives index sources proper per-page ETag change detection on re-sync (each
page fetched via _visit), instead of the all-or-nothing index conditional.
Tests: unit coverage for looks_like_index/parse_llms_index, and crawl-level
tests proving an index is fetched as HTML (not ingested as metadata), links are
scope-filtered, "only"+index crawls the links without BFS, and a real
/llms-full.txt still yields markdown sections. Full ingestion suite: 258 passed,
44 skipped (DB-gated), 1 pre-existing WSL2-only failure unrelated to this change.
discover()'s max_bytes guard previously downloaded the entire file with client.get(...).content and only then checked its length — so an oversized /llms-full.txt (docs.anthropic.com serves a 24MB one) cost a full 24MB transfer just to be rejected before falling back to the index. Stream the body instead and stop reading as soon as it crosses max_bytes, so an oversized file is abandoned mid-transfer. Behavior is otherwise unchanged: non-200, empty, oversize, and error cases still skip the candidate, and a valid small file is returned as before. Extracted into a _fetch_capped() helper. Tests: assert the download aborts after a few KB (not the whole body) once the cap is exceeded, and that a body exactly at the cap boundary is still accepted.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An llms.txt INDEX (
/llms.txt) is a curated list of links to the real docs,whereas
/llms-full.txtis the concatenated documentation prose. The crawlertreated both identically:
discover()falls back from the full file to theindex, and
split_llms_full()was run on whatever it got. The only guard(
_looks_like_llms_txt) just checks for a leading#, which an index alsohas. Result: for any source whose site serves only
/llms.txt— or whose/llms-full.txtexceeds discover()'s 10MB cap and falls back to the index — thecorpus became a list of links with one-line descriptions instead of actual
documentation content ("the crawler only returns metadata").
Reproduced against docs.anthropic.com: its /llms-full.txt is 24MB (rejected by
the 10MB cap), so discovery fell back to the 56KB /llms.txt index and ingested
the link list as content.
Fix — distinguish the two file types and handle the index as a discovery source:
(>=3 bullets AND a majority of content lines), a full file is prose/code.
resolved, deduped, code-fenced links ignored).
each as a normal HTML page (filtered same-host + include/exclude prefixes,
capped to max_pages, still re-validated per-URL in _visit), so pages run
through extract.extract and yield real body text. Full-content files keep the
existing section-splitting fast path. "only" mode crawls an index's links
rather than yielding nothing.
Also gives index sources proper per-page ETag change detection on re-sync (each
page fetched via _visit), instead of the all-or-nothing index conditional.
Tests: unit coverage for looks_like_index/parse_llms_index, and crawl-level
tests proving an index is fetched as HTML (not ingested as metadata), links are
scope-filtered, "only"+index crawls the links without BFS, and a real
/llms-full.txt still yields markdown sections. Full ingestion suite: 258 passed,
44 skipped (DB-gated), 1 pre-existing WSL2-only failure unrelated to this change.