Feat/optional max pages - #3
Merged
Merged
Conversation
A source had to specify max_pages. Make it optional across both services: omitting it (or passing null) means "no page limit" — crawl all in-scope pages, still bounded by same-host + include/exclude prefix scoping. - config.SourceConfig / retrieval.ProposedSourceConfig: max_pages is now `int | None = Field(default=None, gt=0)` — optional, and positive only when provided. The DB column was already NULLABLE; NULL now means unlimited (schema comment updated). - crawler.crawl(): when max_pages is None, use sys.maxsize internally as the cap (page_cap) so the existing sitemap/BFS/llms bound checks and slicing need no special-casing. - propose_doc_source MCP tool + retrieval.propose_source: max_pages optional (default None); docstring updated (omit for no limit; positive if given). - admin: form field no longer required (blank = no limit); dropped the None -> 100 fallbacks in _record_to_config and sources_repo._row_to_record so an unset limit stays unlimited instead of silently becoming 100; index view shows "unlimited". Tests: omitting max_pages is accepted and yields None (config + ProposedSource); a None-limit sitemap crawl fetches all in-scope pages; propose with max_pages=None writes NULL; zero/negative still rejected. Updated the two tests that asserted max_pages was required.
Drop the max_bytes cap on llms-full.txt / llms.txt discovery entirely. The cap rejected legitimately large full-content files (docs.anthropic.com's /llms-full.txt is ~24MB), forcing a fallback to the thin index and, when that index links off-host, an empty crawl. Now the full file is fetched however large it is. - discover(): no max_bytes parameter; no size check. - _fetch_capped (streaming/abort) replaced by a plain _fetch_body GET. - Removed the corresponding cap tests; added one asserting a large body is returned in full (no truncation). Sources are human-approved before any crawl, so an unbounded fetch here is bounded by that approval gate.
ulielitay
force-pushed
the
feat/optional-max-pages
branch
from
July 22, 2026 17:20
9189316 to
9c6aa83
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.