Skip to content

fix(fetchers): enforce policy on PubMed API requests#170

Open
chaliy wants to merge 1 commit into
mainfrom
2026-07-24-fix-pubmed-fetcher-url-policy-bypass
Open

fix(fetchers): enforce policy on PubMed API requests#170
chaliy wants to merge 1 commit into
mainfrom
2026-07-24-fix-pubmed-fetcher-url-policy-bypass

Conversation

@chaliy

@chaliy chaliy commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Motivation

  • A specialized PubMed/PMC fetcher rewrites user-facing article URLs to hardcoded API endpoints but sent those synthesized URLs to the transport without re-applying host/port/allow/block policy, enabling an outbound policy bypass.
  • Operators expect configured outbound restrictions (allow_prefixes, block_prefixes, blocked_hosts, allowed_ports) to govern every fetchkit network hop, including secondary API requests from specialized fetchers.

Description

  • Introduce validate_url_policy in crates/fetchkit/src/fetchers/mod.rs to centralize host, port, allow-prefix, and block-prefix validation and reuse it for registry dispatch via validate_and_find_fetcher.
  • Apply validate_url_policy(&api_url, options)? in crates/fetchkit/src/fetchers/pubmed.rs before calling transport_request so synthesized PubMed/PMC API URLs are validated against the same policy.
  • Add a regression #[tokio::test] in crates/fetchkit/src/fetchers/pubmed.rs that uses a custom HttpTransport to assert a blocked API URL returns FetchError::BlockedUrl and the transport is never invoked.
  • Update specs/fetchers.md to require that specialized fetchers which rewrite matched URLs must apply the configured URL policy before handing requests to the transport.

Testing

  • Ran cargo test -p fetchkit pubmed -- --nocapture and all pubmed-related unit tests (including the new regression test) passed.
  • Ran cargo clippy -p fetchkit --all-targets -- -D warnings with no warnings.
  • Ran cargo fmt --all and updated specs/fetchers.md accordingly; all checks included in the developer workflow succeeded locally.

Codex Task

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant