Skip to content

Adopt #9's html.py extensions on the shared base - #18

Open
ahoblitz wants to merge 1 commit into
MedARC-AI:mainfrom
ahoblitz:shared-base-reconciliation
Open

Adopt #9's html.py extensions on the shared base#18
ahoblitz wants to merge 1 commit into
MedARC-AI:mainfrom
ahoblitz:shared-base-reconciliation

Conversation

@ahoblitz

@ahoblitz ahoblitz commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Per review, the scrapers are out — #8, #9 and #10 rebase onto the registry from #21 and merge themselves.

What's left is review item 3: #9's html_to_markdown, byte-identical to its head, so the HTML scrapers behind it don't reapply it.

@CLAassistant

CLAassistant commented Aug 8, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@ahoblitz ahoblitz changed the title Consolidate shared scraping base changes from the open scraper PRs Consolidate shared scraping base; initial scraper integration Aug 8, 2026
@ahoblitz
ahoblitz marked this pull request as ready for review August 8, 2026 18:00
@ahoblitz
ahoblitz force-pushed the shared-base-reconciliation branch from 1f11801 to 063d9dd Compare August 10, 2026 12:54
@zndr27

zndr27 commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

My recommendation is to merge this first so the other PRs can rebase on this.

Suggested changes (see other comments in code):

  1. Drop the five scraper modules, the five test modules, and the two test fixtures. Let the downstream PRs merge that themselves.
  2. Replace the ScraperSource enum, the match block and _expand_source with one name-to-function dict, and stop init.py re-exporting per-source symbols, so adding a scraper is one line in one file.
  3. Take PR add Canadian Paediatric Society statement scraper #9's version of html.py rather than this snapshot of it, since he has already extended the same function.
  4. Fold in PR Add CCO scraper #17's ClientT change to scrape_listing_documents, so the CCO scraper does not have to edit base.py again.

Comment thread datasets/amfv_datasets/scraping/who.py Outdated
Comment thread datasets/amfv_datasets/scraping/__init__.py
Comment thread datasets/amfv_datasets/scraping/cli.py
Comment thread datasets/amfv_datasets/scraping/html.py
Comment thread datasets/amfv_datasets/scraping/base.py Outdated
@warner-benjamin

Copy link
Copy Markdown
Collaborator

All of @zndr27 suggested changes look good.

@ahoblitz
ahoblitz force-pushed the shared-base-reconciliation branch 2 times, most recently from 71fa31f to 5480c38 Compare August 23, 2026 19:08
@ahoblitz ahoblitz changed the title Consolidate shared scraping base; initial scraper integration Integrate IDSA, CPS and RCH scrapers on the shared base Aug 23, 2026
@ahoblitz
ahoblitz force-pushed the shared-base-reconciliation branch from 5480c38 to 1fce4ab Compare August 23, 2026 19:16
Takes the html_to_markdown changes from MedARC-AI#9 so its CPS scraper, and the
other HTML scrapers behind it, do not have to reapply them after merge:

- in-page anchors stay relative instead of being absolutized
- drop_numeric_citations makes citation stripping opt-out
- sup_symbol="<sup>" preserves superscripts

html.py is byte-identical to MedARC-AI#9's head. It could not be cherry-picked:
all three of MedARC-AI#9's commits also touch cps.py, and the first rewrites
__init__.py and cli.py in the pre-registry shape MedARC-AI#21 replaced.

The unregistered-source assertion in test_scraping_cli.py was pinned to
the literal "all, nice", which goes stale as soon as a source is added.
It now derives from the registry.

Co-authored-by: mkrastev <matey_krastev2@hotmail.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@ahoblitz
ahoblitz force-pushed the shared-base-reconciliation branch from 1fce4ab to 8b1d435 Compare August 23, 2026 19:34
@ahoblitz ahoblitz changed the title Integrate IDSA, CPS and RCH scrapers on the shared base Adopt #9's html.py extensions on the shared base Aug 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants