warcio

Here are 2 public repositories matching this topic...

A scalable, modular, and production-ready Python web crawler framework with multi-process support, domain-specific crawling, and robust data storage.

python data multiprocessing production crawling ready rahul iit-gandhinagar eka-curator rahul-khichar warcio

Command-line tool that reports what's inside a WARC file as JSON: record types, hosts, HTTP status codes, MIME types, and any failed captures.

python cli warc web-archiving digital-preservation webarchive warcio

Add a description, image, and links to the warcio topic page so that developers can more easily learn about it.

To associate your repository with the warcio topic, visit your repo's landing page and select "manage topics."