Queue manager

Queue manager is central part of Heroshi

It accumulates URLs to crawl, and gives list of them to workers on request.
It is a WSGI application run by Spawning.

Of course, it would be better to do storage requests asynchronously too. Rewrite of manager to eventlet is in progress.

Manager workflow:

1. wait for connection from worker
2. process worker’s requests

A request may be either of (see API):

• get URLs to crawl. For response, manager composes a new list of Link to crawl (see below).
• report single URL crawling result. Manager puts result into storage (see below) and responds with 200 OK.

Results processing

Incoming new links (those found on pages) are stored into storage at once.

Manager goes to storage to

• get list of URLs visited long time ago to keep database updated
• put crawling results from workers

Link list composition

List of Links to crawl is composed according to these rules:

• new URLs (never visited before) before already visited
• really old URLs before new. The ‘really old’ margin is configurable, for example, it would be 50 days.
• in-memory list of new-links before accessing storage
• ‘correct’ servers before ‘bad’. Full domain name identifies server.
• each Link is timestamped when it is ‘given-to-crawling’. See Crawling timeout below

Update: these rules currently are not applied. At this time manager just gives not crawled yet URLs to workers.