Efficient Page Filters

[jajajasalu2]

Scrapy has the ability to filter out links that match a list of regular expressions.

Such a list is necessary for links that we know will not yield any patches for a vulnerability, for example links to homepages of sites. Furthermore, if a larger depth is given to the patch-finder, a lot of unnecessary crawling can take place.

A list of regular expressions efficient at filtering out these links is needed.