Testing xpaths

[jajajasalu2]

Current unit tests use HTML files stored in tests/mocks. While using HTML mocks is helpful in testing the parsing logic, there is no way to check if a current online response is parsed correctly by the xpaths. Testing real time data would enable testing the xpaths in patchfinder.entrypoint.

A solution like https://github.com/kevin1024/vcrpy is optimal. vcrpy, unfortunately, does not currently support Scrapy's networking framework Twisted.

Thus, an efficient method to test if a current response is parsed properly is required.

[jajajasalu2]

One possibility for this:

Use requests/urllib in test cases and create fake Scrapy responses from these responses. Decorate the test methods with vcrpy, so that response bodies can be taken from VCR cassettes later, and updated when necessary. This allows to "switch the mock on and off".