Explore use of Scrapy contracts for lightweight testing of spiders

[jpmckinney]

https://docs.scrapy.org/en/latest/topics/contracts.html

We don't need to closely test the spiders, but it would be useful to be able to run scrapy check and quickly find out which spiders are broken in obvious ways (e.g. can't even complete the first request).

This would make it possible to prioritize fixes to spiders, before there is urgency, e.g. it's Monday, an data analyst needs to do an analysis by Friday, the spider is broken, it takes two days for availability to free up to fix it, it takes another day to run, and the analyst is left with little time.

[jpmckinney]

We now run all (relevant) spiders regularly in the data registry, where once we close open-contracting/data-registry#29, we'll be able to monitor whether a spider is breaking or behaving differently.

Having reviewed Scrapy's contracts feature, I'm not sure how far it would get us (e.g. it can be used to check if at least one item is returned – but spiders can break in all kinds of ways, and it's hard to think of what custom contracts to write).