create a weak_cache in Injector

[BurnzZ]

Currently, SCRAPY_POET_CACHE can be set to True to allow scrapy-poet to save instances created by providers locally. By default, this is turned off.

I'm proposing to introduce an Injector.weak_cache that would have its instances stored in memory and be removed when the corresponding Request instance is no longer available. By default, this is turned on. The motivation for this is to allow sharing of instances between providers by default without the risk of accumulating large number of instances locally which can result in running out of disk space.

This also means that the weakref cache mechanism in ZyteApiProvider (code ref) can be removed as it can use this new .weak_cache directly from the Injector.

Other related PRs:

• add new AnyResponse web-poet#195
• support AnyResponse scrapy-plugins/scrapy-zyte-api#161
• support AnyResponse zytedata/zyte-spider-templates#28

[codecov]

Codecov Report

Merging #184 (e402a04) into master (3b2a0f7) will increase coverage by 0.07%.
The diff coverage is 100.00%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #184      +/-   ##
==========================================
+ Coverage   86.83%   86.90%   +0.07%     
==========================================
  Files          15       15              
  Lines         896      901       +5     
==========================================
+ Hits          778      783       +5     
  Misses        118      118              

Files	Coverage Δ
scrapy_poet/injection.py	98.61% <100.00%> (+0.03%)	⬆️

[Gallaecio]

I don’t fully understand the motivation behind this. When do providers need this?

[BurnzZ]

I don’t fully understand the motivation behind this. When do providers need this?

It's quite similar to having the serialized cache turned on via SCRAPY_POET_CACHE=True, but this is turned on by default and only needs the instances as long as the request exists. This is crucial to avoiding filling up the disk space. For example, in a job running in Scrapy Cloud with 1 unit, the spider only has 2.5GB of disk space. Assuming all the dependencies for a request (e.g. HttpResponse, Product, etc) amounts to 0.1 MB, we may run out of available disk space in just 25,000 requests.

class SomePage(BasePage):
    response: AnyResponse
    http_response: HttpResponse

Previously, the example above, results in 422 error, since HttpResponse has been created by HttpResponseProvider earlier but then it's not available to ZyteApiProvider later on. This causes the Zyte API request to not have any page type request since HttpResponse isn't declared in HttpResponseProvider.provided_classes: