Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
356: Try to fix Scrapy ERROR: Spider error processing r=alallema a=alallema Currently when running doc-scraper this error occur: ```sh 2023-03-14 12:10:56 [scrapy.core.scraper] ERROR: Spider error processing <GET https://docs.meilisearch.com/learn/advanced/geosearch.html> (referer: https://docs.meilisearch.com/sitemap.xml) Traceback (most recent call last): File "/Users/amelielallemand/.local/share/virtualenvs/docs-scraper-vWaWSN46/lib/python3.9/site-packages/twisted/internet/defer.py", line 892, in _runCallbacks current.result = callback( # type: ignore[misc] File "/Users/amelielallemand/Projects/meili/repo/docs-scraper/scraper/src/documentation_spider.py", line 170, in parse_from_sitemap self.add_records(response, from_sitemap=True) File "/Users/amelielallemand/Projects/meili/repo/docs-scraper/scraper/src/documentation_spider.py", line 151, in add_records records = self.strategy.get_records_from_response(response) File "/Users/amelielallemand/Projects/meili/repo/docs-scraper/scraper/src/strategies/default_strategy.py", line 44, in get_records_from_response records = self.get_records_from_dom(response.url) File "/Users/amelielallemand/Projects/meili/repo/docs-scraper/scraper/src/strategies/default_strategy.py", line 67, in get_records_from_dom sys.exit('DefaultStrategy.dom is not defined') SystemExit: DefaultStrategy.dom is not defined ``` This PR try to fix it Co-authored-by: alallema <amelie@meilisearch.com>
- Loading branch information