Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fixes Scrapy ERROR: Spider error processing #356

Merged
merged 1 commit into from
Mar 14, 2023
Merged

Fixes Scrapy ERROR: Spider error processing #356

merged 1 commit into from
Mar 14, 2023

Conversation

alallema
Copy link
Contributor

Currently when running doc-scraper this error occur:

2023-03-14 12:10:56 [scrapy.core.scraper] ERROR: Spider error processing <GET https://docs.meilisearch.com/learn/advanced/geosearch.html> (referer: https://docs.meilisearch.com/sitemap.xml)
Traceback (most recent call last):
  File "/Users/amelielallemand/.local/share/virtualenvs/docs-scraper-vWaWSN46/lib/python3.9/site-packages/twisted/internet/defer.py", line 892, in _runCallbacks
    current.result = callback(  # type: ignore[misc]
  File "/Users/amelielallemand/Projects/meili/repo/docs-scraper/scraper/src/documentation_spider.py", line 170, in parse_from_sitemap
    self.add_records(response, from_sitemap=True)
  File "/Users/amelielallemand/Projects/meili/repo/docs-scraper/scraper/src/documentation_spider.py", line 151, in add_records
    records = self.strategy.get_records_from_response(response)
  File "/Users/amelielallemand/Projects/meili/repo/docs-scraper/scraper/src/strategies/default_strategy.py", line 44, in get_records_from_response
    records = self.get_records_from_dom(response.url)
  File "/Users/amelielallemand/Projects/meili/repo/docs-scraper/scraper/src/strategies/default_strategy.py", line 67, in get_records_from_dom
    sys.exit('DefaultStrategy.dom is not defined')
SystemExit: DefaultStrategy.dom is not defined

This PR try to fix it

@alallema alallema requested a review from bidoubiwa March 14, 2023 11:17
@alallema alallema force-pushed the patch-Spider_error_processing branch from f471657 to 4c9514d Compare March 14, 2023 11:28
bidoubiwa
bidoubiwa approved these changes Mar 14, 2023
View reviewed changes
Copy link
Contributor

@bidoubiwa bidoubiwa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good work

@alallema
Copy link
Contributor Author

bors merge

@meili-bors
Copy link
Contributor

meili-bors bot commented Mar 14, 2023

Build succeeded:

@meili-bors meili-bors bot merged commit 4df835a into main Mar 14, 2023
@meili-bors meili-bors bot deleted the patch-Spider_error_processing branch March 14, 2023 12:30
@alallema alallema added the bug Something isn't working label Mar 14, 2023
@alallema alallema changed the title Try to fix Scrapy ERROR: Spider error processing Fixes Scrapy ERROR: Spider error processing Mar 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bidoubiwa bidoubiwa bidoubiwa approved these changes

Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@alallema @bidoubiwa