Adapt scraper to new Phets Web site #150
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@pavel-karatsiuba It's important you describe here what are the impact from the user perspective. You told me separatly, but not here, that at least the topics and descriptions are not scrapped anymore. But the topics are here listed for example https://phet.colorado.edu/en/simulations/build-a-nucleus/about. The description is available on the very same about page in Do we have other impacts? Otherwise do the ZIM files look like the same like the latest successfuly scrape available at https://library.kiwix.org? |
kelson42
requested changes
Nov 30, 2022
kelson42
force-pushed
the
issue-148-not-generating-zim
branch
from
84b5e30 to
7551886
Compare
kelson42
approved these changes
Dec 1, 2022
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes are touched downloading of the simulation start page. Because phet uses the async loader to load page content scrapper should use lib which can wait until the content is loaded. So in this PR used the puppeteer library to load simulations. It downloads simultaneously 5 pages. If set to more than 5 pages then phet site will respond with a 404 or a timeout exception.
Fixes #148