Guidance on downloading whole websites

[BradKML]

Intended goal: Downloading whole "self-help" websites to hot-plugging individual page entries into LLMs, and use it to cluster webpage content.

Requirement:

1. Be able to scrape websites quickly (since most of the sites uses little to no JS, open_in_browser would not make much sense, similar to how bs4 is always faster than selenium for other scraper solutions)
2. Scrape them in a stealthy manner (bypass_robots may be an issue, but delay=None would be good
3. Having ability to scale and centralize (threaded between multiple IPs/computers would be better than just concurrently downloading from multiple websites)
4. Be able to read scraped pages as individual units for HTML extractors (e.g. Trafilatura) and language libraries (e.g. SBERT or SpaCy-USE)

Questions:

1. Is it possible to "smell" if the server might block scrapers? Also, what are the ways to work around it (e.g. user agents)?
2. How are "no-cobbler" like features used in this context for suspended and resumed downloads? https://www.linuxjournal.com/content/downloading-entire-web-site-wget
3. How can one prevent whole-site scraping to leech into other unrelated websites? https://gist.github.com/crittermike/fe02c59fed1aeebd0a9697cf7e9f5c0c
4. How can individual files (either file path OR list-of-strings) be listed as output for piping into other tools?
5. How can this process be broken down into concurrent parts (between websites/servers), asynchronous procedures (thread-like within the same website), or pauseable parts?
6. What would happen if you want to capture a blog with daily updated posts, and have this operation run every month? (likely the multimedia URLs and files would not change much, not worth duplicating the work)

[rajatomar788]

Is it possible to "smell" if the server might block scrapers? Also, what are the ways to work around it (e.g. user agents)?

No there is no way anyone can smell blocking just like in the messenger apps.

But you can definitely change the user-agent string

from pywebcopy.config import get_config
config = get_config(url, project_folder, project_name, bypass_robots, debug, delay, threaded)
webpage = config.create_page()

# here you have the ready webpage instance to do anything
webpage.session.header = anything
webpage.get(url)
    
webpage.save_complete(pop=open_in_browser)

[BradKML]

What about checking the robot file and see if they have a "no scrape" policy (probably the site have other cloudFlare-esque roadblocks too)? Would that work?

[rajatomar788]

It does check the robot file, that is what bypass_robots option is for.

[BradKML]

More in the sense of adjusting crawling speed/tactics based on the robot file rather than merely quitting. As a side option, play by the robot rules, but have multiple IPs distributing the load to increase efficiency (multiple thread on a single IP seems like a different issue).
How hard does bypass_robots go, do you have to adjust pause time manually?

[rajatomar788]

Multiple IPs and all that is the job of the user not the library. How can library generate public IP addresses without having complete control of user device and network. Would you like that?

[BradKML]

Assuming that the user has access to multiple IPs, how could the software adapt to it?

[rajatomar788]

How are "no-cobbler" like features used in this context for suspended and resumed downloads? https://www.linuxjournal.com/content/downloading-entire-web-site-wget

Yes there is this equivalent config key overwrite which you could set to False to prevent redownloading of files.

[BradKML]

Thanks for the note, in this case there was an issue by someone to do conditional check for overwrites through checking headers.
#41
Some other people proposed either checksum (very fragile to ad and dynamic websites) or article metadata extraction (see example, but platform-dependent).

[rajatomar788]

As you said it is very case specific and bandwidth heavy so not worth trouble in one off downloads.

[BradKML]

When it is dealing with whole blog archiving (plus updating), I do think it could be of use.

[rajatomar788]

You can write a mod pull request but right now it does its job light and clean.

[rajatomar788]

How can one prevent whole-site scraping to leech into other unrelated websites? https://gist.github.com/crittermike/fe02c59fed1aeebd0a9697cf7e9f5c0c

Well you have to manually blacklist or whitelist domains in the session object as of pywebcopy 7.0.2, this is a design limitation.

[BradKML]

Would kindly request an example of using it since the documentation is a bit sparse. Trying to hop from WGet to PyWebCopy take a bit of a learning curve.

[rajatomar788]

from pywebcopy.config import get_config
config = get_config(url, project_folder, project_name, bypass_robots, debug, delay, threaded)
webpage = config.create_page()
 

webpage.session.domain_blacklist = your_list[]

webpage.get(url)
webpage.save_complete(pop=open_in_browser)

[rajatomar788]

How can individual files (either file path OR list-of-strings) be listed as output for piping into other tools?

The record of all the urls along with their downloaded path is stored in a pywebcopy.schedulers.RecentOrderedDict in pywebcopy.schedulers.Scheduler objects index attribute.

from pywebcopy.config import get_config
config = get_config(url, project_folder, project_name, bypass_robots, debug, delay, threaded)
webpage = config.create_page()

# here you have the ready webpage scheduler to hook up you custom version of a RecentOrderedDict 

webpage.scheduler.index = your_callback_wrapped_RecentOrderedDict


webpage.get(url)
webpage.save_complete(pop=open_in_browser)

[BradKML]

Does PyWebCopy run on bs4-like (see also scrapy) browserless scraping or Selenium-like browsered scraping? Assuming the websites does not have enough JS to block off article downloads, would pop=False be more suitable?

[rajatomar788]

Pywebcopy uses lxml library for browser less scraping. Pop just opens the first page after completion so i don't know.

[BradKML]

If it is LXML it would operate similar to BeautifulSoup. In that case are there code samples for doing the same thing but for whole websites? Following this documentation only downloaded one page instead of the intended effect.

[rajatomar788]

That documentation link is from 2019. Use the latest Readme file documentation.

[BradKML]

Got it. Thanks for the note, hope to see a more comprehensive doc soon.

[rajatomar788]

How can this process be broken down into concurrent parts (between websites/servers), asynchronous procedures (thread-like within the same website), or pauseable parts?

You have the threaded=True config option.

[rajatomar788]

What would happen if you want to capture a blog with daily updated posts, and have this operation run every month? (likely the multimedia URLs and files would not change much, not worth duplicating the work)

What happens when you visit a social media site daily or even hourly? Nothing.