how to use

1. install requirements

2. create proxies.txt and put proxies to that
   (format)
   no auth : http://localhost:1111
   auth : http://user:pass@localhost:1111
   
   

   http -> http://
   socks4 -> socks4://
   socks5 -> socks5h://
   

3. replace website url in crawler.py with which you want to clone

4. replace target folder in crawler.py with where you want to save

5. run crawler.py
   linux : python3 crawler.py
   windows : py crawler.py

requirements

runtime : python3
pip : requests, bs4, pysocks (only if you use socks proxy)
other : torproxy (if you want to clone file server which on tor network