Web Crawler

A web crawler written in Python 3 that is easy to use and follows politeness policies.

Usage

Prepare

Modify seedfile.txt to set start point for the crawler.

Run

Run

$ python crawler.py start

to start crawling, press CTRL+C to terminate the crawler.

Run

$ python crawler.py restart

to restart from the last exit point.