Google-webcrawler.txt

How to design a web crawler like Google?

A web crawler goes to a website and crawls all links and indexes, like Google, so that they can later appear in a search result. A crawler can also be used for searching a particular file in a set of directories. But be sure to ask yourself about some of the major challenges.