treat a failed fetch (e.g. socket timeout) of robots.txt the same way…

[nlevitt]

… as http errors: full allow

Archive-it has been having problems twitter images hosted at pbs.twimg.com. Sometimes fetching https://pbs.twimg.com/robots.txt results in a socket timeout, while fetching content from the domain does not. With this change, we are able to capture the content.

Note that if the crawl is configured to ignore robots using the "calculate robots only" setting,
as recommended at https://groups.yahoo.com/neo/groups/archive-crawler/conversations/messages/8005
currently a fetch failure of robots.txt precludes any further fetches from the domain.

[anjackson]

The policy looks fine to me. The differences between this and GoogleBot seem reasonable given the difference between the indexing and archiving use cases.

That said, this is one of the areas of the H3 codebase I'm less familiar with, so I can't really look at the code and know that it precisely implements the policy.