historyarchive: Harden the ArchivePool to be robust against errors.

[Shaptic]

What problem does your feature solve?

The current implementation of ArchivePool uses a random archive for each individual call, bubbling up any errors to the caller. It does an initial filtering on creation, but it does not handle any errors that occur during usage.

What would you like to see?

The ArchivePool should intelligently deal with problematic archives:

• evaluate whether rand or round-robin makes more sense, in general
• if there is an error when making a call on a certain archive, it should retry with another
  • this should have an intelligent back-off mechanism: i.e. retry immediately if it's a new archive, but back-off if it's the same or a problematic archive
• the pool should track problematic archives, i.e. ones that give errors with high frequency/likelihood
  • should it remove problematic archives?
  • keep in mind how many remain in the pool after removal
• consider edge cases the whole time, e.g. just one archive in the pool

What alternatives are there?

This can be handled at the caller level, but it makes sense to do this in the pool itself, if possible.

Prior work: #3402