Add exponential back off mechanism while retrying transport errors to metrics store

[dliappis]

The solution introduced in 7724526 to retry certain TransportErrors while writing to the metrics store is not sufficient as there are only 3 retries, 3 seconds each.

We need an exponential back off mechanism instead with a larger amount of retries to increase resiliency for errors.

Note that we still need to investigate how this blocks which thread, but will track it in a separate issue.

[dliappis]

cc @dakrone