Subgraph failure retry fuzzing

[YaroShkvorets]

At Pinax we see a pattern when every 30 minutes there is a large spike of requests to Firehose.

Most likely, the problem is that one or two of the indexers appear to have several broken subgraphs and every 30 minutes they wake up all together and start bombarding the firehose, sometimes overloading the network. We need to find a way to smoothen the line here.

There is a question why graph node can't detect these failures and stop syncing these subgraphs that appear to be failing deterministically, but that's a question for another issue.

30 minutes value here comes from the retry backoff ceiling here

To solve this, I propose adding fuzz_factor to ExponentialBackoff struct. When inited with fuzz_factor, the delay will be randomly selected within delay*(1-fuzz_factor) and delay*(1+fuzz_factor) time.

I.e. if fuzz_factor=0.1 and next delay is calculated to be 10 seconds, it will in fact be, between 9 and 11 seconds.

Retry fuzz factor can be set via ENV variables by the indexer similar to GRAPH_SUBGRAPH_ERROR_RETRY_CEIL_SECS here

If this seems like a good idea I can submit a PR for that feature or listen for other possible solutions.

[maoueh]

Seems like a good idea to add some randomness to the backoff element. Google SRE guide recommend using jitter for retry element, see https://cloud.google.com/blog/products/gcp/how-to-avoid-a-self-inflicted-ddos-attack-cre-life-lessons (point #2 specifically)

As for the fuzz naming, I think most library uses instead the term jitter, at least that's what I've seen in the wild as the most used term from my experience.

[maoueh]

randomization_factor also from https://raw.githubusercontent.com/googleapis/google-http-java-client/da1aa993e90285ec18579f1553339b00e19b3ab5/google-http-client/src/main/java/com/google/api/client/util/ExponentialBackOff.java

[YaroShkvorets]

As for the fuzz naming, I think most library uses instead the term jitter, at least that's what I've seen in the wild as the most used term from my experience.

Cool, I like jitter

[YaroShkvorets]

Sent a PR: #4476

[azf20]

thanks @YaroShkvorets!