Add Retryable HTTP Client

[akashsinghal]

Describe the solution you'd like
Currently Ratify’s built in ORAS store, does not utilize request back off retries when sending requests to registry. The server can receive multiple concurrent requests to validate images from the same registry or multiple images from the same registry in the same request. This can cause the # of simultaneous requests to the registry to the exceed the registry threshold leading to 429s returned. Ratify should use a retry-able http client with ORAS-go to help alleviate the issue.

Environment:

• Ratify version: (use the image tag): Ratify v1.0.0-beta.2
• Kubernetes version: (use kubectl version): 1.25.4

[sajayantony]

/cc @shizhMSFT

[shizhMSFT]

oras-go will implement a default retry mechanism in its version 2.0.0 release as being discussed in oras-project/oras-go#232.

Besides, oras-go also supports 3rd party retry-able HTTP client such as github.com/hashicorp/go-retryablehttp. Here's the example code of how to integrate go-retryablehttp with oras-go/v2/registry/remote/auth.Client:

retryClient := retryablehttp.NewClient()
retryClient.RetryMax = 10

authClient := &auth.Client{
    Client: retryClient.StandardClient()
}

// use `authClient` for registry access

Since ratify is dealing with lots of 429s, it is suggested to configure retry with jitters.

To be more scalable, ratify may consider caching or other mechanisms to reduce the request rate to a single registry.