agent: allow templating to indefinitely retry if explicitly set to zero

[calvn]

This PR re-introduces the ability for agent's template subsystem to perform indefinite retries that was removed in 1.6.0 through #9670. Indefinite retry can be toggled on by setting vault.retry.num_retries = 0.

Since vault.retry.num_retries is used by both the cache client and the template config, the changeset moves setting these values closer to where the update is called. This reduces the behavioral change to the bare minimum, but introduces the difference on what 0 means for template vs cache.

Templating:

vault.retry.num_retries	Old	New
-1	Disabled	Disabled
0	Default (12x)	∞ retries
N > 0	N retries	N retries
Absent	Default (12x)	Default (12x)

Cache:

vault.retry.num_retries	Old	New
-1	Disabled	Disabled
0	Default (12x)	API Client default (2x)
N > 0	N retries	N retries
Absent	Default (12x)	Default (12x)

Related to #9670
Related to #11113

Backport note:
We need to backport #11696 if/when we backport this PR.

TODO:

• Update/add tests
• Update docs
• Include changelog

[calvn]

One case that may look odd -- if vault.retry.num_retries = 0 and persist is non-nil, it means that the client will try 2x per request. Functionally, this doesn't matter too much since the number of attempts by template is infinite, but does make each template attempt be performed 12x by the cache client.

[calvn]

I'm tempted to change this to be api.Client default of 2 retries here (for a total of 3 calls). Even though 12 is the current value, the client actually times out before making all 12 attempts due to the default request timeout value of 60s, thus the failed to renew: context deadline exceeded errors per attempt.

[WARN] (view) vault.read(database/creds/readonly): vault.read(database/creds/readonly): context deadline exceeded (retry attempt 1 after "250ms")
[TRACE] (view) vault.read(database/creds/readonly) starting fetch
[TRACE] vault.read(database/creds/readonly): starting renewer
2021-05-26T15:54:39.425-0700 [INFO]  cache: received request: method=PUT path=/v1/sys/leases/renew
2021-05-26T15:54:39.425-0700 [DEBUG] cache.leasecache: forwarding request: method=PUT path=/v1/sys/leases/renew
2021-05-26T15:54:39.425-0700 [INFO]  cache.apiproxy: forwarding request: method=PUT path=/v1/sys/leases/renew
[WARN] vault.read(database/creds/readonly): failed to renew: context deadline exceeded
[WARN] vault.read(database/creds/readonly): renewer done (maybe the lease expired)
[TRACE] vault.read(database/creds/readonly): GET /v1/database/creds/readonly
2021-05-26T15:55:39.428-0700 [INFO]  cache: received request: method=GET path=/v1/database/creds/readonly
2021-05-26T15:55:39.428-0700 [DEBUG] cache.leasecache: forwarding request: method=GET path=/v1/database/creds/readonly
2021-05-26T15:55:39.428-0700 [INFO]  cache.apiproxy: forwarding request: method=GET path=/v1/database/creds/readonly
[WARN] (view) vault.read(database/creds/readonly): vault.read(database/creds/readonly): context deadline exceeded (retry attempt 2 after "500ms")
[TRACE] (view) vault.read(database/creds/readonly) starting fetch
[TRACE] vault.read(database/creds/readonly): starting renewer

[jasonodonnell]

I'm OK with this change

[calvn]

I've gone ahead and made this change on 03d248f, and also updated the table on the PR description.

[calvn]

Another suggestion that was made was to break retry values by the cache's client and the template to be different values, e.g. vault.retry.cache_retries and vault.retry.template_retries. We'd still have to support the existing num_retries even if we deprecate it though.

[calvn]

Closing in favor of #11775.