Tweak dashboard retries functionality for 401/403 errors

[dkwon17]

Is your enhancement related to a problem? Please describe

Dashboard retries feature has been implemented to retry requests that result in 401/403 errors in the dashboard:

• fix: Retry request when token is missed che-dashboard#942
• fix: Retry request when token is missed. Part 2. che-dashboard#946

Here are the results of this fix in the dogfooding cluster. To summarize, out of 118 failing requests, ~94% of those requests have succeeded thanks to retries.

My dashboard refresh script was able to reproduce the error for a few hours on Oct 31:

According to the metrics, out of about 2849 refreshes to the dashboard (maybe this is too much, I reduced the refresh frequency to 60 times every 10 mins) there were about 118 requests that required a retry. Note that the sample size (118) is quite small because the 401/403 issues are rare on the dogfooding cluster.

Out of 118 requests, 111 of them (~94%) succeeded thanks to the retries:

• 92 requests succeeded after 1 retry
• 14 requests succeeded after 2 retries
• 2 requests succeeded after 3 retries
• 3 requests succeeded after 4 retries (**)

Out of 118 requests, 7 of them (~6%) failed despite the retries:

• 2 requests failed after 3 retries
• 5 requests failed after 7 retries (**)

(**) retries greater than 3 is unique to the provision request. The provision request is currently being retried up to 7 times because the provision request is also being retried in testBackends()

Describe the solution you'd like

The 94% success rate for requests that would have otherwise all failed is nice. Some ideas to further tweak the retries to improve the percentage could be:

• increase the number of retries to 4 (code)
• have an exponential delay between requests, rather than a fixed delay (code)

Describe alternatives you've considered

No response

Additional context

No response

[dkwon17]

cc @ibuziuk @tolusha

[tolusha]

@dkwon17
Thank you for detailed investigation.
What are those 2 requests that failed after 3 retries?

[dkwon17]

@tolusha no problem, the two requests are:

• dashboard/api/namespace/dkwon17-che/devworkspaces (one instance of failure after 3 retries)
• dashboard/api/userprofile/dkwon17-che (one instance of failure after 3 retries)

[ibuziuk]

@dkwon17 wdyt, can we include the fix for 7.77.x for 3.10?

[dkwon17]

@ibuziuk that works for me, I'll try to have it for 7.77.x