test: wait longer when incrementing error count waiting for pod Ready

[jackfrancis]

Reason for Change:

We implement the following algorithm when assessing for pod readiness in our E2E tests:

• wait for n Ready reponses
• keep track of non-Ready responses as f
• if/when f reaches a count that equals the desired n count before we have tracked the desired Ready responses, return a failure response, as we interpret that to mean that the pod may be thrashing between Ready and non-Ready

(The above algorithm is only implemented if n is greater than 1. If n is 1 then we simply wait for a single Ready response in a given timeout interval.)

In order to eliminate false positive signals due to "normal" operational pod reconciliation, we should wait longer if we get a non-Ready signal, and allow the k8s control plane more time to reconcile the pod into a consistent Ready state.

Issue Fixed:

Requirements:

• uses conventional commit messages
• includes documentation
• adds unit tests
• tested upgrade from previous version

Notes:

[codecov]

Codecov Report

Merging #560 into master will decrease coverage by 1.94%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #560      +/-   ##
==========================================
- Coverage    56.7%   54.75%   -1.95%     
==========================================
  Files          91       97       +6     
  Lines       13859    14708     +849     
==========================================
+ Hits         7859     8054     +195     
- Misses       5334     5980     +646     
- Partials      666      674       +8

[mboersma]

/lgtm

[acs-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jackfrancis, mboersma

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [jackfrancis,mboersma]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment