Attempt to address some of the consistently flaky integration tests

[fselmo]

What was wrong?

Some of our integration tests dealing with the way geth --dev mines within an interval window flaky and still not passing with a flaky decorator in some cases. I'm going to try to run this branch many times and try to address them here.

How was it fixed?

• Add flaky geth dev mining decorator to receipt unmined tests (cadf7f8)
• Mark certain very flaky tests with xfail + strict=False (899a909)
• Some tests were incorrectly checking that the gasPrice on dynamic fee transactions was equal to the maxFeePerGas. Really, it's the minimum between the maxFeePerGas and maxPriorityFeePerGas + baseFeePerGas (block), which is the "effective gas price". Use <= in that comparison.

Feature:

• Introduce a RequestTimedOut exception for when requests to the node time out.

• Use this exception in xfail tests that fail specifically with a client-side timeout to distinguish the fail case. It isn't possible to use xfail and flaky on the same test out-of-the-box and have them work well enough. This PR adds a decorator that:

  • Accepts flaky arguments max_runs and min_passes and retries the test according to flaky rules.
  • Accepts an exception that if by the last flaky retry, if this particular exception is raised, it xfails the test with a given reason string argument, as is also accepted by xfail.

What this means is that we can retry a test up to max_runs times, but if by the end of it we have some flaky exception case that is sometimes expected, we xfail the test and it silently fails for that test run.

Todo:

• Clean up commit history
• Add entry to the release notes

Cute Animal Picture

[fselmo]

Ok @kclowes, @pacrob, @reedsa this should be ready for review. Let me know if you have any questions. This commit fixes a wrong check in some tests that the gasPrice should always equal the maxFeePerGas. In reality, the effective gas price is the minimum between maxFeePerGas and maxPriorityFeePerGas + block's baseFeePerGas.

[pacrob]

🐑

[reedsa]

lgtm