V2 Integration isssue on Arm/nix

[dougch]

Security issue notifications

If you discover a potential security issue in s2n we ask that you notify
AWS Security via our vulnerability reporting page. Please do not create a public github issue.

Problem:

As part of #3841, we're chasing down tests that fail under Nix on only Arm instances, needing extra retries. Specifically:

• test_happy_path::test_s2n_client_happy_path - retries bumped from 2 to 5
• test_record_padding::test_s2n_client_handles_padded_records - retries bumped from 2 to 5
• test_renegotiate:: test_s2n_client_renegotiate_with_client_auth_with_openssl - retries bumped from 2 to 3, and the timeout increased from 5 sec to 8 sec
• test_session_resumption::test_s2nd_falls_back_to_full_connection - retries bumped from 2 to 7

These flaky marks only apply to Arm, so no changes to the flaky settings should affect tests running on x86.
These values were arrived at by trial and error, running on CodeBuild (graviton2, 8 cores).

Steps taken to rule these out as an s2n issue

• Repro locally, with no retries (main)
• Add in the top 10 slowest pytest argument --durations=10

============================= slowest 10 durations =============================
5.01s call     test_renegotiate.py::test_s2n_client_renegotiate_with_client_auth_with_openssl[OpenSSL-TLS1.2-RSA_2048_SHA512-P-256-ECDHE-RSA-CHACHA20-POLY1305]
0.97s call     test_renegotiate.py::test_s2n_client_renegotiate_with_openssl[OpenSSL-TLS1.0-RSA_4096_SHA256-P-384-DHE-RSA-AES256-SHA]
...
=========================== short test summary info ============================
FAILED test_renegotiate.py::test_s2n_client_renegotiate_with_client_auth_with_openssl[OpenSSL-TLS1.2-RSA_2048_SHA512-P-256-ECDHE-RSA-CHACHA20-POLY1305]

• Notice that the failure was at the timeout value of 5 seconds
• Bump the timeout to 60 seconds, to rule out "this is just slow on arm"

============================= slowest 10 durations =============================
60.07s call     test_renegotiate.py::test_s2n_client_renegotiate_with_client_auth_with_openssl[OpenSSL-TLS1.2-RSA_2048_SHA512-P-384-ECDHE-RSA-AES128-SHA]
0.97s call     test_renegotiate.py::test_s2n_client_renegotiate_with_openssl[OpenSSL-TLS1.0-RSA_4096_SHA256-P-384-DHE-RSA-AES128-SHA]
...
FAILED test_renegotiate.py::test_s2n_client_renegotiate_with_client_auth_with_openssl[OpenSSL-TLS1.2-RSA_2048_SHA512-P-384-ECDHE-RSA-AES128-SHA]

Same result. Slowest test was the failed test, at the timeout value of 60 seconds.

• Move the test_s2n_client_renegotiate_with_client_auth_with_openssl test to a new test file, all by itself, still timing out on the same parameters.
• Reduce the fixtures into the test: only use TLS1.2, only use RSA_2048_SHA512 Cert, Only use P384 as the Curve, now the timeout has moved to a different set of parameters test_renegotiate_client.py::test_s2n_client_renegotiate_with_client_auth_with_openssl[OpenSSL-TLS1.2-RSA_2048_SHA512-P-384-ECDHE-RSA-AES128-GCM-SHA256]
• further reduce the fixtures, only use the cipher ECDHE_RSA_AES128_SHA,, test passes.

280: PASSED test_renegotiate_client.py::test_s2n_client_renegotiate_with_client_auth_with_openssl[OpenSSL-TLS1.2-RSA_2048_SHA512-P-384-ECDHE-RSA-AES128-SHA]

While this process wasn't done for all of the flaky tests above, I did observe the same timeouts == reruns across all of them.

Solution:

?

[maddeleine]

Can we get the actual errors that occurred when these tests failed?