[hailtop] Dont assume exact error message match for ClientPayloadErro…

…r retrying (#14545)

The treatment of `ClientPayloadError` as a sometimes transient error was
originally made in response to [an existing
issue](aio-libs/aiohttp#4581) in aiohttp that
can cause transient errors on the client that are difficult to
distinguish from a real broken server. What's in `main` matched exactly
on the error message, but that error message has [since
changed](aio-libs/aiohttp@dc38630)
to include more information, breaking our transient error handling. This
change relaxes the requirement of the error response string to fix
transient error handling for our current version of `aiohttp`.

I wish I had a better approach. `ClientPayloadError` can also be thrown
in the case of malformed data, so I am reticent to treat it as always
transient, but we could perhaps make it a `limited_retries_error` and
avoid inspecting the error message.

hail/python/hailtop/utils/utils.py

@@ -654,7 +654,7 @@ def is_transient_error(e: BaseException) -> bool:
     # appears to happen when the connection is lost prematurely, see:
     # https://github.com/aio-libs/aiohttp/issues/4581
     # https://github.com/aio-libs/aiohttp/blob/v3.7.4/aiohttp/client_proto.py#L85
-    if isinstance(e, aiohttp.ClientPayloadError) and e.args[0] == "Response payload is not completed":
+    if isinstance(e, aiohttp.ClientPayloadError) and "Response payload is not completed" in e.args[0]:
         return True
     if isinstance(e, aiohttp.ClientOSError) and 'sslv3 alert bad record mac' in e.strerror:
         # aiohttp.client_exceptions.ClientOSError: [Errno 1] [SSL: SSLV3_ALERT_BAD_RECORD_MAC] sslv3 alert bad record mac (_ssl.c:2548)