Investigate Sentry error: ProtocolError "Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED"

[callahad]

Our monitoring for Syngal on matrix.org occasionally sees an error:

KeyError: (<ConnectionState.CLOSED: 3>, <ConnectionInputs.SEND_HEADERS: 0>)
  File "h2/connection.py", line 224, in process_input
    func, target_state = self._transitions[(self.state, input_)]

and

ProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED
  File "sygnal/http.py", line 265, in _handle_dispatch
    result = await pushkin.dispatch_notification(notif, d, context)
  File "sygnal/notifications.py", line 180, in dispatch_notification
    return await self._dispatch_notification_unlimited(n, device, context)
  File "sygnal/apnspushkin.py", line 277, in _dispatch_notification_unlimited
    log, span, device, shaved_payload, prio
  File "sygnal/apnspushkin.py", line 210, in _dispatch_request
    response = await self._send_notification(request)
  File "sygnal/apnspushkin.py", line 476, in _send_notification
    asyncio.ensure_future(self.apns_client.send_notification(request))
  File "twisted/internet/defer.py", line 824, in adapt
    extracted = result.result()
  File "aioapns/client.py", line 53, in send_notification
    response = await self.pool.send_notification(request)
  File "aioapns/connection.py", line 377, in send_notification
    response = await connection.send_notification(request)
  File "aioapns/connection.py", line 176, in send_notification
    headers=headers
  File "h2/connection.py", line 766, in send_headers
    self.state_machine.process_input(ConnectionInputs.SEND_HEADERS)
  File "h2/connection.py", line 229, in process_input
    "Invalid input %s in state %s" % (input_, old_state)

So... what causes this to happen? What calls does Sygnal make that eventually trigger that error? Can we handle the error more gracefully?

(For Element employees, this is https://sentry.matrix.org/sentry/sygnal-matrixorg/issues/119705/)

[zanieb]

Cross-linking for people who end up here on Google. We're seeing this in PrefectHQ/prefect#7442 as well and are investigating the source of the bug (which is certainly somewhere in the http stack).