OK, there were two problems:

1. A timeout of 5s appears to be now too short for Google Cloud Storage. I am not sure why but we
   timeout substantially more frequently. I have observed this myself on my laptop. Just this
   morning I saw it happen to Daniel.

2. When using an `aiohttp.AsyncIterablePayload`, it is *critical* to always check if the coroutine
   which actually writes to GCS (which is stashed in the variable `request_task`) is still
   alive. In the current `main`, we do not do this which causes hangs (in particular the timeout
   exceptions are never thrown ergo we never retry).

To understand the second problem, you must first recall how writing works in aiogoogle. There are
two Tasks and an `asyncio.Queue`. The terms "writer" and "reader" are somewhat confusing, so let's
use left and right. The left Task has the owning reference to both the source "file" and the
destination "file". In particular, it is the *left* Task which closes both "files". Moreover, the
left Task reads chunks from the source file and places those chunks on the `asyncio.Queue`. The
right Task takes chunks off the queue and writes those chunks to the destination file.

This situation can go awry in two ways.

First, if the right Task encounters any kind of failure, it will stop taking chunks off of the
queue. When the queue (which has a size limit of one) is full, the left Task will hang. The system
is stuck. The left Task will wait forever for the right Task to empty the queue.

The second scenario is exactly the same except that the left Task is trying to add the "stop"
message to the queue rather than a chunk.

In either case, it is critical that the left Task waits simultaneously on the queue operation *and*
on the right Task completing. If the right Task has died, no further writes can occur and the left
Task must raise an exception. In the first scenario, we do not observe the right Task's exception
because that will be done when we close the `InsertObjectStream` (which represents the destination
"file").

---

I also added several types, assertions, and a few missing `async with ... as resp:` blocks.

[copy] fix the TimeoutError and ServerDisconnected issues in copy

…-main-0.2.95

Merge upstream 0.2.95

Allow to select a pool for a job through a label

Use Spot VMs on GCP

* Revert "Sort"

This reverts commit c08b295.

* Revert "Use spot machines on GCP"

This reverts commit 13c377b.

* Revert "Sort"

This reverts commit c08b295.

* Revert "Use spot machines on GCP"

This reverts commit 13c377b.

* Revert "Revert "Use spot machines on GCP""

This reverts commit b4f4371.

* Only set instanceTerminationAction for Spot VMs

* Use dict.update

Release 0.2.96

…202)

Co-authored-by: Patrick Schultz <pschultz@broadinstitute.org>

0.2.97 merge from upstream

…s file (#203)

* ORGANIZATION_DOMAIN --> GITHUB_ORGANIZATION

* Add github_organization to global.tfvars template

* Use github_organization in main.tf for credentials paths

* Update credentials paths

…tions (#211)

# Conflicts:
#	build.yaml
#	infra/gcp/main.tf

Merge from upstream to DB migrations

Merge up until migrating pool IDs

…billing-500_error

Query_batch_jobs_for_billing-500_error

See hail-is#12128 and
https://hail.zulipchat.com/#narrow/stream/300487-Hail-Batch-Dev/topic/GKE.20Autoscaling.20bug/near/294969288

V0.2.98 upstream1

V0.2.98 upstream2

V0.2.98 upstream3

V0.2.99 upstream

Upstream 1

Upstream 2

Upstream 0.2.100

…1-0.2.101-release

Release 0.2.101: Upstream 1

…ease

Release 0.2.101: Upstream 2

Upstream 0.2.102

Fix Terraform

Upstream merge to pick up hail-is#12305

* Revert "More debugging instrumentation for Tim Poterba (#239)"

This reverts commit eb2cbba.

* Take 2 for instrumentation patch for Tim Poterba

Co-authored-by: Leonhard Gruenschloss <leonhard.gruenschloss@populationgenomics.org.au>

Co-authored-by: Leonhard Gruenschloss <leonhard.gruenschloss@populationgenomics.org.au>

Co-authored-by: Leonhard Gruenschloss <leonhard.gruenschloss@populationgenomics.org.au>

See hail-is#12487

…" (#245)

This reverts commit 5a2758a.

This reverts commit 3c66763.

This reverts commit 0c057d6.

This reverts commit eb2cbba.

This reverts commit 7921901.

Revert instrumentation

# Conflicts:
#	batch/batch/front_end/front_end.py
#	ci/ci/github.py
#	docker/Dockerfile.service-java-run-base
#	hail/src/main/scala/is/hail/backend/service/ServiceBackend.scala
#	hail/src/main/scala/is/hail/backend/service/Worker.scala
#	hail/src/main/scala/is/hail/io/fs/AzureStorageFS.scala
#	web_common/web_common/templates/header.html