Merge upstream changes (#82)

* [batch] Worker cleanup (hail-is#10155)

* [batch] Worker cleanup

* more changes

* wip

* delint

* additions?

* fix

* [query] Add `source_file_field` to `import_table` (hail-is#10164)

* [query] Add `source_file_field` to `import_table`

CHANGELOG: Add `source_file_field` parameter to `hl.import_table` to allow lines to be associated with their original source file.

* ugh

* [ci] add authorize sha and action items table to user page (hail-is#10142)

* [ci] add authorize sha and action items table to user page

* [ci] track review requested in addition to assigned for PR reviews

* [ci] add CI dropdown with link to user page (hail-is#10163)

* [batch] add more logs and do not wait for asyncgens (hail-is#10136)

* [batch] add more logs and do not wait for asyncgens

I think there is some unresolved issue with asyncgen shutdown that is keeping
workers alive. This is not an issue in worker because worker calls sys.exit
which forcibly stops execution. cc: @daniel-goldstein @jigold.

* fix lint

* [query-service] maybe fix event loop not initialized (hail-is#10153)

* [query-service] maybe fix event loop not initialized

The event loop is supposed to be initialized in the main thread. Sometimes
our tests get placed in the non-main thread (always a thread named Dummy-1).
Hopefully the session-scoped fixture is run in the main thread.

* fix

* [prometheus] add prometheus to track SLIs (hail-is#10165)

* [prometheus] add prometheus to track SLIs

* add wraps

* [query] apply nest-asyncio as early as possible (hail-is#10158)

* [query] apply nest-asyncio as early as possible

* fix

* [grafana] set pod fsGroup to grafana user (hail-is#10162)

* fix linting errors (hail-is#10171)

* [query] Remove verbose print (hail-is#10167)

Looks like this got added in some dndarray work

* [ci] update assignees and reviewers on PR github update (hail-is#10168)

* [query-service] fix receive logic (hail-is#10159)

* [query-service] fix receive logic

Only one coro waits on receive now. We still error if a message is sent before
we make our first response.

* fix

* fix

* CHANGELOG: Fixed incorrect error message when incorrect type specified with hl.loop (hail-is#10174)

* [linting] add curlylint check for any service that renders jinja2 (hail-is#10172)

* [linting] add curlylint check for any service that renders jinja2 templates

* [linting] spaces not tabs

* [website] fix website (hail-is#10173)

* [website] fix website

I build old versions of the docs and use them in new websites. This does not
work for versions of the docs before I introduced the new system. In particular
versions 0.2.63 and before generate old-style docs.

* tutorials are templated

* [ci] change mention for deploy failure (hail-is#10178)

* [gateway] move ukbb routing into gateway (hail-is#10179)

* [query] Fix filter intervals (keep=False) memory leak (hail-is#10182)

* [query-service] remove service backend tests (hail-is#10180)

They are too flaky currently due to the version issue.

* [website] pass response body as kwarg (hail-is#10176)

* Release 0.2.64 (hail-is#10183)

* Bump version number

* Updated changelog

* [nginx] ensure nginx configs dont overwrite each other in build.yaml (hail-is#10181)

* [query-service] teach query service to read MTs and Ts created by Spark (hail-is#10184)

* [query-service] teach query service to read MTs and Ts created by Spark

Hail-on-Spark uses HadoopFS which emulates directories by creating size-zero files with
the name `gs://bucket/dirname/`. Note: the object name literally ends in a slash. Such files
should not be included in `listStatus` (they should always be empty anyway). Unfortunately,
my fix in hail-is#9914 was wrong because `GoogleStorageFileStatus` removes
the trailing slash. This prevented the path from matching `path`, which always ends in a `/`.

* fix

Co-authored-by: jigold <jigold@users.noreply.github.com>
Co-authored-by: Tim Poterba <tpoterba@broadinstitute.org>
Co-authored-by: Daniel Goldstein <danielgold95@gmail.com>
Co-authored-by: Dan King <daniel.zidan.king@gmail.com>
Co-authored-by: John Compitello <johnc@broadinstitute.org>

Makefile

@@ -15,7 +15,7 @@ check-hail:
 .PHONY: check-services
 check-services: check-auth check-batch check-ci check-gear check-memory \
   check-notebook check-query check-router-resolver check-scorecard check-web-common \
-  check-atgu
+  check-atgu check-website
 
 .PHONY: check-auth
 check-auth:
@@ -64,3 +64,7 @@ check-web-common:
 .PHONY: check-atgu
 check-atgu:
 	$(MAKE) -C atgu check
+
+.PHONY: check-website
+check-website:
+	$(MAKE) -C website check

address/Makefile

@@ -10,6 +10,7 @@ PYTHON := PYTHONPATH=$(PYTHONPATH) python3
 check:
 	$(PYTHON) -m flake8  --config ../setup.cfg address
 	$(PYTHON) -m pylint --rcfile ../pylintrc address --score=n
+	curlylint .
 
 .PHONY: build
 build:

address/address/templates/index.html

@@ -22,14 +22,13 @@ <h1>Cache</h1>
   <tbody>
     {% for row in rows %}
     <tr>
-        <td>{{ row["name"] }}</td>
-        <td>{{ row["addresses"] }}</td>
-        <td>{{ row["ports"] }}</td>
-        <td>{{ row["lifetime"] }}</td>
-        <td>{{ row["lock"] }}</td>
-      </tr>
+      <td>{{ row["name"] }}</td>
+      <td>{{ row["addresses"] }}</td>
+      <td>{{ row["ports"] }}</td>
+      <td>{{ row["lifetime"] }}</td>
+      <td>{{ row["lock"] }}</td>
+    </tr>
     {% endfor %}
   </tbody>
 </table>
-</div>
 {% endblock %}

atgu/Makefile

@@ -5,4 +5,5 @@ PYTHON := PYTHONPATH=$${PYTHONPATH:+$${PYTHONPATH}:}$(EXTRA_PYTHONPATH) python3
 check:
 	$(PYTHON) -m flake8  --config ../setup.cfg atgu
 	$(PYTHON) -m pylint --rcfile ../pylintrc atgu --score=n
+	curlylint .
 	../check-sql.sh

auth/Makefile

@@ -10,6 +10,7 @@ PYTHON := PYTHONPATH=$${PYTHONPATH:+$${PYTHONPATH}:}$(EXTRA_PYTHONPATH) python3
 check:
 	$(PYTHON) -m flake8 auth
 	$(PYTHON) -m pylint --rcfile ../pylintrc auth --score=n
+	curlylint .
 	../check-sql.sh
 
 .PHONY: build

auth/auth/templates/account-error.html

@@ -4,10 +4,12 @@
 <div class="vcentered">
  <div><b>Account is in a bad state!</b></div>
   <br>
-  <div>Please contact <a href="mailto:hail-team@broadinstitute.org">hail-team@broadinstitute.org</a><div>
+  <div>
+    Please contact <a href="mailto:hail-team@broadinstitute.org">hail-team@broadinstitute.org</a>
     <br>
     <div class="small">Username: <b>{{ username }}</b></div>
     <div class="small">Email: <b>{{ email }}</b></div>
     <div class="small">State: <b>{{ state }}</b></div>
   </div>
+</div>
 {% endblock %}

batch/Makefile

@@ -13,6 +13,7 @@ PYTHON := PYTHONPATH=$${PYTHONPATH:+$${PYTHONPATH}:}$(EXTRA_PYTHONPATH) python3
 check:
 	$(PYTHON) -m flake8  --config ../setup.cfg batch
 	$(PYTHON) -m pylint --rcfile ../pylintrc batch --score=n
+	curlylint .
 	../check-sql.sh
 
 .PHONY: build-prereqs

batch/batch/driver/main.py

@@ -79,11 +79,13 @@ async def wrapped(request):
     return wrapped
 
 
-def instance_from_request(request):
+def instance_name_from_request(request):
     instance_name = request.headers.get('X-Hail-Instance-Name')
-    if not instance_name:
-        return None
+    return instance_name
 
+
+def instance_from_request(request):
+    instance_name = instance_name_from_request(request)
     inst_coll_manager = request.app['inst_coll_manager']
     return inst_coll_manager.get_instance(instance_name)
 
@@ -93,24 +95,25 @@ def activating_instances_only(fun):
     async def wrapped(request):
         instance = instance_from_request(request)
         if not instance:
-            log.info('instance not found')
+            instance_name = instance_name_from_request(request)
+            log.info(f'instance {instance_name} not found')
             raise web.HTTPUnauthorized()
 
         if instance.state != 'pending':
-            log.info('instance not pending')
+            log.info(f'instance {instance.name} not pending')
             raise web.HTTPUnauthorized()
 
         activation_token = authorization_token(request)
         if not activation_token:
-            log.info('activation token not found')
+            log.info(f'activation token not found for instance {instance.name}')
             raise web.HTTPUnauthorized()
 
         db = request.app['db']
         record = await db.select_and_fetchone(
             'SELECT state FROM instances WHERE name = %s AND activation_token = %s;',
             (instance.name, activation_token))
         if not record:
-            log.info('instance, activation token not found in database')
+            log.info(f'instance {instance.name}, activation token not found in database')
             raise web.HTTPUnauthorized()
 
         resp = await fun(request, instance)
@@ -124,24 +127,25 @@ def active_instances_only(fun):
     async def wrapped(request):
         instance = instance_from_request(request)
         if not instance:
-            log.info('instance not found')
+            instance_name = instance_name_from_request(request)
+            log.info(f'instance not found {instance_name}')
             raise web.HTTPUnauthorized()
 
         if instance.state != 'active':
-            log.info('instance not active')
+            log.info(f'instance not active {instance.name}')
             raise web.HTTPUnauthorized()
 
         token = authorization_token(request)
         if not token:
-            log.info('token not found')
+            log.info(f'token not found for instance {instance.name}')
             raise web.HTTPUnauthorized()
 
         db = request.app['db']
         record = await db.select_and_fetchone(
             'SELECT state FROM instances WHERE name = %s AND token = %s;',
             (instance.name, token))
         if not record:
-            log.info('instance, token not found in database')
+            log.info(f'instance {instance.name}, token not found in database')
             raise web.HTTPUnauthorized()
 
         await instance.mark_healthy()