Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sept 23 upstream 4 (dc1f086) #305

Merged
merged 181 commits into from
Sep 19, 2023
Merged
Show file tree
Hide file tree
Changes from 179 commits
Commits
Show all changes
181 commits
Select commit Hold shift + click to select a range
e9dc0db
[batch] Add identifying metadata to Batch requests (#13219)
jigold Jun 28, 2023
71ea012
[query] Fix BAD loop deoptimization with StreamAgg (#12995)
tpoterba Jun 29, 2023
245dfdd
[ci] Add merge candidate to UI page (#13217)
jigold Jun 29, 2023
d53a266
[auth] Only resolve the identity UID for active users (#13221)
daniel-goldstein Jun 29, 2023
b910be3
[ci] Add test identities for services in test namespaces (#13202)
daniel-goldstein Jun 30, 2023
8d6b171
[query/vds] Rework VDS combiner so that it can run via Query on Batch…
chrisvittal Jul 5, 2023
3e0bb6f
[hailctl] Restore pkgs alias for dataproc start packages option (#13220)
daniel-goldstein Jul 5, 2023
b7cc5f3
[batch] Reuse AzureCredentials on the worker (#13225)
daniel-goldstein Jul 6, 2023
07c4930
Fix #13045 - Comparison of Structs and Struct Exprs (#13226)
ehigham Jul 7, 2023
f48a4d1
[batch] Add Azure debugging information when deleting deployments (#1…
jigold Jul 10, 2023
479ef56
[compiler] don't force TableRead to produce a TableValue (#13229)
patrick-schultz Jul 11, 2023
39a9de0
[batch] Add new list_batches query interface to REST calls (#13199)
jigold Jul 11, 2023
de11fba
[batch] Fix list batches query and test (#13237)
jigold Jul 12, 2023
068c0f2
[batch] Fix schedulable cores in UI on pools page (#13236)
jigold Jul 14, 2023
da86326
[ci/ui] filters jobs by status (#13148)
iris-garden Jul 14, 2023
7581167
[query] skip combiner tests until combiner is fast in QoB (#13240)
danking Jul 17, 2023
7db520f
[transient-error] new transient error from GCS (#13170)
danking Jul 17, 2023
4d8f090
[tests] prevents collisions in batch advanced search partial match te…
iris-garden Jul 18, 2023
044d6bb
[hailtop.fs] use parallelism to list directories (#13253)
danking Jul 18, 2023
138d69e
[ci] Add sjparsa as an authorized user (#13257)
jigold Jul 18, 2023
a599c55
[k8s] update to avoid unsupported apiVersions (#13277)
danking Jul 21, 2023
dae3778
[query] reenable long dormant localize_entries tests (#13241)
patrick-schultz Jul 21, 2023
7f718fb
[batch] reads `HAIL_BATCH_REGIONS` environment variable correctly (#1…
iris-garden Jul 21, 2023
9b835f4
[batch] Add feature switches (#13262)
jigold Jul 21, 2023
18c4d53
[query] use nPartitions arg of KeyByAndAgg in lowered impl (#13264)
patrick-schultz Jul 22, 2023
bcc9046
[query] back off spectral moments timeout for local, spark, and batch…
danking Jul 22, 2023
6daed0e
[batch] Fix missing UI checkbox label for feature flags (#13288)
jigold Jul 25, 2023
ae3f81e
[batch] Fix feature-flags UI on driver page (#13290)
jigold Jul 25, 2023
4d583cc
[tests] Increase service backend test timeout to 600s (#13297)
jigold Jul 25, 2023
5326b01
[batch] Add feature flag for oms agent (#13293)
jigold Jul 25, 2023
8db23d7
[auth] Deleting users in test auth should not delete any namespaces (…
daniel-goldstein Jul 27, 2023
ecf4cdc
[batch] Increase max worker idle time to 2 mins in test and dev (#13314)
jigold Jul 27, 2023
ff5d66a
[infra] updates bootstrap dependencies script to use make target (#13…
iris-garden Jul 27, 2023
bf57511
[hailtop.batch] UI Advanced Search Help Documentation (#13233)
jigold Jul 27, 2023
c74530d
[query] removes unnecessary serialization in is.hail.rvd.RVD.localSor…
iris-garden Jul 27, 2023
be37cb8
[batch] Support python 3.11 in default python jobs (#13310)
chrisvittal Jul 27, 2023
f00f916
[release] 0.2.120 (#13265)
danking Jul 27, 2023
d04c9b1
[all] update versions (#13276)
danking Jul 28, 2023
479f134
[gear] Add typing to the transaction decorator (#13308)
daniel-goldstein Jul 28, 2023
8d69b54
update query on batch tmp config flag (#13258)
sjparsa Jul 28, 2023
fbec609
[qob] restart docker after vep initialization (#13327)
patrick-schultz Jul 28, 2023
5f68716
[omnibus] all approved passing PRs including ones critical to get CI …
danking Jul 28, 2023
3b4253e
[qob] use default parallelism for table parallelize (4, not 16) (#13335)
danking Jul 31, 2023
15b30d9
[k8s] Dont require access to default secrets when make deploying (#13…
daniel-goldstein Jul 31, 2023
d6af81a
[security] delete after 5 days, ignore system managed, consider two s…
danking Aug 1, 2023
fa41ed7
[azure] Add connection timeouts for Azure FS in both Python and Scala…
jigold Aug 1, 2023
0e5419f
[qob] disable QoB PR tests (keep deploy and dev) (#13353)
danking Aug 1, 2023
ceb343b
[services] use AWS-inspired retry delays (#13354)
danking Aug 1, 2023
ddcc896
[memory] Remove unused memory service (#13352)
daniel-goldstein Aug 1, 2023
5d29f89
[compiler] incremental hashing based on Threefry (#13317)
patrick-schultz Aug 2, 2023
3fb7e05
[query] Fix copy_log when log path is relative. (#13340)
chrisvittal Aug 2, 2023
d83d9c7
[query/vds] fix transform_gvcf docs (#13341)
chrisvittal Aug 2, 2023
8e30017
[deps] removes dependency on hurry.filesize (#13255)
iris-garden Aug 2, 2023
4446b91
[batch] Fix rare bug in list batches v2 test (#13350)
jigold Aug 2, 2023
b30dd05
[lint] Add aiohttp handler types and remove some pylint disable direc…
daniel-goldstein Aug 2, 2023
5e1df2e
[batch] Compact aggregated_billing_*_resources_v3 tables periodically…
jigold Aug 2, 2023
4da70e2
[fs] do not unlink the directory if there was an exception (#13362)
danking Aug 2, 2023
763f431
[ci] Remove unnecessary worker-deploy-config secret (#13343)
daniel-goldstein Aug 2, 2023
d3ac1c4
[query] Add file info to Tabix VCF read exception (#13342)
chrisvittal Aug 2, 2023
00c5701
[fs] add debugging information to flaky test_hadoop_methods_3 (#13363)
danking Aug 3, 2023
36fef80
[query] ensure functions need by DSP are not moved (#13369)
danking Aug 3, 2023
8421b18
[query/combiner] Fix import_gvcf_interval's default PGT handling (#13…
chrisvittal Aug 3, 2023
6921e6c
Replace invocations of some deprecated Hail functions with their repl…
jmarshall Aug 3, 2023
9f481ce
[query/vds] Teach read_vds to supress the 'no ref block max length wa…
chrisvittal Aug 3, 2023
3388359
[query] allow inteval filters to work with indirect row key (#13333)
patrick-schultz Aug 3, 2023
1575d76
[auth] Dont use the auth client for authz in the auth service (#13330)
daniel-goldstein Aug 3, 2023
6733e04
[batch] Fix transaction decorator to work with additional args (#13373)
daniel-goldstein Aug 3, 2023
4ecc851
[batch] Specify batch-driver as the default container in the batch-dr…
daniel-goldstein Aug 4, 2023
5771a50
[devdocs] Add instructions on giving a dev admin access to their dev …
daniel-goldstein Aug 4, 2023
6608ab9
[auth] Fix verify_dev_or_sa_credentials (#13378)
daniel-goldstein Aug 4, 2023
0a6f128
[qob] cancel the QoB test batch after a test is complete (#13326)
danking Aug 4, 2023
7eb8964
[query] clarify the guarantees on hail.experimental and hail.vds (#13…
danking Aug 4, 2023
973a1f8
[security] bump cryptography again (#13375)
danking Aug 4, 2023
e32182f
[query] Parameterise BlockMatrix tests in test_linalg.py (fixes #132…
ehigham Aug 4, 2023
a31f941
[azure] Optimize file status operations (#13368)
jigold Aug 5, 2023
1c23b08
[k8s] more PDBs to allow scale down of non-preemptibles (#13384)
danking Aug 6, 2023
df17d7f
Upgrade to Python 3.9 (#13275)
danking Aug 7, 2023
1f170c3
[batch] do not rename Session to BaseSession b/c BaseSession already …
danking Aug 7, 2023
97590dc
[ci] Add easier way to add all production developers to a dev namespa…
daniel-goldstein Aug 7, 2023
d5f94c6
[batch] Stream job logs in the batch front-end (#13322)
daniel-goldstein Aug 7, 2023
b7442e8
[vds/combiner] Teach VDS combiner how to handle custom call fields (#…
chrisvittal Aug 7, 2023
a399352
[batch] novel way for GCR to say 403 (#13388)
danking Aug 8, 2023
d5b045c
Revert "[batch] Stream job logs in the batch front-end" (#13392)
daniel-goldstein Aug 8, 2023
14e002f
[batch] billing manager needs to close its billing_client (#13387)
danking Aug 8, 2023
18f09ac
[ci] Remove unused step that adds billing project for benchmark (#13393)
daniel-goldstein Aug 8, 2023
081ad5e
[batch] Mitigate siwei bug by removing cache (#13399)
danking Aug 9, 2023
0e93c74
[ci] Pass developers to dev deploy branches (#13398)
daniel-goldstein Aug 9, 2023
4d9e50c
[hailctl] Add basic tests for hailctl config (#13389)
daniel-goldstein Aug 9, 2023
364e300
[batch] Dont wait on a network namespace forever (#13403)
daniel-goldstein Aug 10, 2023
c46f0ff
[batch] expose Job.spot (#13396)
danking Aug 10, 2023
cd952f1
[ci] Point to batch job page instead of CI job page (#13406)
daniel-goldstein Aug 10, 2023
fef71e1
[batch] Fix bitrot in VM image creation script (#13404)
daniel-goldstein Aug 10, 2023
c1e6bcc
[build.yaml] logging in create_accounts step (#13394)
danking Aug 10, 2023
bd4134f
[build.yaml] scale dev namespaces to zero 9p-9a weekdays and weekends…
danking Aug 10, 2023
35d6a64
[batch] add missing types and deduplicate code (#13401)
danking Aug 10, 2023
6f79d0c
[batch] Use new v3 billing tables on the driver and front end (#13408)
jigold Aug 10, 2023
e922d6f
[query] changes in charr docstring (#13411)
wlu04 Aug 10, 2023
5b6bc96
[build.yaml] use an acocunt that actually has admin privs (#13415)
danking Aug 11, 2023
0f00dcd
[batch] Up network allocation timeout to a minute (#13412)
daniel-goldstein Aug 11, 2023
69168e5
[gear] some trivial type annotations (#13419)
danking Aug 11, 2023
d4a2f1d
[ci] Use ci-agent not ci-admin for cron (#13420)
daniel-goldstein Aug 11, 2023
959a085
[batch] monitor in prometheus/grafana the *schedulable* free cores (#…
danking Aug 11, 2023
64cff71
[batch] ignore aiomysql warnings about batch.billing_projects.name (#…
danking Aug 11, 2023
4623e55
[batch] show schedulable cores instead of schedulable + pending in po…
danking Aug 11, 2023
32cb129
[build.yaml] finally fix up the namespace autoscaler (#13426)
danking Aug 11, 2023
ca4ab15
[hailctl] Autocomplete for hailctl config {get,set,unset} (#13224)
jigold Aug 14, 2023
f889f76
[query] Reset custom references after a test finishes (#13423)
daniel-goldstein Aug 15, 2023
a238167
[aiogoogle] Fix bucket name parsing (#13435)
jigold Aug 15, 2023
59a9d0d
[batch] Raise instead of returning aiohttp objects that subclass Exce…
daniel-goldstein Aug 15, 2023
8fb7d90
[compiler] fix TableUnion lowering (#13414)
patrick-schultz Aug 16, 2023
7b65bf9
[prometheus] update to 2.46.0 to attempt to avoid WAL issues (#13443)
danking Aug 16, 2023
9237758
[ci] Make dev namespace scaling cron in UTC (#13429)
daniel-goldstein Aug 16, 2023
b62647a
[dev-docs] Update dev docs for using dev namespaces (#13449)
daniel-goldstein Aug 16, 2023
bfdd53e
[query] force `aggregate_cols` to be local (#13405)
patrick-schultz Aug 16, 2023
14d6f9a
[aioclient] Refactor BatchBuilder submit for ease of job groups chang…
jigold Aug 17, 2023
7530ad0
[hailctl] Dont change the current python environment in test fixture …
daniel-goldstein Aug 18, 2023
7b7c9d6
[dev-docs] Instructions for resetting a dev database (#13456)
daniel-goldstein Aug 18, 2023
c032637
[compiler] restore support for dropRows… (#13460)
patrick-schultz Aug 18, 2023
83a6c05
[hailctl] clarify suggested use of -- and allow it in hailctl batch s…
danking Aug 18, 2023
541e820
[query] fix ./gradlew test (#13459)
danking Aug 18, 2023
8f09683
[k8s] Scale down dev namespaces every night (#13462)
daniel-goldstein Aug 19, 2023
86563a7
[query] add more examples of using hl.import_vcf (#13464)
danking Aug 21, 2023
eeb3168
[batch] maybe simplify jobs_after_update (#13372)
danking Aug 21, 2023
e49c6a3
[batch_client.aioclient] Get rid of BatchBuilder (#13458)
jigold Aug 21, 2023
3de55e0
[devdocs] Batch System Developer Docs (#13427)
jigold Aug 21, 2023
4120b6e
[ci] fix borked CI (#13473)
danking Aug 22, 2023
900a5b5
[batch] Move config.json into the container bundle (#13438)
daniel-goldstein Aug 22, 2023
d7f5d4c
[batch] Add aiomonitor to worker process (#13439)
daniel-goldstein Aug 22, 2023
70359ac
[query] fix services-jvm-test and fs-jvm-test Make targets (#13471)
danking Aug 22, 2023
86f9773
[batch] Turn on compaction in test and dev (#13476)
jigold Aug 22, 2023
3a8be2b
[hailtop.batch_client] Merge Unsubmitted/Submitted Job into Job (#13467)
jigold Aug 22, 2023
1d80c26
[query] eliminate use of type in ldscsim (#13477)
danking Aug 23, 2023
57cf14d
[ci/ui] fixes 500 when there are no jobs to display (#13254)
iris-garden Aug 23, 2023
de7510f
[query] frozenlist.__repr__ should be eval-able (#13422)
danking Aug 23, 2023
1391db0
[hailctl] Initialize Hail Environment Interactively (#13279)
jigold Aug 23, 2023
8be7118
[aiotools] Log exception is task manager task errors (#13453)
daniel-goldstein Aug 23, 2023
90886da
[query] Add bounds checks to local_to_global (#13485)
chrisvittal Aug 23, 2023
2ca85a6
[ci] Choose order for testing PRs by merge priority order (#13478)
jigold Aug 24, 2023
7ff6832
[batch/ui] adds advanced search box and help link (#13463)
iris-garden Aug 24, 2023
6c15ea3
[batch] report unclosed ClientSession alloc location in batch-driver …
danking Aug 24, 2023
ae999da
[batch] Fix default query to match v2 (#13493)
daniel-goldstein Aug 24, 2023
ae5c960
[dev-docs] Add document detailing git practices (#13492)
daniel-goldstein Aug 24, 2023
1b46201
[github] apply needs-triage label to each bug report (#13417)
danking Aug 25, 2023
9cb49a0
[fs] remove Azure _debug parameters (#13472)
danking Aug 25, 2023
ab316ce
[query] clarify file format version vs library version (#13494)
danking Aug 25, 2023
b6e749e
[batch] remove use of smart quotes in docs (#13496)
danking Aug 25, 2023
98bbea0
[batch] use headings instead of captions (#13499)
danking Aug 25, 2023
9499633
[batch] Search area fixes (#13501)
jigold Aug 25, 2023
b63041d
[query] improve type hinting for tables (#13468)
danking Aug 25, 2023
e702d25
[batch] Favor configuration_of and get_remote_tmpdir over get_user_co…
daniel-goldstein Aug 25, 2023
06e20ae
[batch] Drop unused SQL functions (#13510)
jigold Aug 28, 2023
a4490b7
[batch] Actually do the cleanup unused functions migration (#13514)
jigold Aug 28, 2023
fa5e933
[auth] Fix the auth rest API for developers (#13515)
daniel-goldstein Aug 28, 2023
a3b7f0b
[lint] Swap mypy for pyright (#13437)
daniel-goldstein Aug 29, 2023
0132c19
[devbin] Generate cleanup policies for GCP artifact registry (#13489)
jigold Aug 29, 2023
0a2816f
[ci] Add instruction for regenerating AR policy file (#13522)
daniel-goldstein Aug 30, 2023
f658a92
[batch] Dont treat unknown in DockerError 500s as transient (#13521)
daniel-goldstein Aug 30, 2023
b5e0c3c
[ci] Use skopeo 1.13.2 (#13527)
daniel-goldstein Aug 30, 2023
09c092f
[batch] More debugging information for network namespace errors (#13509)
daniel-goldstein Aug 30, 2023
228e98e
[batch] Name the batch worker container (#13519)
daniel-goldstein Aug 31, 2023
91f5a0b
[lint] Add pre-commit to dev requirements (#13518)
daniel-goldstein Aug 31, 2023
b1a1f81
[batch] treat search field as plaintext: no spellcheck, no autocorrec…
danking Aug 31, 2023
5d57d09
[query] All non-spark task context use in using blocks (#13534)
chrisvittal Sep 1, 2023
a2f29c3
[query] fix pickling of hl.Struct (#13523)
danking Sep 1, 2023
07a483a
[batch] Show cost breakdown in the UI (#13491)
jigold Sep 1, 2023
a22168b
[batch] re-allow sending kwargs to python jobs (#13505)
danking Sep 1, 2023
344b9e3
[python-dill][hailtop] update past the bad version of dill (#13536)
danking Sep 1, 2023
5205d61
[pre-commit] add ruff, check-sql checks (#13474)
danking Sep 1, 2023
2342d1b
[batch] update ubuntu image version to latest (#13440)
sjparsa Sep 1, 2023
adc34d3
[docker] copy even if the tag already exists (#13538)
danking Sep 1, 2023
f063e96
[batch] fix misnamed resources (#13539)
danking Sep 1, 2023
799ae7d
[fs] fix bugs in glob and list in our GSFS (#13434)
danking Sep 2, 2023
5d545c8
[ci] Don't test PRs with do-not-test label (#13540)
jigold Sep 2, 2023
076a587
[query] avoid signed integer overlow in partition function (#13550)
danking Sep 3, 2023
5f8fcf5
[batch] Do not mount the xfs_quota from the worker host (#13544)
daniel-goldstein Sep 5, 2023
dbb28ae
[batch] remove redundant super class (#13541)
danking Sep 5, 2023
dc1f086
[auth] Allow use of cloud access tokens for hail batch authorization …
daniel-goldstein Sep 5, 2023
b015fef
Merge commit 'dc1f086' into sept-23-upstream-4-dc1f086
illusional Sep 8, 2023
f76da5a
Fix dropped change
illusional Sep 12, 2023
ba7792d
Address review feedback for bad merge
illusional Sep 18, 2023
2b74e60
Apply review suggestions
illusional Sep 19, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
3 changes: 2 additions & 1 deletion .github/ISSUE_TEMPLATE/bug-report.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
name: Bug Report
description: Report a bug
labels: needs-triage
body:
- type: textarea
id: what-happened
Expand All @@ -19,5 +20,5 @@ body:
id: log
attributes:
label: Relevant log output
description: Please copy and paste any relevant log output.
description: Please copy and paste any relevant log output. If too large, please put in https://gist.github.com and provide the link.
render: shell
17 changes: 17 additions & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -1,5 +1,22 @@
---
default_install_hook_types: [pre-commit, pre-push]
repos:
- repo: local
hooks:
- id: ruff
name: ruff
entry: ruff --fix --show-fixes
language: system
types: [python]
require_serial: true
- id: pyright
name: pyright
entry: pyright
language: system
types: [python]
require_serial: true
stages:
- pre-push
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v2.3.0
hooks:
Expand Down
19 changes: 6 additions & 13 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

include config.mk

SERVICES := auth batch ci memory notebook monitoring website
SERVICES := auth batch ci notebook monitoring website
SERVICES_PLUS_ADMIN_POD := $(SERVICES) admin-pod
SERVICES_IMAGES := $(patsubst %, %-image, $(SERVICES_PLUS_ADMIN_POD))
SERVICES_MODULES := $(SERVICES) gear web_common
Expand Down Expand Up @@ -35,7 +35,7 @@ check-all: check-hail check-services
check-hail-fast:
ruff check hail/python/hail
ruff check hail/python/hailtop
$(PYTHON) -m mypy --config-file setup.cfg hail/python/hailtop
$(PYTHON) -m pyright hail/python/hailtop

.PHONY: pylint-hailtop
pylint-hailtop:
Expand All @@ -55,7 +55,7 @@ pylint-%:
.PHONY: check-%-fast
check-%-fast:
ruff check $*
$(PYTHON) -m mypy --config-file setup.cfg $*
$(PYTHON) -m pyright $*
$(PYTHON) -m black $* --check --diff
curlylint $*
cd $* && bash ../check-sql.sh
Expand All @@ -78,8 +78,7 @@ check-pip-requirements:
web_common \
auth \
batch \
ci \
memory
ci

.PHONY: check-linux-pip-requirements
check-linux-pip-requirements:
Expand All @@ -91,8 +90,7 @@ check-linux-pip-requirements:
web_common \
auth \
batch \
ci \
memory
ci

.PHONY: install-dev-requirements
install-dev-requirements:
Expand All @@ -103,8 +101,7 @@ install-dev-requirements:
-r web_common/pinned-requirements.txt \
-r auth/pinned-requirements.txt \
-r batch/pinned-requirements.txt \
-r ci/pinned-requirements.txt \
-r memory/pinned-requirements.txt \
-r ci/pinned-requirements.txt

hail/python/hailtop/pinned-requirements.txt: hail/python/hailtop/requirements.txt
./generate-linux-pip-lockfile.sh hail/python/hailtop
Expand All @@ -130,9 +127,6 @@ batch/pinned-requirements.txt: web_common/pinned-requirements.txt batch/requirem
ci/pinned-requirements.txt: web_common/pinned-requirements.txt ci/requirements.txt
./generate-linux-pip-lockfile.sh ci

memory/pinned-requirements.txt: gear/pinned-requirements.txt memory/requirements.txt
./generate-linux-pip-lockfile.sh memory

.PHONY: generate-pip-lockfiles
generate-pip-lockfiles: hail/python/hailtop/pinned-requirements.txt
generate-pip-lockfiles: hail/python/pinned-requirements.txt
Expand All @@ -142,7 +136,6 @@ generate-pip-lockfiles: web_common/pinned-requirements.txt
generate-pip-lockfiles: auth/pinned-requirements.txt
generate-pip-lockfiles: batch/pinned-requirements.txt
generate-pip-lockfiles: ci/pinned-requirements.txt
generate-pip-lockfiles: memory/pinned-requirements.txt

$(HAILTOP_VERSION):
$(MAKE) -C hail python/hailtop/hail_version
Expand Down
Loading