Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

do family-based association testing/TDT #29

Closed
cseed opened this issue Oct 29, 2015 · 1 comment
Closed

do family-based association testing/TDT #29

cseed opened this issue Oct 29, 2015 · 1 comment

Comments

@cseed
Copy link
Collaborator

cseed commented Oct 29, 2015

From @cseed on August 31, 2015 22:22

Compare plink:

http://pngu.mgh.harvard.edu/~purcell/plink/fanal.shtml#tdt

Copied from original issue: cseed/hail#37
@jbloom22
Copy link
Contributor

TDT is now a Hail command.

cseed added a commit to cseed/hail that referenced this issue Sep 22, 2018
@cseed
don't log the (pod) log (hail-is#29)
2be8185
danking pushed a commit to danking/hail that referenced this issue Sep 24, 2018
@cseed
Merge pull request hail-is#29 from hail-is/fix-port
cae334c 
fix port
danking pushed a commit that referenced this issue Sep 25, 2018
@cseed @danking
Merge batch (#4414)
945d68d 
* initial revision

* wip

* added deployment

updated logging

* added service

* wip

* wip

* cancellation, primitive client library

(unused) bidict
jupyter deployment

* batches, higher level api

* fixed readme

* minor

* added tests, reorg

* tests run in itself

* wip

added attributes
removed job names
create batch objects
added callback (untested)

* added callbacks, callback test

* makefile changes

* wip

try to get callback test working on k8s cluster

* tweaks

* callback test works on k8s

* starting to play with containerized spark

some cleanup

* Create setup.py

* move setup.py to correct location

* ignore compiled pyc files

* describe how to use minikube with local docker images

* add special note about `imagePullPolicy`

* add an environment yaml

* add a getting started section

* add a dockerignore file

* do not fail if callback fails

* expose pod/container volumes

* fix volumeMounts field name

* add resources

* add tolerations

* add jobs listing

* add jobs test

* fix the environment

* update Batch.create_job for all the new parameters

* fix missing id

* avoid crashing on bad event types

* update dockerignore

* retry event loops

* Update server.py

* fix dockerignore

* fix dockerignore

* stash the attributes sent by api

* cache the status

* fixed tests (#28)

* Pr builder image (#27)

* added pr-builder image logic

* added missing files

* address comments

* add hail-ci-build.sh (#30)

* add hail-ci-build.sh wip

* wip

* make hail-ci script work

added shutdown endpoint

* fix

* don't log the (pod) log (#29)

* added bash to pr-builder image (#32)

* Add git (#35)

* test

* added git to pr-builder image

* add python alias in pr-builder image

* added curl to pr-builder image

* Job delete (#36)

* wip: added job/delete

need to test

* added delete, various

* tag executable images (#38)

* update deployment (#39)

* make batch single threaded (#40)

* make batch single threaded

polling and k8s watch thread request into main thread
in addition to k8s notifications, periodically poll k8s state

* address comments

* label batch job pods (#41)

* add hail-ci-deploy.sh (#42)

* added hail-ci-deploy.sh

add docker to pr-builder image

* added missing deployment.yaml.in

removed deployment.yaml

* consistency

* addessed comments

* install kubectl in image (#44)

`gcloud components install kubectl` is failing with a non-obvious error message so I installed it directly

* make batch subproject (#46)

* make batch subproject

fixed more deployment bugs

* updated build image

* fix test race condition

job can complete before it is cancelled

* authenticate docker to push to gcr.io (#48)

* authenticate docker to push to gcr.io

* add gcloud quiet (-q)

* remove errant tab

* logging to match ci

* do callback asynchronously

* do callback asynchronously

* expose pod name as well (#49)

* restart ci on deploy (#50)

* Fix SHA check (#51)

* Update hail-ci-deploy.sh

* Update deployment.yaml.in

* Update hail-ci-deploy.sh

* Update hail-ci-deploy.sh

* add /jobs/<id>/log endpoint (#52)

* add /jobs/<id>/log endpoint

added to api, client
return completed job logs (including deleted ones)
return in-progress logs (if there are any)
return 404 (not empty) if there no logs to be found

status['log'] returns the same log

* fixed bug

* fixed typo

* prep to merge into monorepo

clean up unused files, experiments (jupyter, spark, etc.)

* updated build image

* fixes

hail-ci-build.sh handles all known projects
fix batch to deploy only when changed

* fixed typo

* fixed README.md conflict.

* activate environment

* add cloudtools to list of project-changed.py projects
tpoterba pushed a commit to tpoterba/hail that referenced this issue Feb 12, 2019
Merge pull request hail-is#29 from konradjk/patch-1
f9d776d 
Remove sleep
daniel-goldstein referenced this issue in daniel-goldstein/hail Apr 20, 2021
@vladsavelyev @lgruen @daniel-goldstein
Switch to Artifact Registry (#29)
374e038 
* config.mk: use AR docker image

* build.yaml: use AR for images

* Gitignore artifacts

* fetch-image.sh: gcloud auth for AR

* image-fetcher dockerfile: base on AR

* image fetcher make file: use AR

* build-batch-worker-image-startup.sh: add gcloud auth configure-docker for AR, user AR for images

* batch worker: ue AR for public_gcr_images

* Worker dockerfile: use AR

* ci build.py: add gcloud auth configure-docker for AR, use AR base image

* Makefiles: use AR docker images

* stat-permissions.sh: fix for macOS gnu-grep

* third-party dockers: use AR

* ci dockerFile create_namespace: use AR image

* Docker hail-ubuntu: use AR image

* ci build.py: add region into the jinja render config for Dockerfiles

* CI deploy config: add region

* Python-dill: use AR image

* Auth deployment.yaml: add REGION

* Use hail-dvc default ci image pointer

* Add Artifact Registry in main.tf

* Add AR in dev-docs

* Add AR for ukbb-rg

* Add AR in more makefiles

* Fix AR usage for image-fetcher

* Specify "region" in the "hail-ubuntu-stmp" target.

* Fix zone assignment in startup script.

* Allow ci to read from Artifact Registry.

* Makefiles: parametrise `DOCKER_ROOT_IMAGE` to support both GCR and AR

* Parametrise `DOCKER_PREFIX` to support both GCR and AR

* Un-gitignore "vlad"

* Fix

* batch: pass DOCKER_PREFIX to worker

* DOCKER_PREFIX in build-batch-worker-image-startup.sh

* build.yaml: add ubuntu tags back

* ci/ci/build.py: parametrise docker prefix

* infra README: configure-docker AR

Co-authored-by: Leonhard Gruenschloss <leonhard.gruenschloss@populationgenomics.org.au>
daniel-goldstein referenced this issue in daniel-goldstein/hail Apr 21, 2021
@vladsavelyev @lgruen @daniel-goldstein
Switch to Artifact Registry (#29)
223421e 
* config.mk: use AR docker image

* build.yaml: use AR for images

* Gitignore artifacts

* fetch-image.sh: gcloud auth for AR

* image-fetcher dockerfile: base on AR

* image fetcher make file: use AR

* build-batch-worker-image-startup.sh: add gcloud auth configure-docker for AR, user AR for images

* batch worker: ue AR for public_gcr_images

* Worker dockerfile: use AR

* ci build.py: add gcloud auth configure-docker for AR, use AR base image

* Makefiles: use AR docker images

* stat-permissions.sh: fix for macOS gnu-grep

* third-party dockers: use AR

* ci dockerFile create_namespace: use AR image

* Docker hail-ubuntu: use AR image

* ci build.py: add region into the jinja render config for Dockerfiles

* CI deploy config: add region

* Python-dill: use AR image

* Auth deployment.yaml: add REGION

* Use hail-dvc default ci image pointer

* Add Artifact Registry in main.tf

* Add AR in dev-docs

* Add AR for ukbb-rg

* Add AR in more makefiles

* Fix AR usage for image-fetcher

* Specify "region" in the "hail-ubuntu-stmp" target.

* Fix zone assignment in startup script.

* Allow ci to read from Artifact Registry.

* Makefiles: parametrise `DOCKER_ROOT_IMAGE` to support both GCR and AR

* Parametrise `DOCKER_PREFIX` to support both GCR and AR

* Un-gitignore "vlad"

* Fix

* batch: pass DOCKER_PREFIX to worker

* DOCKER_PREFIX in build-batch-worker-image-startup.sh

* build.yaml: add ubuntu tags back

* ci/ci/build.py: parametrise docker prefix

* infra README: configure-docker AR

Co-authored-by: Leonhard Gruenschloss <leonhard.gruenschloss@populationgenomics.org.au>
daniel-goldstein referenced this issue in daniel-goldstein/hail Apr 22, 2021
@vladsavelyev @lgruen @daniel-goldstein
Switch to Artifact Registry (#29)
9073e29 
* config.mk: use AR docker image

* build.yaml: use AR for images

* Gitignore artifacts

* fetch-image.sh: gcloud auth for AR

* image-fetcher dockerfile: base on AR

* image fetcher make file: use AR

* build-batch-worker-image-startup.sh: add gcloud auth configure-docker for AR, user AR for images

* batch worker: ue AR for public_gcr_images

* Worker dockerfile: use AR

* ci build.py: add gcloud auth configure-docker for AR, use AR base image

* Makefiles: use AR docker images

* stat-permissions.sh: fix for macOS gnu-grep

* third-party dockers: use AR

* ci dockerFile create_namespace: use AR image

* Docker hail-ubuntu: use AR image

* ci build.py: add region into the jinja render config for Dockerfiles

* CI deploy config: add region

* Python-dill: use AR image

* Auth deployment.yaml: add REGION

* Use hail-dvc default ci image pointer

* Add Artifact Registry in main.tf

* Add AR in dev-docs

* Add AR for ukbb-rg

* Add AR in more makefiles

* Fix AR usage for image-fetcher

* Specify "region" in the "hail-ubuntu-stmp" target.

* Fix zone assignment in startup script.

* Allow ci to read from Artifact Registry.

* Makefiles: parametrise `DOCKER_ROOT_IMAGE` to support both GCR and AR

* Parametrise `DOCKER_PREFIX` to support both GCR and AR

* Un-gitignore "vlad"

* Fix

* batch: pass DOCKER_PREFIX to worker

* DOCKER_PREFIX in build-batch-worker-image-startup.sh

* build.yaml: add ubuntu tags back

* ci/ci/build.py: parametrise docker prefix

* infra README: configure-docker AR

Co-authored-by: Leonhard Gruenschloss <leonhard.gruenschloss@populationgenomics.org.au>
daniel-goldstein referenced this issue in daniel-goldstein/hail Apr 22, 2021
@vladsavelyev @lgruen @daniel-goldstein
Switch to Artifact Registry (#29)
1e52cb4 
* config.mk: use AR docker image

* build.yaml: use AR for images

* Gitignore artifacts

* fetch-image.sh: gcloud auth for AR

* image-fetcher dockerfile: base on AR

* image fetcher make file: use AR

* build-batch-worker-image-startup.sh: add gcloud auth configure-docker for AR, user AR for images

* batch worker: ue AR for public_gcr_images

* Worker dockerfile: use AR

* ci build.py: add gcloud auth configure-docker for AR, use AR base image

* Makefiles: use AR docker images

* stat-permissions.sh: fix for macOS gnu-grep

* third-party dockers: use AR

* ci dockerFile create_namespace: use AR image

* Docker hail-ubuntu: use AR image

* ci build.py: add region into the jinja render config for Dockerfiles

* CI deploy config: add region

* Python-dill: use AR image

* Auth deployment.yaml: add REGION

* Use hail-dvc default ci image pointer

* Add Artifact Registry in main.tf

* Add AR in dev-docs

* Add AR for ukbb-rg

* Add AR in more makefiles

* Fix AR usage for image-fetcher

* Specify "region" in the "hail-ubuntu-stmp" target.

* Fix zone assignment in startup script.

* Allow ci to read from Artifact Registry.

* Makefiles: parametrise `DOCKER_ROOT_IMAGE` to support both GCR and AR

* Parametrise `DOCKER_PREFIX` to support both GCR and AR

* Un-gitignore "vlad"

* Fix

* batch: pass DOCKER_PREFIX to worker

* DOCKER_PREFIX in build-batch-worker-image-startup.sh

* build.yaml: add ubuntu tags back

* ci/ci/build.py: parametrise docker prefix

* infra README: configure-docker AR

Co-authored-by: Leonhard Gruenschloss <leonhard.gruenschloss@populationgenomics.org.au>
danking pushed a commit that referenced this issue Apr 26, 2021
@daniel-goldstein @vladsavelyev @lgruen
[docker] Add config support for docker prefix and root image (#10347)
6dfb499 
* Switch to Artifact Registry (#29)

* config.mk: use AR docker image

* build.yaml: use AR for images

* Gitignore artifacts

* fetch-image.sh: gcloud auth for AR

* image-fetcher dockerfile: base on AR

* image fetcher make file: use AR

* build-batch-worker-image-startup.sh: add gcloud auth configure-docker for AR, user AR for images

* batch worker: ue AR for public_gcr_images

* Worker dockerfile: use AR

* ci build.py: add gcloud auth configure-docker for AR, use AR base image

* Makefiles: use AR docker images

* stat-permissions.sh: fix for macOS gnu-grep

* third-party dockers: use AR

* ci dockerFile create_namespace: use AR image

* Docker hail-ubuntu: use AR image

* ci build.py: add region into the jinja render config for Dockerfiles

* CI deploy config: add region

* Python-dill: use AR image

* Auth deployment.yaml: add REGION

* Use hail-dvc default ci image pointer

* Add Artifact Registry in main.tf

* Add AR in dev-docs

* Add AR for ukbb-rg

* Add AR in more makefiles

* Fix AR usage for image-fetcher

* Specify "region" in the "hail-ubuntu-stmp" target.

* Fix zone assignment in startup script.

* Allow ci to read from Artifact Registry.

* Makefiles: parametrise `DOCKER_ROOT_IMAGE` to support both GCR and AR

* Parametrise `DOCKER_PREFIX` to support both GCR and AR

* Un-gitignore "vlad"

* Fix

* batch: pass DOCKER_PREFIX to worker

* DOCKER_PREFIX in build-batch-worker-image-startup.sh

* build.yaml: add ubuntu tags back

* ci/ci/build.py: parametrise docker prefix

* infra README: configure-docker AR

Co-authored-by: Leonhard Gruenschloss <leonhard.gruenschloss@populationgenomics.org.au>

* Fix building worker image (#40)

* Add Artifact Registry permissions for the batch-agent service account.

* Gitignore */hail.jar and *.dylib

* Comment in gitignore on dylib

* Default use_artifact_registry to false

* Use DOCKER_ROOT_IMAGE to pass to batch worker image

* Use DOCKER_ROOT_IMAGE wherever makes sense

* Render build-batch-worker-image-startup.sh with Jinja to set up docker root image

* Fix jinja params

* Add phony rule to configure gcloud and Docker

* Add comments on HAIL_DOCKER_PREFIX

* Make sure the env variable HAIL_DOCKER_PREFIX available to the batch worker instance

* Add missing docker_root_image

* Infra README: correct env var name

* Fix env var case

* remove other instances of gcr.io in favor of DOCKER_PREFIX

* add hail_version to gitignore once and for all

* debug

* add back yaml changes after rebase

* remove gcloud target from hail/Makefile

Co-authored-by: Vlad Savelyev <vladislav.sav@gmail.com>
Co-authored-by: Leonhard Gruenschloss <leonhard.gruenschloss@populationgenomics.org.au>
Co-authored-by: vladsaveliev <vladislav.savelyev@populationgenomics.org.au>
daniel-goldstein referenced this issue in daniel-goldstein/hail Feb 3, 2022
@vladsavelyev @lgruen
Switch to Artifact Registry (#29)
f5d6f86 
* config.mk: use AR docker image

* build.yaml: use AR for images

* Gitignore artifacts

* fetch-image.sh: gcloud auth for AR

* image-fetcher dockerfile: base on AR

* image fetcher make file: use AR

* build-batch-worker-image-startup.sh: add gcloud auth configure-docker for AR, user AR for images

* batch worker: ue AR for public_gcr_images

* Worker dockerfile: use AR

* ci build.py: add gcloud auth configure-docker for AR, use AR base image

* Makefiles: use AR docker images

* stat-permissions.sh: fix for macOS gnu-grep

* third-party dockers: use AR

* ci dockerFile create_namespace: use AR image

* Docker hail-ubuntu: use AR image

* ci build.py: add region into the jinja render config for Dockerfiles

* CI deploy config: add region

* Python-dill: use AR image

* Auth deployment.yaml: add REGION

* Use hail-dvc default ci image pointer

* Add Artifact Registry in main.tf

* Add AR in dev-docs

* Add AR for ukbb-rg

* Add AR in more makefiles

* Fix AR usage for image-fetcher

* Specify "region" in the "hail-ubuntu-stmp" target.

* Fix zone assignment in startup script.

* Allow ci to read from Artifact Registry.

* Makefiles: parametrise `DOCKER_ROOT_IMAGE` to support both GCR and AR

* Parametrise `DOCKER_PREFIX` to support both GCR and AR

* Un-gitignore "vlad"

* Fix

* batch: pass DOCKER_PREFIX to worker

* DOCKER_PREFIX in build-batch-worker-image-startup.sh

* build.yaml: add ubuntu tags back

* ci/ci/build.py: parametrise docker prefix

* infra README: configure-docker AR

Co-authored-by: Leonhard Gruenschloss <leonhard.gruenschloss@populationgenomics.org.au>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@cseed @jbloom22 @alexb-3