Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

docs/naming edits #2

Merged
merged 2 commits into from
Oct 18, 2016
Merged

docs/naming edits #2

merged 2 commits into from
Oct 18, 2016

Conversation

tpoterba
Copy link

No description provided.

jigold
jigold reviewed Oct 18, 2016
View reviewed changes
docs/reference/HailExpressionLanguage.md
If the genotype data was not imported using the [`importbgen`](commands.html#importbgen) or [`importgen`](commands.html#importgen) commands, then the results for all variants will be `score = NA` and `nIncluded = 0`.

**Note:**
It only makes sense to compute info score for an `Aggregable[Genotype]` per variant. While performing a per-sample info score will run
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is incomplete... I assume you mean While performing a per-sample info score will run, the result is not interpretable

docs/reference/HailExpressionLanguage.md
exportsamples -c 'Sample = s, sa.inbreeding.*' -o ib_stats.tsv
```

Calculate the inbreeding metrics per variant and export these mtrics to a TSV file:
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

metrics

@jigold jigold merged commit 94b8a03 into jigold:jg_ibc2 Oct 18, 2016
jigold added a commit that referenced this pull request Aug 8, 2017
@jigold
Reference Genome #2
88bfbd3 
- Added methods to `Variant` that take a `GenomeReference` object
- Reconfigured VariantSubgen to take a Gen[Contig]
- Added a Gen[Contig] from a GenomeReference object
jigold added a commit that referenced this pull request Aug 9, 2017
@jigold
Reference Genome #2 (hail-is#2080)
6d6b069 
* Reference Genome #2

- Added methods to `Variant` that take a `GenomeReference` object
- Reconfigured VariantSubgen to take a Gen[Contig]
- Added a Gen[Contig] from a GenomeReference object

* fixed export bug and changed contig gen slightly
jigold pushed a commit that referenced this pull request May 4, 2018
@danking
RVD spicy meatball (now with only +876/-587) (hail-is#3414)
824968e 
* # This is a combination of 22 commits.
# This is the 1st commit message:

apply resettable context

forgot to fix one use of AutoCloseable

fix

add setup iterator

more sensible method ordering

make TrivialContext Resettable

a few more missing resettablecontexts

address comments

apply resettable context

forgot to fix one use of AutoCloseable

fix

add setup iterator

more sensible method ordering

remove rogue element type type

make TrivialContext Resettable

wip

wip

wip

wip

use safe row in join suite

pull over hailcontext

remove Region.clear(newEnd)

add selectRegionValue

# This is the commit message #2:

convert relational.scala
;

# This is the commit message #3:

scope the extract aggregators constfb call

# This is the commit message #4:

scope interpret

# This is the commit message #5:

typeAfterSelect used by selectRegionValue

# This is the commit message #6:

load matrix

# This is the commit message #7:

imports

# This is the commit message #8:

loadbgen converted

# This is the commit message #9:

convert loadplink

# This is the commit message hail-is#10:

convert loadgdb

# This is the commit message hail-is#11:

convert loadvcf

# This is the commit message hail-is#12:

convert blockmatrix

# This is the commit message hail-is#13:

convert filterintervals

# This is the commit message hail-is#14:

convert ibd

# This is the commit message hail-is#15:

convert a few methods

# This is the commit message hail-is#16:

convert split multi

# This is the commit message hail-is#17:

convert VEP

# This is the commit message hail-is#18:

formatting fix

# This is the commit message hail-is#19:

add partitionBy and values

# This is the commit message hail-is#20:

fix bug in localkeysort

# This is the commit message hail-is#21:

fixup HailContext.readRowsPartition use

# This is the commit message hail-is#22:

port balding nichols model

* apply resettable context

forgot to fix one use of AutoCloseable

fix

add setup iterator

more sensible method ordering

make TrivialContext Resettable

a few more missing resettablecontexts

address comments

apply resettable context

forgot to fix one use of AutoCloseable

fix

add setup iterator

more sensible method ordering

remove rogue element type type

make TrivialContext Resettable

wip

wip

wip

wip

use safe row in join suite

pull over hailcontext

remove Region.clear(newEnd)

add selectRegionValue

convert relational.scala
;

scope the extract aggregators constfb call

scope interpret

typeAfterSelect used by selectRegionValue

load matrix

imports

loadbgen converted

convert loadplink

convert loadgdb

convert loadvcf

convert blockmatrix

convert filterintervals

convert ibd

convert a few methods

convert split multi

convert VEP

formatting fix

add partitionBy and values

fix bug in localkeysort

fixup HailContext.readRowsPartition use

port balding nichols model

port over table.scala

couple fixes

convert matrix table

remove necessary use of rdd

variety of fixups

wip

add a clear

* Remove direct Region allocation from FilterColsIR

When regions are off-heap, we can allow the globals to live
in a separate, longer-lived Region that is not cleared until
the whole partition is finished. For now, we pay the
memory cost.

* Use RVDContext in MatrixRead zip

This Region will get cleared by consumers.

I introduced the zip primitive which is a safer way to
zip two RVDs because it does not rely on the user correctly
clearing the regions used by the left and right hand sides
of the zip.

* Control the Regions in LoadGDB

I do not fully understand how LoadGDB is working, but a simple
solution to the use-case is to serialize to arrays of bytes
and parallelize those.

I realize there is a proliferation of `coerce` methods. I plan
to trim this down once we do not have RDD and ContextRDD coexisting

* wip

* unify RVD.run

* reset in write

* fixes

* use context region when allocating

* also read RVDs using RVDContext

* formatting

* address comments

* remove unused val

* abstract over boundary

* little fixes

* whoops forgot to clear before persisting

This fixes the LDPrune if you dont clear the region things go wrong.
Not sure what causes that bug. Maybe its something about encoders?

* serialize for shuffles, region.scoped in matrixmapglobals, fix joins

* clear more!

* wip

* wip

* rework GeneralRDD to ease ContextRDD transition

* formatting

* final fixes

* formatting

* merge failures

* more bad merge stuff

* formatting

* remove unnecessary stuff

* remove fixme

* boom!

* variety of merge mistakes

* fix destabilize bug

* add missing newline

* remember to clear the producer region in localkeysort

* switch def to val

* cleanup filteralleles and exporbidbimfam

* fix clearing and serialization issue

* fix BitPackedVectorView

Previously it always assumed the variant struct started at offset
zero, which is not true

* address comments, remove a comment

* remove direct use of Region

* oops

* werrrks, mebbe

* needs cleanup

* fix filter intervals

* fixes

* fixes

* fix filterintervals

* remove unnecessary copy in TableJoin

* and finally fix the last test

* re-use existing CodecSpec definition

* remove unnecessary boundaries

* use RVD abstraction when possible

* formatting

* bugfix: RegionValue must know its region

* remove unnecessary val and comment

* remove unused methods

* eliminate unused constructors

* undo debug change

* formatting

* remove unused imports

* fix bug in tablejoin

* fix RichRDDSuite test

If you have no data, then you have no partitions, not 1 partition
jigold added a commit that referenced this pull request Jan 9, 2019
@jigold
Attempt #2
b26c5cf
jigold added a commit that referenced this pull request Jan 30, 2019
@jigold
Attempt #2
20a774c
jigold added a commit that referenced this pull request Jan 30, 2019
@jigold @danking
Prototype for batch Python interface (hail-is#4937)
c883545 
* Prototype for batch Python interface

* wip

* Attempt #2

* change how temp dir is formatted

* change directory structure

* Add tests and actually execute pipeline

* added project

* better error msg

* added docker support

* Add infrastructure for testing, conda, etc.

* support for new pipeline env

* comment out docker test
jigold pushed a commit that referenced this pull request Oct 23, 2020
[batch] add close billing project functionality (hail-is#9579)
b3d7618 
* [batch] add close billing project functionality

* address comments

* rewrite subquery to take advantage of time_completed index

* fix check

* undo accidental change to query

* tweak table format

* add 'deleted' state for billing projects

* oops

* update estimated-current

* fix

* define custom errors

* address comments

* fix check

* address comments

* fix check

* fix check #2

* fix check #3

* fix check-sql
jigold added a commit that referenced this pull request Sep 27, 2021
@jigold
[batch] Replace GCS with GoogleStorageAsyncFS (attempt #2) (hail-is#1…
d9bb866 
…0814)

* [batch] replace GCS with GoogleStorageAsyncFS in LogStore

* delint

* addr comments

* fix s3 exception handling

* fix catching nonexistent blob in azurefs

* flip version number of instance
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jigold jigold jigold left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@tpoterba @jigold