Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

metagenomic additions #60

Merged
merged 51 commits into from
May 11, 2020
Merged

metagenomic additions #60

merged 51 commits into from
May 11, 2020

Conversation

dpark01
Copy link
Member

@dpark01 dpark01 commented May 7, 2020 •
edited
Loading

  • adds kraken2 classification task and workflow
  • adds kraken2 as part of demux_metag workflow
  • adds kraken2_build task and workflow
  • adds blastx task, taking contig inputs and emitting a krona plot using ktImportBLAST
  • adds classify_multi workflow which is basically demux_metag without the demux
  • adds local "tinytest" for kraken2 classification that is small enough to run in a travis vm and the db fits in the github repo
  • removes default value of minscoretofilter=60 from align_and_count (spike-in counting task) due to out-of-memory issues on large bam inputs (custom python code runs a big in-memory map on read ids)
  • adds a default input value for samplename for assemble_refbased to simplify the one-bam-input scenario

dpark01 added 30 commits May 5, 2020 12:57
@dpark01
first pass at kraken2 build and classification
3eb2c95
@dpark01
missing comma
7e90484
@dpark01
documentation
3696a16
@dpark01
tune up tasks a bit, more documentation
33cdfdb
@dpark01
unquote array of inputs
a81ed6e
@dpark01
fix multiple optional arg argparse passing
c17ecae
@dpark01
remove plant and nt from default kraken2 db build
15e20bc
@dpark01
change TMPDIR handling in TMPDIR-intensive tasks, add a blastx metage…
2c5fa4c 
…nomics task
@dpark01
vm tuning
0e9d44f
@dpark01
hm, this seems to have to be exported
eea92cd
@dpark01
test CI runs on new container
e13d36f
@dpark01
add workflows for single and batch classification via k2
d55b74f
@dpark01
remove a stray double quote
2445d93
@dpark01
add kraken2 tinytest
abcb9bb
@dpark01
more ram for build
dc48420
@dpark01
new tagged release
5a1f3bf
@dpark01
bugfix input variable
2e7a710
@dpark01
forgot to specify CWD for tar -C; otherwise it cowardly refuses to cr…
295f863 
…eate an empty archive
@dpark01
vm tuning for kraken2, reduce zstd compression level for kraken2 and …
1e0aaab 
…krakenuniq build process to hit a more optimal speed, expose optional params for kraken2
@dpark01
actually just parameterize the zstd compression level
4c3927d
@dpark01
make zstd compression level parameter fully optional and default to 3…
f9d596b 
… (zstd default), initial trial attempts at a dnanexus-like heartbeat monitor
@dpark01
attempt to reduce dnanexus CI test of demux_plus to kraken tiny db
5b8c3ad
@dpark01
remove the vmstat heartbeat for now... work on it more later
df49ad4
@dpark01
Revert "attempt to reduce dnanexus CI test of demux_plus to kraken ti…
73e7f71 
…ny db"

(the tarball is missing the taxdb.. need to fix that)
This reverts commit 5b8c3ad.
@dpark01
resize blastx vm
4a9e3dd
@dpark01
try new tinytest for krakenuniq dnanexus CI
b0b20d6
@dpark01
add dx defaults pointing to new resources
af0e097
@dpark01
remove --minScoreToFilter default value from task align_and_count, si…
ca31b6d 
…nce it causes out of memory issues for large bam inputs
@dpark01
name nr
ebaba69
@dpark01
in build_kraken2_db, parallelize all the decompressions
512faf0
dpark01 added 21 commits May 7, 2020 22:28
@dpark01
vm tuning and change blastx to output fmt 7
77f74c8
@dpark01
vm reshape for kraken2, unset the defaults for the optional params, a…
a327dd9 
…nd add a du summary of the size of the db
@dpark01
update task filter_bam_to_taxa to latest cli invocation and options; …
992ad4f 
…update classify_multi workflow to use kraken2 based human read depletion, add dnanexus CI test for classify_multi
@dpark01
stick to dev branch for classify for current dev work
00d23d0
@dpark01
oops
8499966
@dpark01
remove old deplete_taxa inputs
0a6f54f
@dpark01
fix directory structure for taxdump tarball, add meta and parameter_meta
a165b06
@dpark01
correct dx instance type
d3deaf0
@dpark01
remove blastx from clasify_multi by default
70445e9
@dpark01
wdl syntax fixups to classify_multi
42e329a
@dpark01
attempt independent scatter blocks
a2e0ada
@dpark01
simplify krona basenames and add more test input files to dx
dd31807
@dpark01
add default value for assemble_refbased.sample_name for simple scenarios
31a698c
@dpark01
make krona merge of kraken2 results take the original summary report …
ac9bf5a 
…instead of merging html files
@dpark01
provide taxdb for krona invocation
0363246
@dpark01
mammalia doesnt have children? go for vertebrata
c76e7a3
@dpark01
double quote --taxonomic_names so that we can pass taxonomy strings w…
3c6240c 
…ith spaces in them
@dpark01
make filter_bam_to_taxa tolerant of spaces in taxonomic_names array s…
25126a5 
…trings
@dpark01
update viral-classify to 2.0.21.3 and add new optional param from kra…
62078c2 
…ken2 2.0.9beta
@dpark01
explicitly ask for kraken2-type input for krona call
eb11d93
@dpark01
update demux_plus and classify_krakenuniq default krona taxonomy data…
7c1caa3 
…bases to be more current than the kuniq dbs
@dpark01
Copy link
Member Author

dpark01 commented May 11, 2020

The code looks good and is behaving as it should. As for the currently poor performance of classification, we can tune the db and parameters later -- this PR does not change demux_plus or preexisting production workflows to kraken2 yet.

@dpark01 dpark01 merged commit 18386c6 into master May 11, 2020
@dpark01 dpark01 deleted the dp-metagenomics branch May 11, 2020 16:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@dpark01