Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update viral-core 2.3.0 -> 2.3.1 #527

Merged
merged 21 commits into from
Mar 19, 2024
Merged

update viral-core 2.3.0 -> 2.3.1 #527

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
939707d
update viral-core 2.3.0 -> 2.3.1
tomkinsc Mar 12, 2024
a4e9db1
bump nextstrain/base build-20230905T192825Z -> build-20240223T221827Z
tomkinsc Mar 12, 2024
838d3bf
change Seq.ungap("-") to Seq.replace("-","")
tomkinsc Mar 12, 2024
6af69bc
specify docker.io for nextstrain images
tomkinsc Mar 12, 2024
d7276cc
roll back nextstrain/base image version
tomkinsc Mar 12, 2024
e042ccd
roll back nextstrain/base to build-20230905T192825Z
tomkinsc Mar 12, 2024
81dd40a
docker image prune between miniwdl calls to reduce disk space usage o…
tomkinsc Mar 12, 2024
de70ed8
try specifying nextstrain/base image by hash rather than tag
tomkinsc Mar 12, 2024
20678e8
bump python version in GitHub Actions build to 3.10 for all build jobs
tomkinsc Mar 13, 2024
3eb7fee
update nextstrain/base tag
tomkinsc Mar 15, 2024
c738abe
pin nextstrain/base image to build-20240209T204939Z
tomkinsc Mar 15, 2024
e17e195
viral-classify pinned to version 2.2.4.1
tomkinsc Mar 15, 2024
ffc0d64
restore bin of viral-classify image to 2.2.4.0
tomkinsc Mar 18, 2024
b193ff6
debugging: disable exit on pipefail in kraken2 task
tomkinsc Mar 18, 2024
8b58868
update nextstrain/base image to build-20240318T173028Z
tomkinsc Mar 18, 2024
e00d465
update nextstrain/base tags
tomkinsc Mar 18, 2024
94b3d08
list files in cromwell execution directory
tomkinsc Mar 18, 2024
46b142c
wildcard stderr and stdout output
tomkinsc Mar 18, 2024
f452599
change cromwell docker cmd so we wait to remove docker images until d…
tomkinsc Mar 18, 2024
b87ed1b
prune docker images between cromwell tests to save space
tomkinsc Mar 18, 2024
7d0699d
restore pipefail condition for kraken2 command block
tomkinsc Mar 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .github/workflows/build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ jobs:
- name: install python
uses: actions/setup-python@v5
with:
python-version: '3.8'
python-version: '3.10'
- name: install system dependencies
run: |
sudo apt-get update
Expand Down Expand Up @@ -168,7 +168,7 @@ jobs:
- name: install python
uses: actions/setup-python@v5
with:
python-version: '3.8'
python-version: '3.10'
- name: install docs dependencies
shell: bash
run: |
Expand Down Expand Up @@ -339,7 +339,7 @@ jobs:
- name: install python
uses: actions/setup-python@v5
with:
python-version: '3.8'
python-version: '3.10'
- name: install java
uses: actions/setup-java@v4
with:
Expand Down
45 changes: 24 additions & 21 deletions github_actions_ci/tests-cromwell.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,28 +7,31 @@ cp -r test workflows/
cd workflows

for workflow in ../pipes/WDL/workflows/*.wdl; do
workflow_name=$(basename $workflow .wdl)
input_json="test/input/WDL/cromwell-local/test_inputs-$workflow_name-local.json"
if [ -f $input_json ]; then
date
echo "Executing $workflow_name using Cromwell on local instance"
# the "cat" is to allow a pipe failure (otherwise it halts because of set -e)
java -Dconfig.file=../pipes/cromwell/cromwell.local-github_actions.conf \
-jar cromwell.jar run \
$workflow_name.wdl \
-i $input_json | tee cromwell.out
if [ ${PIPESTATUS[0]} -gt 0 ]; then
echo "error running $workflow_name"
error_logs=$(grep stderr cromwell.out | perl -lape 's/.*\s(\S+)$/$1/g')
for log in $error_logs; do
echo "contents of stderr ($log):"
cat `dirname $log`/stderr | sed "s/^/[STDERR] /"
echo "contents of stdout ($log):"
cat `dirname $log`/stdout | sed "s/^/[STDOUT] /"
done
sync; sleep 20; exit 1
fi
workflow_name=$(basename $workflow .wdl)
input_json="test/input/WDL/cromwell-local/test_inputs-$workflow_name-local.json"
if [ -f $input_json ]; then
date
echo "Executing $workflow_name using Cromwell on local instance"
# the "cat" is to allow a pipe failure (otherwise it halts because of set -e)
java -Dconfig.file=../pipes/cromwell/cromwell.local-github_actions.conf \
-jar cromwell.jar run \
$workflow_name.wdl \
-i $input_json | tee cromwell.out
if [ ${PIPESTATUS[0]} -gt 0 ]; then
echo "error running $workflow_name"
error_logs=$(grep stderr cromwell.out | perl -lape 's/.*\s(\S+)$/$1/g')
for log in $error_logs; do
ls -lah $(dirname $log)

echo "contents of stderr ($log):"
cat `dirname $log`/stderr* | sed "s/^/[STDERR] /"
echo "contents of stdout ($log):"
cat `dirname $log`/stdout* | sed "s/^/[STDOUT] /"
done
sync; sleep 20; exit 1
fi
fi
docker image prune --all --force # prune images from this workflow execution to save space before the next
done

cd -
Expand Down
1 change: 1 addition & 0 deletions github_actions_ci/tests-miniwdl.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,7 @@ for workflow in ../pipes/WDL/workflows/*.wdl; do
echo "$workflow_name FAILED"
exit 1
fi
docker image prune --all --force # prune images from this workflow execution to save space before the next
fi
done

Expand Down
4 changes: 2 additions & 2 deletions pipes/WDL/tasks/tasks_assembly.wdl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -450,7 +450,7 @@ task align_reads {
Boolean skip_mark_dupes = false

Int? machine_mem_gb
String docker = "quay.io/broadinstitute/viral-core:2.3.0"
String docker = "quay.io/broadinstitute/viral-core:2.3.1"

String sample_name = basename(basename(basename(reads_unmapped_bam, ".bam"), ".taxfilt"), ".clean")
}
Expand Down Expand Up @@ -847,7 +847,7 @@ task run_discordance {
String out_basename = "run"
Int min_coverage = 4

String docker = "quay.io/broadinstitute/viral-core:2.3.0"
String docker = "quay.io/broadinstitute/viral-core:2.3.1"
}
parameter_meta {
reads_aligned_bam: {
Expand Down
4 changes: 2 additions & 2 deletions pipes/WDL/tasks/tasks_demux.wdl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ task merge_tarballs {
String out_filename

Int? machine_mem_gb
String docker = "quay.io/broadinstitute/viral-core:2.3.0"
String docker = "quay.io/broadinstitute/viral-core:2.3.1"
}

Int disk_size = 2625
Expand Down Expand Up @@ -163,7 +163,7 @@ task illumina_demux {

Int? machine_mem_gb
Int disk_size = 2625
String docker = "quay.io/broadinstitute/viral-core:2.3.0"
String docker = "quay.io/broadinstitute/viral-core:2.3.1"
}

parameter_meta {
Expand Down
2 changes: 1 addition & 1 deletion pipes/WDL/tasks/tasks_interhost.wdl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -351,7 +351,7 @@ task index_ref {
File? novocraft_license

Int? machine_mem_gb
String docker = "quay.io/broadinstitute/viral-core:2.3.0"
String docker = "quay.io/broadinstitute/viral-core:2.3.1"
}

Int disk_size = 100
Expand Down
6 changes: 3 additions & 3 deletions pipes/WDL/tasks/tasks_ncbi.wdl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -192,7 +192,7 @@ task structured_comments {

File? filter_to_ids

String docker = "quay.io/broadinstitute/viral-core:2.3.0"
String docker = "quay.io/broadinstitute/viral-core:2.3.1"
}
String out_base = basename(assembly_stats_tsv, '.txt')
command <<<
Expand Down Expand Up @@ -272,7 +272,7 @@ task rename_fasta_header {

String out_basename = basename(genome_fasta, ".fasta")

String docker = "quay.io/broadinstitute/viral-core:2.3.0"
String docker = "quay.io/broadinstitute/viral-core:2.3.1"
}
command {
set -e
Expand Down Expand Up @@ -437,7 +437,7 @@ task sra_meta_prep {
Boolean paired

String out_name = "sra_metadata.tsv"
String docker="quay.io/broadinstitute/viral-core:2.3.0"
String docker="quay.io/broadinstitute/viral-core:2.3.1"
}
Int disk_size = 100
parameter_meta {
Expand Down
38 changes: 19 additions & 19 deletions pipes/WDL/tasks/tasks_nextstrain.wdl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -280,7 +280,7 @@ task derived_cols {
String? lab_highlight_loc
Array[File] table_map = []

String docker = "quay.io/broadinstitute/viral-core:2.3.0"
String docker = "quay.io/broadinstitute/viral-core:2.3.1"
Int disk_size = 50
}
parameter_meta {
Expand Down Expand Up @@ -450,7 +450,7 @@ task nextstrain_build_subsample {
File? drop_list

Int machine_mem_gb = 50
String docker = "nextstrain/base:build-20230905T192825Z"
String docker = "docker.io/nextstrain/base:build-20240318T173028Z"
String nextstrain_ncov_repo_commit = "30435fb9ec8de2f045167fb90adfec12f123e80a"
Int disk_size = 750
}
Expand Down Expand Up @@ -594,7 +594,7 @@ task nextstrain_build_subsample {
task nextstrain_ncov_defaults {
input {
String nextstrain_ncov_repo_commit = "30435fb9ec8de2f045167fb90adfec12f123e80a"
String docker = "nextstrain/base:build-20230905T192825Z"
String docker = "docker.io/nextstrain/base:build-20240318T173028Z"
Int disk_size = 50
}
command {
Expand Down Expand Up @@ -632,7 +632,7 @@ task nextstrain_deduplicate_sequences {
Boolean error_on_seq_diff = false

String nextstrain_ncov_repo_commit = "30435fb9ec8de2f045167fb90adfec12f123e80a"
String docker = "nextstrain/base:build-20230905T192825Z"
String docker = "docker.io/nextstrain/base:build-20240318T173028Z"
Int disk_size = 750
}

Expand Down Expand Up @@ -686,7 +686,7 @@ task nextstrain_ncov_sanitize_gisaid_data {
String? prefix_to_strip

String nextstrain_ncov_repo_commit = "30435fb9ec8de2f045167fb90adfec12f123e80a"
String docker = "nextstrain/base:build-20230905T192825Z"
String docker = "docker.io/nextstrain/base:build-20240318T173028Z"
Int disk_size = 750
}

Expand Down Expand Up @@ -762,7 +762,7 @@ task filter_subsample_sequences {
Array[String]? exclude_where
Array[String]? include_where

String docker = "nextstrain/base:build-20230905T192825Z"
String docker = "docker.io/nextstrain/base:build-20240318T173028Z"
Int disk_size = 750
}
parameter_meta {
Expand Down Expand Up @@ -847,8 +847,8 @@ task filter_sequences_to_list {
Array[File]? keep_list

String out_fname = sub(sub(basename(sequences, ".zst"), ".vcf", ".filtered.vcf"), ".fasta$", ".filtered.fasta")
# Prior docker image: "nextstrain/base:build-20211012T204409Z"
String docker = "quay.io/broadinstitute/viral-core:2.3.0"
# Prior docker image: "nextstrain/base:build-20240318T173028Z"
String docker = "quay.io/broadinstitute/viral-core:2.3.1"
Int disk_size = 750
}
parameter_meta {
Expand Down Expand Up @@ -1145,7 +1145,7 @@ task augur_mafft_align {
Boolean fill_gaps = true
Boolean remove_reference = true

String docker = "nextstrain/base:build-20230905T192825Z"
String docker = "docker.io/nextstrain/base:build-20240318T173028Z"
Int disk_size = 750
Int mem_size = 180
Int cpus = 64
Expand Down Expand Up @@ -1220,7 +1220,7 @@ task augur_mask_sites {
File sequences
File? mask_bed

String docker = "nextstrain/base:build-20230905T192825Z"
String docker = "docker.io/nextstrain/base:build-20240318T173028Z"
Int disk_size = 750
}
parameter_meta {
Expand Down Expand Up @@ -1279,7 +1279,7 @@ task draft_augur_tree {

Int cpus = 64
Int machine_mem_gb = 32
String docker = "nextstrain/base:build-20230905T192825Z"
String docker = "docker.io/nextstrain/base:build-20240318T173028Z"
Int disk_size = 1250
}
parameter_meta {
Expand Down Expand Up @@ -1348,7 +1348,7 @@ task refine_augur_tree {
String? divergence_units = "mutations"
File? vcf_reference

String docker = "nextstrain/base:build-20230905T192825Z"
String docker = "docker.io/nextstrain/base:build-20240318T173028Z"
Int disk_size = 750
Int machine_mem_gb = 75
}
Expand Down Expand Up @@ -1422,7 +1422,7 @@ task ancestral_traits {
Float? sampling_bias_correction

Int machine_mem_gb = 32
String docker = "nextstrain/base:build-20230905T192825Z"
String docker = "docker.io/nextstrain/base:build-20240318T173028Z"
Int disk_size = 750
}
String out_basename = basename(tree, '.nwk')
Expand Down Expand Up @@ -1475,7 +1475,7 @@ task ancestral_tree {
File? vcf_reference
File? output_vcf

String docker = "nextstrain/base:build-20230905T192825Z"
String docker = "docker.io/nextstrain/base:build-20240318T173028Z"
Int disk_size = 300
}
parameter_meta {
Expand Down Expand Up @@ -1536,7 +1536,7 @@ task translate_augur_tree {
File? vcf_reference_output
File? vcf_reference

String docker = "nextstrain/base:build-20230905T192825Z"
String docker = "docker.io/nextstrain/base:build-20240318T173028Z"
Int disk_size = 300
}
String out_basename = basename(tree, '.nwk')
Expand Down Expand Up @@ -1593,7 +1593,7 @@ task tip_frequencies {
Boolean include_internal_nodes = false

Int machine_mem_gb = 64
String docker = "nextstrain/base:build-20230905T192825Z"
String docker = "docker.io/nextstrain/base:build-20240318T173028Z"
String out_basename = basename(tree, '.nwk')
Int disk_size = 200
}
Expand Down Expand Up @@ -1653,7 +1653,7 @@ task assign_clades_to_nodes {
File ref_fasta
File clades_tsv

String docker = "nextstrain/base:build-20230905T192825Z"
String docker = "docker.io/nextstrain/base:build-20240318T173028Z"
Int disk_size = 300
}
String out_basename = basename(basename(tree_nwk, ".nwk"), "_timetree")
Expand Down Expand Up @@ -1698,7 +1698,7 @@ task augur_import_beast {
String? tip_date_delimiter

Int machine_mem_gb = 3
String docker = "nextstrain/base:build-20230905T192825Z"
String docker = "docker.io/nextstrain/base:build-20240318T173028Z"
Int disk_size = 150
}
String tree_basename = basename(beast_mcc_tree, ".tree")
Expand Down Expand Up @@ -1759,7 +1759,7 @@ task export_auspice_json {
String out_basename = basename(basename(tree, ".nwk"), "_timetree")

Int machine_mem_gb = 64
String docker = "nextstrain/base:build-20230905T192825Z"
String docker = "docker.io/nextstrain/base:build-20240318T173028Z"
Int disk_size = 300
}

Expand Down
14 changes: 7 additions & 7 deletions pipes/WDL/tasks/tasks_read_utils.wdl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,7 @@ task group_bams_by_sample {
task get_bam_samplename {
input {
File bam
String docker = "quay.io/broadinstitute/viral-core:2.3.0"
String docker = "quay.io/broadinstitute/viral-core:2.3.1"
}
Int disk_size = round(size(bam, "GB")) + 50
command <<<
Expand All @@ -111,7 +111,7 @@ task get_sample_meta {
input {
Array[File] samplesheets_extended

String docker = "quay.io/broadinstitute/viral-core:2.3.0"
String docker = "quay.io/broadinstitute/viral-core:2.3.1"
}
Int disk_size = 50
command <<<
Expand Down Expand Up @@ -172,7 +172,7 @@ task merge_and_reheader_bams {
File? reheader_table
String out_basename = basename(in_bams[0], ".bam")

String docker = "quay.io/broadinstitute/viral-core:2.3.0"
String docker = "quay.io/broadinstitute/viral-core:2.3.1"
}

Int disk_size = 750
Expand Down Expand Up @@ -244,7 +244,7 @@ task rmdup_ubam {
String method = "mvicuna"

Int machine_mem_gb = 7
String docker = "quay.io/broadinstitute/viral-core:2.3.0"
String docker = "quay.io/broadinstitute/viral-core:2.3.1"
}

Int disk_size = 375
Expand Down Expand Up @@ -303,7 +303,7 @@ task downsample_bams {
Boolean deduplicateAfter = false

Int? machine_mem_gb
String docker = "quay.io/broadinstitute/viral-core:2.3.0"
String docker = "quay.io/broadinstitute/viral-core:2.3.1"
}

Int disk_size = 750
Expand Down Expand Up @@ -367,7 +367,7 @@ task FastqToUBAM {
String? sequencing_center
String? additional_picard_options

String docker = "quay.io/broadinstitute/viral-core:2.3.0"
String docker = "quay.io/broadinstitute/viral-core:2.3.1"
}
Int disk_size = 375
parameter_meta {
Expand Down Expand Up @@ -418,7 +418,7 @@ task read_depths {
File aligned_bam

String out_basename = basename(aligned_bam, '.bam')
String docker = "quay.io/broadinstitute/viral-core:2.3.0"
String docker = "quay.io/broadinstitute/viral-core:2.3.1"
}
Int disk_size = 200
command <<<
Expand Down
Loading