groHMM overhaul #165

edmundmiller · 2024-09-26T19:44:23Z

Closes #127

Massive overhaul of the groHMM system. The parameter tuning steps now have a scatter-gather pattern to significantly speed up the total runtime and better utilize available resources.

PR checklist

github-actions · 2024-09-26T19:47:01Z

`nf-core pipelines lint` overall result: Passed ✅ ⚠️

Posted for pipeline commit 9c65113

+| ✅ 216 tests passed       |+
#| ❔   4 tests were ignored |#
!| ❗   4 tests had warnings |!

❗ Test warnings:

pipeline_todos - TODO string in nextflow.config: Optionally, you can add a pipeline-specific nf-core config at https://github.com/nf-core/configs
pipeline_todos - TODO string in README.md: Describe the minimum required steps to execute the pipeline, e.g. how to prepare samplesheets.
pipeline_todos - TODO string in base.config: Check the defaults for all processes
pipeline_todos - TODO string in methods_description_template.yml: #Update the HTML below to your preferred methods description, e.g. add publication citation for this pipeline

❔ Tests ignored:

files_unchanged - File ignored due to lint config: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_unchanged - File ignored due to lint config: .github/workflows/linting.yml
files_unchanged - File ignored due to lint config: assets/email_template.html
actions_ci - actions_ci

✅ Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: .gitignore
files_exist - File found: .nf-core.yml
files_exist - File found: .editorconfig
files_exist - File found: .prettierignore
files_exist - File found: .prettierrc.yml
files_exist - File found: CHANGELOG.md
files_exist - File found: CITATIONS.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/CONTRIBUTING.md
files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
files_exist - File found: .github/ISSUE_TEMPLATE/config.yml
files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File found: .github/workflows/branch.yml
files_exist - File found: .github/workflows/ci.yml
files_exist - File found: .github/workflows/linting_comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: assets/nf-core-nascent_logo_light.png
files_exist - File found: conf/modules.config
files_exist - File found: conf/test.config
files_exist - File found: conf/test_full.config
files_exist - File found: docs/images/nf-core-nascent_logo_light.png
files_exist - File found: docs/images/nf-core-nascent_logo_dark.png
files_exist - File found: docs/output.md
files_exist - File found: docs/README.md
files_exist - File found: docs/README.md
files_exist - File found: docs/usage.md
files_exist - File found: main.nf
files_exist - File found: assets/multiqc_config.yml
files_exist - File found: conf/base.config
files_exist - File found: conf/igenomes.config
files_exist - File found: conf/igenomes_ignored.config
files_exist - File found: .github/workflows/awstest.yml
files_exist - File found: .github/workflows/awsfulltest.yml
files_exist - File found: modules.json
files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .markdownlint.yml
files_exist - File not found check: .nf-core.yaml
files_exist - File not found check: .yamllint.yml
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: docs/images/nf-core-nascent_logo.png
files_exist - File not found check: lib/Checks.groovy
files_exist - File not found check: lib/Completion.groovy
files_exist - File not found check: lib/NfcoreTemplate.groovy
files_exist - File not found check: lib/Utils.groovy
files_exist - File not found check: lib/Workflow.groovy
files_exist - File not found check: lib/WorkflowMain.groovy
files_exist - File not found check: lib/WorkflowNascent.groovy
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: pipeline_template.yml
files_exist - File not found check: Singularity
files_exist - File not found check: lib/nfcore_external_java_deps.jar
files_exist - File not found check: .travis.yml
nextflow_config - Found nf-schema plugin
nextflow_config - Config variable found: manifest.name
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: manifest.homePage
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: process.cpus
nextflow_config - Config variable found: process.memory
nextflow_config - Config variable found: process.time
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: params.input
nextflow_config - Config variable found: validation.help.enabled
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable found: validation.help.beforeText
nextflow_config - Config variable found: validation.help.afterText
nextflow_config - Config variable found: validation.help.command
nextflow_config - Config variable found: validation.summary.beforeText
nextflow_config - Config variable found: validation.summary.afterText
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config variable (correctly) not found: params.enable_conda
nextflow_config - Config variable (correctly) not found: params.max_cpus
nextflow_config - Config variable (correctly) not found: params.max_memory
nextflow_config - Config variable (correctly) not found: params.max_time
nextflow_config - Config variable (correctly) not found: params.validationFailUnrecognisedParams
nextflow_config - Config variable (correctly) not found: params.validationLenientMode
nextflow_config - Config variable (correctly) not found: params.validationSchemaIgnoreParams
nextflow_config - Config variable (correctly) not found: params.validationShowHiddenParams
nextflow_config - Config timeline.enabled had correct value: true
nextflow_config - Config report.enabled had correct value: true
nextflow_config - Config trace.enabled had correct value: true
nextflow_config - Config dag.enabled had correct value: true
nextflow_config - Config manifest.name began with nf-core/
nextflow_config - Config variable manifest.homePage began with https://github.com/nf-core/
nextflow_config - Config dag.file ended with .html
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - Config manifest.version ends in dev: 2.3.0dev
nextflow_config - Config params.custom_config_version is set to master
nextflow_config - Config params.custom_config_base is set to https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Lines for loading custom profiles found
nextflow_config - nextflow.config contains configuration profile test
nextflow_config - Config default value correct: params.aligner= bwa
nextflow_config - Config default value correct: params.skip_grohmm= false
nextflow_config - Config default value correct: params.grohmm_min_uts= 5
nextflow_config - Config default value correct: params.grohmm_max_uts= 45
nextflow_config - Config default value correct: params.grohmm_min_ltprobb= -100
nextflow_config - Config default value correct: params.grohmm_max_ltprobb= -400
nextflow_config - Config default value correct: params.igenomes_base= s3://ngi-igenomes/igenomes/
nextflow_config - Config default value correct: params.human_pangenomics_base= https://s3-us-west-2.amazonaws.com/human-pangenomics
nextflow_config - Config default value correct: params.custom_config_version= master
nextflow_config - Config default value correct: params.custom_config_base= https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Config default value correct: params.publish_dir_mode= copy
nextflow_config - Config default value correct: params.max_multiqc_email_size= 25.MB
nextflow_config - Config default value correct: params.validate_params= true
nextflow_config - Config default value correct: params.pipelines_testdata_base_path= https://raw.githubusercontent.com/nf-core/test-datasets/
files_unchanged - .gitattributes matches the template
files_unchanged - .prettierrc.yml matches the template
files_unchanged - CODE_OF_CONDUCT.md matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/CONTRIBUTING.md matches the template
files_unchanged - .github/ISSUE_TEMPLATE/bug_report.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/config.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
files_unchanged - .github/PULL_REQUEST_TEMPLATE.md matches the template
files_unchanged - .github/workflows/branch.yml matches the template
files_unchanged - .github/workflows/linting_comment.yml matches the template
files_unchanged - assets/email_template.txt matches the template
files_unchanged - assets/sendmail_template.txt matches the template
files_unchanged - assets/nf-core-nascent_logo_light.png matches the template
files_unchanged - docs/images/nf-core-nascent_logo_light.png matches the template
files_unchanged - docs/images/nf-core-nascent_logo_dark.png matches the template
files_unchanged - docs/README.md matches the template
files_unchanged - .gitignore matches the template
files_unchanged - .prettierignore matches the template
actions_awstest - '.github/workflows/awstest.yml' is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml does not use -profile test
readme - README Nextflow minimum version badge matched config. Badge: 24.04.2, Config: 24.04.2
readme - README Zenodo placeholder was replaced with DOI.
plugin_includes - No wrong validation plugin imports have been found
pipeline_name_conventions - Name adheres to nf-core convention
template_strings - Did not find any Jinja template strings (0 files)
schema_lint - Schema lint passed
schema_lint - Schema title + description lint passed
schema_lint - Input mimetype lint passed: 'text/csv'
schema_params - Schema matched params returned from nextflow config
system_exit - No System.exit calls found
actions_schema_validation - Workflow validation passed: linting.yml
actions_schema_validation - Workflow validation passed: branch.yml
actions_schema_validation - Workflow validation passed: fix-linting.yml
actions_schema_validation - Workflow validation passed: release-announcements.yml
actions_schema_validation - Workflow validation passed: awsfulltest.yml
actions_schema_validation - Workflow validation passed: template_version_comment.yml
actions_schema_validation - Workflow validation passed: download_pipeline.yml
actions_schema_validation - Workflow validation passed: ci.yml
actions_schema_validation - Workflow validation passed: clean-up.yml
actions_schema_validation - Workflow validation passed: awstest.yml
actions_schema_validation - Workflow validation passed: linting_comment.yml
merge_markers - No merge markers found in pipeline files
modules_json - Only installed modules found in modules.json
multiqc_config - assets/multiqc_config.yml found and not ignored.
multiqc_config - assets/multiqc_config.yml contains report_section_order
multiqc_config - assets/multiqc_config.yml contains export_plots
multiqc_config - assets/multiqc_config.yml contains report_comment
multiqc_config - assets/multiqc_config.yml follows the ordering scheme of the minimally required plugins.
multiqc_config - assets/multiqc_config.yml contains a matching 'report_comment'.
multiqc_config - assets/multiqc_config.yml contains 'export_plots: true'.
modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'
base_config - conf/base.config found and not ignored.
modules_config - conf/modules.config found and not ignored.
modules_config - FASTQC found in conf/modules.config and Nextflow scripts.
modules_config - MULTIQC found in conf/modules.config and Nextflow scripts.
modules_config - GFFREAD found in conf/modules.config and Nextflow scripts.
modules_config - FASTP found in conf/modules.config and Nextflow scripts.
modules_config - BWA_MEM found in conf/modules.config and Nextflow scripts.
modules_config - BWAMEM2_MEM found in conf/modules.config and Nextflow scripts.
modules_config - BOWTIE2_ALIGN found in conf/modules.config and Nextflow scripts.
modules_config - DRAGMAP_ALIGN found in conf/modules.config and Nextflow scripts.
modules_config - STAR_ALIGN found in conf/modules.config and Nextflow scripts.
modules_config - PRESEQ_ found in conf/modules.config and Nextflow scripts.
modules_config - BBMAP_PILEUP found in conf/modules.config and Nextflow scripts.
modules_config - BEDTOOLS_GENOMECOV_PLUS found in conf/modules.config and Nextflow scripts.
modules_config - BEDTOOLS_GENOMECOV_MINUS found in conf/modules.config and Nextflow scripts.
modules_config - DEEPTOOLS_BAMCOVERAGE_PLUS found in conf/modules.config and Nextflow scripts.
modules_config - DEEPTOOLS_BAMCOVERAGE_MINUS found in conf/modules.config and Nextflow scripts.
modules_config - DREG_PREP found in conf/modules.config and Nextflow scripts.
modules_config - HOMER_ found in conf/modules.config and Nextflow scripts.
modules_config - HOMER_FINDPEAKS found in conf/modules.config and Nextflow scripts.
modules_config - HOMER_MAKETAGDIRECTORY found in conf/modules.config and Nextflow scripts.
modules_config - HOMER_MAKEUCSCFILE found in conf/modules.config and Nextflow scripts.
modules_config - PINTS_CALLER found in conf/modules.config and Nextflow scripts.
modules_config - GROHMM_ found in conf/modules.config and Nextflow scripts.
modules_config - BEDTOOLS_SORT found in conf/modules.config and Nextflow scripts.
modules_config - BEDTOOLS_MERGE found in conf/modules.config and Nextflow scripts.
modules_config - BEDTOOLS_INTERSECT_FILTER found in conf/modules.config and Nextflow scripts.
modules_config - BEDTOOLS_INTERSECT found in conf/modules.config and Nextflow scripts.
modules_config - SUBREAD_FEATURECOUNTS_GENE found in conf/modules.config and Nextflow scripts.
modules_config - SUBREAD_FEATURECOUNTS_PREDICTED found in conf/modules.config and Nextflow scripts.
nfcore_yml - Repository type in .nf-core.yml is valid: pipeline
nfcore_yml - nf-core version in .nf-core.yml is set to the latest version: 3.0.2

Run details

nf-core/tools version 3.0.2
Run at 2024-10-22 17:18:24

edmundmiller · 2024-10-21T15:09:01Z

I'm giving up on getting #172 fixed in this PR. I'll make a follow-up but I don't want to get held up trying to make this perfect.

Apparently we weren't doing this. It's in the guidelines and in Shayne's scripts

Since these are are split into seperate jobs now, they're not "long"(longest I've seen one take is ~143 minutes). Adding arrays later because there's so many of them and they're the same

Didn't know that was a thing!

With the help of cursor

refactor: Clean up code more

Scrap all of the CHM13 attempts I think it's the gtf conversion stuff that's causing the memory overload. But I can get them all to load locally. So I'm going to try to use gff3 files, if they're supplied, otherwise fall back to gtf.

Editorconfig, styler, consistant naming

edmundmiller changed the base branch from master to dev September 26, 2024 19:45

edmundmiller self-assigned this Sep 26, 2024

nf-core deleted a comment from github-actions bot Sep 26, 2024

edmundmiller force-pushed the grohmm-overhaul branch 3 times, most recently from 75040bc to 0ca9dce Compare September 30, 2024 12:36

edmundmiller force-pushed the grohmm-overhaul branch 2 times, most recently from abc2eea to 308911a Compare October 9, 2024 17:54

edmundmiller marked this pull request as ready for review October 15, 2024 13:34

edmundmiller force-pushed the grohmm-overhaul branch from 41b5e55 to 505251b Compare October 15, 2024 13:45

edmundmiller force-pushed the dev branch from c63bfba to a9ae47a Compare October 15, 2024 17:25

edmundmiller force-pushed the grohmm-overhaul branch 6 times, most recently from b6a93ca to e7b2513 Compare October 20, 2024 02:51

edmundmiller mentioned this pull request Oct 21, 2024

groHMM fails with CHM13 occasionally #172

Open

edmundmiller added 10 commits October 21, 2024 10:09

style: Run styler

1df35bd

test: Write grohmm tests

f7ddbb6

fix(grohmm): bam => bams

21ba6f7

fix(grohmm): Link up tuning files and samples

39265aa

refactor(grohmm): Add notes of brainstorming

f431c73

feat: Add grohmm min and max tuning params

727fdd1

fix: of => fromList

bb59488

refactor(grohmm): Use parameter tuning split

ab20315

fix(grohmm): Use windowAnalysis

6e0b9a8

Apparently we weren't doing this. It's in the guidelines and in Shayne's scripts

refactor: Setup transcript calling

114136c

edmundmiller added 21 commits October 21, 2024 10:11

feat(grohmm): Add Native MultiQC support

d9e94aa

fix(grohmm): Update labels for parametertuning

a535c8c

Since these are are split into seperate jobs now, they're not "long"(longest I've seen one take is ~143 minutes). Adding arrays later because there's so many of them and they're the same

chore: Add a note about the subworkflow functionality

2f1c81c

chore: oras => https

cd9151e

refactor(grohmm): Use each input

26626fa

Didn't know that was a thing!

test(grohmm): Write a failing test

9272060

With the help of cursor

fix(grohmm): Update to work with CHM13

3d5cf85

refactor: Clean up code more

refactor: Replace jpg with png

b8548a3

test: Remove skip_tuning test

0c7a9ba

test: Bump snapshot

0958778

Update CHANGELOG

d17d9df

docs: Update groHMM docs

3c947b0

fix(grohmm): Try calling no more than 10 cores

2b2b90f

chore: Add a custom makeConsensusAnnotations function

e101553

Try creating a custom makeConsensusAnnotations and setup test

3028ae1

Get custom running

d4ac91f

Start over

3444c5b

refactor: Split into chunks

508452b

More tests

b538a16

fix(grohmm): Support gxf

ef677b1

Scrap all of the CHM13 attempts I think it's the gtf conversion stuff that's causing the memory overload. But I can get them all to load locally. So I'm going to try to use gff3 files, if they're supplied, otherwise fall back to gtf.

fix(chm13): Prevent some errors during transcriptcalling

afbb7b6

edmundmiller force-pushed the grohmm-overhaul branch 2 times, most recently from e26248a to 54759ca Compare October 21, 2024 15:58

style(grohmm): Clean up

5c6b24a

Editorconfig, styler, consistant naming

edmundmiller force-pushed the grohmm-overhaul branch from 54759ca to 5c6b24a Compare October 21, 2024 16:08

edmundmiller changed the title ~~Grohmm overhaul~~ groHMM overhaul Oct 21, 2024

sateeshperi approved these changes Oct 22, 2024

View reviewed changes

chore: Move test data to test-datasets repo

9c65113

edmundmiller merged commit bb65227 into dev Oct 23, 2024
12 checks passed

edmundmiller added this to the 2.3.0 milestone Nov 26, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

groHMM overhaul #165

groHMM overhaul #165

edmundmiller commented Sep 26, 2024 •

edited

Loading

github-actions bot commented Sep 26, 2024 •

edited

Loading

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

edmundmiller commented Oct 21, 2024

groHMM overhaul #165

groHMM overhaul #165

Conversation

edmundmiller commented Sep 26, 2024 • edited Loading

PR checklist

github-actions bot commented Sep 26, 2024 • edited Loading

nf-core pipelines lint overall result: Passed ✅ ⚠️

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

edmundmiller commented Oct 21, 2024

edmundmiller commented Sep 26, 2024 •

edited

Loading

github-actions bot commented Sep 26, 2024 •

edited

Loading

`nf-core pipelines lint` overall result: Passed ✅ ⚠️