Proposed treevalgal work plan - December 4 2013

[fubar2]

Decomposing the TreeVal NF workflow for translation into Galaxy.

December 12 status

repeat_density is ready for testing and comment.
Until window_masker is available for the usegalaxy.* servers, it cannot be used on them, so you will need to use a server where it is available, to import this workflow.

It is extremely uncomplicated compared to the NF DDL because JBrowse does most of the work. Example screen grabs show how it works.

December 10 status

gap_finder is ready for testing and comment.
A new IUC tool implementing seqtk_telo is ready for testing and comment as a telo_finder replacement.

December 4 status.

A first pass over ll the NF DDL for the rapid TreeVal workflow has been completed so each of those subworkflows has been documented by someone who has never used NF :) so needs review.

Suggest we choose some modules that the VGP would want to test out and get prototypes implemented.

gap_finder is a good candidate:

• It might satisfy the 3rd requirement on Nadolina's list
• All the tools are available - except GAP_LENGTH

GAP_LENGTH is a local nf module that runs awk. awk does not have an abs function, so seems a strange choice for the task of enforcing positive values and writing them to the end of each bed file row - we can do better but this is very specific for bed files.:

    $/
    cat "${file}" \
    | awk '{print $0"\t"sqrt(($3-$2)*($3-$2))}' > ${prefix}_gap.bedgraph

There is a generic tool that can perform this function but requires the workflow builder to configure it correctly. A dedicated tool that does nothing else so requires no configuration is also available. Which is better for users?

December 3 status

Former content of this document with background, resources and development content has been moved from a Google doc to this treeval_gal repository

Anna and Bjoern will join in.

Work plan:

Planning for Galaxy components that the VGP really need is the long term goal.
Having exemplar working tools and subworkflow prototypes to show, will really help focus the conversations.

The current proposal is to prepare some demonstrations of the rapid workflow logic in Galaxy, while gathering and documenting requirements from the VGP at the same time for harmonisation.

1. Review the first pass at deconstructing the actual NF steps - see the top level readme for links
2. Create or reuse suitable Galaxy components for at least one potentially useful example
3. Make initial subworkflows
4. Test and test again.
5. Testing plan will be needed. Default test data is available.
6. Documentation/training plan will be needed.
7. Publish something about converting a complex NF workflow into Galaxy

Notes:

1. Nadolina has suggested:

   Regardless, the main things curators need are:

   1. a brief pipeline that prepares the decontaminated draft assemblies for dual curation
      *some text manipulation so we don't have duplicate names
      *concatenating the haplotypes into a single fasta
   2. a coverage track in bedgraph/bigwig format, derived from an alignment of HiFi reads to the diploid assembly
   3. gap track, which can be derived from the HiFi alignment, or from gfastats [fasta] -b gaps, or a couple of other tools I am sure
   4. telomere track, also in bedgraph or bigwig
   5. close comparator alignment (i.e./ with mashmap)

2. All dependencies will eventually go into Conda and tools to IUC. Development will necessarily be in a working environment and may temporarily involve non-conda dependencies for demonstration purposes.

Proposed action:

• Ross has finished a first pass through the DDL to document the rapid workflow ready for review
• Ross has made a new repository with a structure to reflect the treeval subworkflows containing the documentation for each and eventually, the tools and subworkflows for testing. Moved the material from this document below there.
• Bjoern and Anna please plan to figure out what VGP really needs.
  • Initially, having treeval rapid available, even if not optimal, will provide a working model for users to identify improvements.
  • Maybe walk them through the three things that might meet some of Nadolina’s listed needs - gap_finder, longread_coverage and telo_finder to see if they would be a useful start and artifacts to test, while discussions continue.
  • Not sure if the decontaminated draft preparation for dual curation on Nadolina's list is done anywhere - perhaps a new project?
  • mashmap is not in the toolfactory so a new project/tool?
  • Telo_finder is an interesting challenge. We could initially just use what Sanger uses in the short term for testing at least.
  • Show the VGP people the Sanger outputs page - there might be other useful subworkflows. There is a synteny subworkflow but not used in the rapid one - it uses minimap2 and needs a class specific set of synteny genomes so a datamanager probably needed 🙁

Please see the treeval_gal repository for details.

[Nadolina]

• There is a working galaxy decontamination pipeline, but Anton is nearly done working on an implementation of NCBI's FCS. When that is done, we will have to test the tool and reconfigure the current pipeline.
• Yes, I don't think mashmap is installed right now; I use it at several points in curation and do prefer it to mummer/nucmer, so it would be helpful to have installed.
• I've had problems installing telofinder myself and have always resorted to tidk-explore, even though it is not my preference. Identifying telomeres is trickier than one might think and I would say there is no tool available right now that is totally comprehensive, although telofinder has been the gold standard recently. I am very curious about this other seqtk tool!!
• I don't use gap finder when generating curation data on my own; I either grep for the 0 coverage lines on the bedgraph file produced by genomecov or using gfastats [fasta] -b gaps.