Revised blank handling/copying #158
Conversation
Method allows caller to copy the fastq files associated w/a sample-name and a project into another project.
copy_sequences() will now copy all replicates if a sample contains replicates. Copying a single replicate is no longer an option.
Updates from Charlie's repo
Charlie's changes from master
# Conflicts: # sequence_processing_pipeline/Pipeline.py
Updates from master by Charlie
|
|
||
| # regex based on studying all filenames of all fastq files in | ||
| # $WKDIR. Works with _R1_, _R2_, _I1_, _I2_, etc. | ||
| rgx = r"^" + re.escape(curr_sample_info[SS_SAMPLE_ID_KEY]) + \ |
There was a problem hiding this comment.
This won't work for how we (as far as I know) generate filenames for tellseq, which are C\d\d\d.[R,I]\d.fastq.gz if I remember right
There was a problem hiding this comment.
If the sample sheet describes a paired-end sequencing run, should we assert that both R1 and R2 for a sample were captured?
There was a problem hiding this comment.
You are correct, this regex won't work for the fastq files generated by tellread. However we do have a post-processing step that will rename the files into something fitting the above pattern used by Illumina machines. The C\d\d\d component will be mapped to the appropriate sample-name/id based on what's defined in the sample-sheet. The trailing digits '_001.fastq.gz' will always be '_001'; technically this isn't adding false metadata because Illumina documentation says this is always '_001' for bcl-convert/bcl2fastq output as well.
L\d\d\d is of course the lane value. The only current issue is what value to give for S\d\d\d. Looking into that right now.
There was a problem hiding this comment.
Just to add, this code will not execute over TellRead output to begin with so this is a non-issue.
There was a problem hiding this comment.
We use controls with tell-seq, why isn't it applicable?
There was a problem hiding this comment.
The scope of this ConvertJob object is just to wrap bcl-convert/bcl2fastq and perform pre/post processing on their output. This functionality will need to be added to the new TellReadJob() class as well.
There was a problem hiding this comment.
@charles-cowart , maybe we need to take this one off-line for more clarification; not sure I understand whether the new TellReadJob class will be using this functionality or different functionality to copy controls when needed.
|
|
||
| # regex based on studying all filenames of all fastq files in | ||
| # $WKDIR. Works with _R1_, _R2_, _I1_, _I2_, etc. | ||
| rgx = r"^" + re.escape(curr_sample_info[SS_SAMPLE_ID_KEY]) + \ |
There was a problem hiding this comment.
You are correct, this regex won't work for the fastq files generated by tellread. However we do have a post-processing step that will rename the files into something fitting the above pattern used by Illumina machines. The C\d\d\d component will be mapped to the appropriate sample-name/id based on what's defined in the sample-sheet. The trailing digits '_001.fastq.gz' will always be '_001'; technically this isn't adding false metadata because Illumina documentation says this is always '_001' for bcl-convert/bcl2fastq output as well.
L\d\d\d is of course the lane value. The only current issue is what value to give for S\d\d\d. Looking into that right now.
| @@ -133,6 +145,34 @@ class Pipeline: | |||
|
|
|||
| assay_types = [AMPLICON_ATYPE, METAGENOMIC_ATYPE, METATRANSCRIPTOMIC_ATYPE] | |||
|
|
|||
| @staticmethod | |||
| def make_sif_fname(run_id, full_project_name): | |||
There was a problem hiding this comment.
sample-information file. 😄
|
|
||
| # regex based on studying all filenames of all fastq files in | ||
| # $WKDIR. Works with _R1_, _R2_, _I1_, _I2_, etc. | ||
| rgx = r"^" + re.escape(curr_sample_info[SS_SAMPLE_ID_KEY]) + \ |
There was a problem hiding this comment.
Just to add, this code will not execute over TellRead output to begin with so this is a non-issue.
Pull Request Test Coverage Report for Build 11750317921Details
💛 - Coveralls |
|
@wasade , could you take a look at the outstanding comments and let me know if there are any you feel strongly about getting addressed before this gets merged into main? |
This code includes Charlie's
copy_sequencesmethod (from his branch) forConvertJoband builds on it with a newcopy_controls_between_projectsmethod that uses it to copy blanks fastqs from their "primary" project into any secondary projects. It also incorporates the new blanks identification into the sif creation inPipelineand adds empo_4 to the blanks metadata ;)Note that it removes the use of the
addl_infoinput ingenerate_sample_info_filessince it is hard to refactor and, after discussion with Charlie, I don't think this is necessary. It is used whenqp-klpcalls this method: it sends in all the qiita sample names for the project, from which this method was adding any blanks into the sif file it creates--butqp-klplater filters back OUT any blanks in the sif that are also already in qiita, so the end result is as if nothing had ever been passed. Once there is an opportunity to changeqp-klpso that it doesn't pass the superfluous info in, I will take theaddl_infoparameter off this method altogether.