diff --git a/README.rst b/README.rst index 94021af..dcd40dc 100644 --- a/README.rst +++ b/README.rst @@ -26,16 +26,41 @@ :Python version: Python 3.8, 3.9, 3.10, 3.11 :Citation: Cokelaer et al, (2017), ‘Sequana’: a Set of Snakemake NGS pipelines, Journal of Open Source Software, 2(16), 352, `JOSS DOI doi:10.21105/joss.00352 `_ +Installation +============ + +from pypi website:: + + pip install sequana_pipetools + +No dependencies for this package except Python itself. In practice, this package +has no interest if not used within a Sequana pipeline. It is installed automatically when you install +a Sequana pipelines. For example:: + + pip install sequana_rnaseq + pip install sequana_fastqc + +See `Sequana `_ for a list of pipelines ready for production. + + +Targetted audience +================== + +This package is intended for `Sequana `_ developers seeking to integrate Snakemake pipelines into the Sequana project. Please refer below for more information. Additionally, note that as a developer, you can generate the reference documentation using Sphinx:: + + make html + browse build/html/index.html + What is sequana_pipetools ? ============================ -**sequana_pipetools** is a collection of tools that assists with the management of `Sequana `_ pipelines, which includes next-generation sequencing (NGS) pipelines like RNA-seq, variant calling, ChIP-seq, and others. +**sequana_pipetools** is a collection of tools designed to facilitate the management of `Sequana `_ pipelines, which includes next-generation sequencing (NGS) pipelines like RNA-seq, variant calling, ChIP-seq, and others. -The aim of this package is to simplify the deployment of `Sequana pipelines `_ by -creating a pure Python library that includes commonly used tools for different pipelines. +The aim of this package is to streamline the deployment of `Sequana pipelines `_ by +creating a pure Python library that includes commonly used tools for various pipelines. -Previously, the Sequana framework incorporated alll bioinformatics, Snakemake rules, +Previously, the Sequana framework incorporated all bioinformatics, Snakemake rules, pipelines, and pipeline management tools into a single library (Sequana) as illustrated in **Fig 1** below. @@ -45,29 +70,30 @@ in **Fig 1** below. **Figure 1** Old Sequana framework will all pipelines and Sequana library in the same place including pipetools (this library). -Whenever changes were made to the Sequana library, a thorough check of the entire library was necessary, despite -having 80% test coverage. Adding new pipelines also necessitated the addition of new dependencies, and the process was becoming increasingly complex. To mitigate this issue, we initially made all pipelines independent, as illustrated in **Fig. 2**. This way, pipeline changes could be made without updating Sequana and vice versa, which was a significant improvment. +Despite maintaining an 80% test coverage, whenever changes were introduced to the Sequana library, a comprehensive examination of the entire library was imperative. The complexity escalated further when incorporating new pipelines or dependencies. To address this challenge, we initially designed all pipelines to operate independently, as depicted in **Fig. 2**. This approach allowed modifications to pipelines without necessitating updates to Sequana and vice versa, resulting in a significant improvement. + .. figure:: https://raw.githubusercontent.com/sequana/sequana_pipetools/main/doc/old.png :scale: 40% - **Figure 2** v0.8 of Sequana moved the Snakemake pipelines in indepdendent + **Figure 2** v0.8 of Sequana moved the Snakemake pipelines in independent repositories. A `cookie cutter `_ - ease the creation of scuh pipelines + ease the creation of such pipelines + + +Nevertheless, certain tools, including those utilized for user interface and input data sanity checks, were essential for all pipelines, as illustrated by the pipetools box in the figure. With the continuous addition of new pipelines each month, our goal was to enhance the modularity of both the pipelines and Sequana. As a result, we developed a pure Python library named sequana_pipetools, depicted in **Fig. 3**, to further empower the autonomy of the pipelines. -However, certain tools, such as those used for user interface and input data sanity checks, were required by all pipelines, as depicted by the pipetools box in the figure. As new pipelines were being added every month, we aimed to make the pipelines and Sequana more modular. Consequently, we created a pure Python library known as **sequana_pipetools**, as shown in **Fig. 3**, to make the pipelines even more autonomous. .. figure:: https://raw.githubusercontent.com/sequana/sequana_pipetools/main/doc/new.png :scale: 40% - **Figure 3** New Sequana framework. The library contains the core and - bioinformatics tools and is now distinct from the pipelines. Additionally, the - sequana_pipetools library supplies common tools to assist in the creaton and management of all pipelines, - such as shared parser for options. + **Figure 3** New Sequana framework. The new Sequana framework comprises the core library + and bioinformatics tools, which are now separate from the pipelines. Moreover, the + sequana_pipetools library provides essential tools for the creation and management + of all pipelines, including a shared parser for options -Finally, we dropped the rules/ available in Sequana to build an independent package with a set of Snakemake -wrappers. These wrappers are available on https://github.com/sequana/sequana-wrappers and have also the advantage of being tested through continuous integration. +As a final step, we separated the rules originally available in Sequana to create an independent package featuring a collection of Snakemake wrappers. These wrappers can be accessed at https://github.com/sequana/sequana-wrappers and offer the added benefit of being rigorously tested through continuous integration. .. figure:: https://raw.githubusercontent.com/sequana/sequana_pipetools/main/doc/wrappers.png :scale: 40% @@ -76,48 +102,60 @@ wrappers. These wrappers are available on https://github.com/sequana/sequana-wra bioinformatics tools is now fully independent of the pipelines. -Installation -============ -from pypi website:: +Quick tour of the standalone +============================ - pip install sequana_pipetools +The **sequana_pipetools** package provide a standalone called **sequana_pipetools**. Here is a snapshot of the user interface: -No dependencies for this package except Python itself. In practice, this package -has no interest if not used with a Sequana pipeline. So, when using it, -you will need to install the relevant Sequana pipelines that you wish to use. For example:: - - pip install sequana_rnaseq - pip install sequana_fastqc - ... +.. figure:: https://raw.githubusercontent.com/sequana/sequana_pipetools/main/doc/UI.png +There are currently 3-4 main application. The first one is for Linux users under +bash to obtain completion of a sequana pipeline command line arguments:: -This package is for `Sequana `_ developers. -To get more help, go to the doc directory and build the local sphinx directory using:: + sequana_pipetools --completion fastqc - make html - browse build/html/index.html +The second is used to introspect slurm files to get a summary of the SLURM log +files:: -Quick tour -========== + sequana_pipetools --slurm-diag -There are currently two standalone tools. The first one is for Linux users under -bash to obtain completion of a sequana pipeline command line arguments:: +It searches for files with pattern **slurm** in the current directory and slurm files in the ./logs directory. +This is used within th pipeline but can be used manually as well and is useful to get a quick summary of common errors found in slurm files. - sequana_completion --name fastqc +The following command provides statistics about Sequana pipelines installed on your system (number of rules, wrappers +used):: -The second is used to introspect slurm files to get a summary of the SLURM log -files:: + sequana_pipetools -- stats - sequana_slurm_status --directory . +And for developpers, a quick creation of schema file given a config file (experimental, developers would still need to edit the schema but it does 90% of the job):: -Will print a short summary report with common errors (if any). + sequana_pipetools --config-to-schema config.yaml > schema.yaml +For Sequana developers +====================== The library is intended to help Sequana developers to design their pipelines. See the `Sequana organization repository for examples `_. +In addition to the standalone shown above, **sequana_pipetools** main goal is to provide utilities to help Sequana developers. + +First, let us create a pipeline + +Initiate a project (Sequana pipeline) with cookiecutter +------------------------------------------------------- + +You can start a Sequana pipeline skeleton as follows:: + + pip install cookiecuter + cookiecutter https://github.com/sequana/sequana_pipeline_template -o . --overwrite-if-exists + +and then follow the instructions. You will be asked some questions such as the name of your pipeline (eg. variant), a description, keywords and the *project_slug* (just press enter). + +Update the main script +----------------------- + +Go to sequana_pipelines/NAME and look at the main.py script. -In addition to those standalones, sequana_pipetools goal is to provide utilities to help Sequana developers. We currently provide a set of Options classes that should be used to design the API of your pipelines. For example, the sequana_pipetools.options.SlurmOptions can be used as follows inside a standard @@ -185,18 +223,8 @@ https://github.com/sequana/sequana_variant_calling) to get help from examples. The Options classes provided can be used and combined to design pipelines. -How to create skeleton of a Sequana pipeline with cookiecutter -================================================================= - -You can start a Sequana pipeline skeleton as follows:: - - pip install cookiecuter - cookiecutter https://github.com/sequana/sequana_pipeline_template -o . --overwrite-if-exists - -and then follow the instructions. You will be asked some questions such as the name of your pipeline (eg. variant), a description, keywords and the *project_slug* (just press enter). - How to use sequana pipetools within your Pipeline -################################################## +-------------------------------------------------- For FastQ files (paired ot not), The config file should look like:: @@ -234,11 +262,8 @@ And your pipeline could make use of this as follows:: manager.teardown() - - - Setting up and Running Sequana pipelines -========================================= +----------------------------------------- When you execute a sequana pipeline, e.g.:: @@ -272,8 +297,8 @@ What is Sequana ? #. A Python library dedicated to NGS analysis (e.g., tools to visualise standard NGS formats). #. A set of Pipelines dedicated to NGS in the form of Snakefiles (Makefile-like with Python syntax based on snakemake framework) with more - than 80 re-usable rules. -#. Standalone applications. + common wrappers. +#. Standalone applications such as sequana_coverage and sequana_taxonomy. See the `sequana home page `_ for details. @@ -288,6 +313,7 @@ Changelog ========= ====================================================================== Version Description ========= ====================================================================== +0.16.9 0.16.8 * stats command add the number of rules per pipeline * better slurm parsing using profile tree directory (slurm in logs/) 0.16.7 * add missing --trimming-quality option in list of TrimmingOption diff --git a/doc/UI.png b/doc/UI.png new file mode 100644 index 0000000..85dbee5 Binary files /dev/null and b/doc/UI.png differ diff --git a/sequana_pipetools/misc.py b/sequana_pipetools/misc.py index 809ffeb..da3c4a1 100644 --- a/sequana_pipetools/misc.py +++ b/sequana_pipetools/misc.py @@ -90,7 +90,7 @@ def print_version(name): try: ver = get_package_version(f"sequana_{name}") print(f"sequana_{name} version: {ver}") - except Exception as err: + except Exception as err: # pragma: no cover print(err) print(f"sequana_{name} version: ?") diff --git a/sequana_pipetools/snaketools/slurm.py b/sequana_pipetools/snaketools/slurm.py index 8125287..7885ac2 100644 --- a/sequana_pipetools/snaketools/slurm.py +++ b/sequana_pipetools/snaketools/slurm.py @@ -48,35 +48,43 @@ class SlurmParsing: registry = { "oom_kill event in": "Out of memory. Consider increasing memory for the rule", "command not found": "Command not found. Check the missing tool is installed or use --use-apptainer", - "1 of 1 steps (100%) done": "Finished", + # "1 of 1 steps (100%) done": "Finished", } def __init__(self, working_directory, logs_directory="logs", pattern="*/*slurm*.out"): # get the master slurm file main_slurms = list(Path(working_directory).glob("slurm-*")) - self.master = sorted(main_slurms)[-1] + + try: + self.master = sorted(main_slurms)[-1] + print(f"Found slurm master {self.master}") + except Exception as err: + self.master = None log_dir = Path(working_directory) / logs_directory self.slurms = sorted([f for f in log_dir.glob(pattern)]) + # no sys exit (even zero) since it is used within snakemake N = len(self.slurms) - if N > 0: - print(f"Found {N} slurm files to introspect. Please wait.") - else: # pragma: no cover + self.errors = [] + self.percent = "undefined " + + if N == 0: # pragma: no cover logger.warning(f"No {pattern} slurm files were found") - sys.exit(0) + else: # pragma: no cover + print(f"Found {N} slurm files to introspect in {logs_directory}. Processing.") - # main percentage of error from master slurm - self.percent = self._get_percent() + # main percentage of error from master slurm + if self.master: + self.percent = self._get_percent() - # get rules with errors - errors = self._get_rules_with_errors() - self.errors = [] + # whether or not we have a master file, we can scan the logs + errors = self._get_rules_with_errors() - if len(errors): - for error in errors: - self.errors.append({"rule": error["rule"], "slurm_id": error, "hint": error}) + if len(errors): + for error in errors: + self.errors.append({"rule": error["rule"], "slurm_id": error}) def __repr__(self): return self._report() @@ -115,14 +123,29 @@ def _get_rules_with_errors(self): errors = """Error executing rule {rule:S} on cluster (jobid: {jobid:d}, external: Submitted batch job {slurm_id:d}, jobscript: {jobscript}). For error details see the cluster log and the log files of the involved rule(s).""" - with open(self.master, "r") as f: - data = f.read() - return list(parse.findall(errors, data)) + if self.master: + with open(self.master, "r") as f: + data = f.read() + return list(parse.findall(errors, data)) + else: # we need to introspect all slurm files + errors = [] + for filename in self.slurms: + with open(filename, "r") as fin: + data = fin.read() + ID = filename.name.strip(".out").split("-")[-1] + rule = filename.name.split("-")[0] + for k in self.registry.keys(): + print(k) + if k in data: + errors.append({"rule": rule, "slurm_id": ID}) + break + return errors def _get_error(self, filename): + """Find known errors with a file""" with open(filename, "r") as f: data = f.read() for k in self.registry.keys(): if k in data: return self.registry[k] - return "\n No error found" + return "\n No registered error found" # pragma: no cover diff --git a/tests/snaketools/data/slurm_error1/logs/minimap2_and_genomecov/minimap2_and_genomecov-sample=H149_P1_200723-slurm-54467146.out b/tests/snaketools/data/slurm_error1/logs/minimap2_and_genomecov/minimap2_and_genomecov-sample=H149_P1_200723-slurm-54467146.out index 0c7c5bf..d998249 100755 --- a/tests/snaketools/data/slurm_error1/logs/minimap2_and_genomecov/minimap2_and_genomecov-sample=H149_P1_200723-slurm-54467146.out +++ b/tests/snaketools/data/slurm_error1/logs/minimap2_and_genomecov/minimap2_and_genomecov-sample=H149_P1_200723-slurm-54467146.out @@ -8,7 +8,7 @@ Select jobs to execute... [Sat Jan 6 02:46:49 2024] rule minimap2_and_genomecov: - input: /pasteur/zeus/projets/p02/BiomicsData/2023/DryLab/B17420/20230720_1448_MN18671_FAV94301_57a76bad/nanomerge/B17420/H149_P1_200723.fastq.gz, H149_P1_200723/sorted_contigs/H149_P1_200723.fasta + input: /Data/2023/DryLab/B17420/20230720_1448_MN18671_FAV94301_57a76bad/nanomerge/B17420/H149_P1_200723.fastq.gz, H149_P1_200723/sorted_contigs/H149_P1_200723.fasta output: H149_P1_200723/minimap2/H149_P1_200723.bam, H149_P1_200723/minimap2/H149_P1_200723.bed jobid: 0 reason: Forced execution @@ -17,25 +17,25 @@ rule minimap2_and_genomecov: resources: mem_mb=4000, mem_mib=3815, disk_mb=6933, disk_mib=6612, tmpdir=/local/scratch/tmp, partition=common, qos=normal, mem=4G, gres= - minimap2 -t 4 -ax map-ont H149_P1_200723/sorted_contigs/H149_P1_200723.fasta /pasteur/zeus/projets/p02/BiomicsData/2023/DryLab/B17420/20230720_1448_MN18671_FAV94301_57a76bad/nanomerge/B17420/H149_P1_200723.fastq.gz | samtools sort -@ $((4 - 1)) -o H149_P1_200723/minimap2/H149_P1_200723.bam && samtools index H149_P1_200723/minimap2/H149_P1_200723.bam && samtools depth -aa H149_P1_200723/minimap2/H149_P1_200723.bam > H149_P1_200723/minimap2/H149_P1_200723.bed + minimap2 -t 4 -ax map-ont H149_P1_200723/sorted_contigs/H149_P1_200723.fasta /Data/2023/DryLab/B17420/20230720_1448_MN18671_FAV94301_57a76bad/nanomerge/B17420/H149_P1_200723.fastq.gz | samtools sort -@ $((4 - 1)) -o H149_P1_200723/minimap2/H149_P1_200723.bam && samtools index H149_P1_200723/minimap2/H149_P1_200723.bam && samtools depth -aa H149_P1_200723/minimap2/H149_P1_200723.bam > H149_P1_200723/minimap2/H149_P1_200723.bed -Activating singularity image /pasteur/zeus/projets/p02/Biomics/resources/condas/sequana_0.16/apptainers/c3e4a8244ce7b65fa873ebda134fea7f.simg +Activating singularity image //resources/condas/sequana_0.16/apptainers/c3e4a8244ce7b65fa873ebda134fea7f.simg [M::mm_idx_gen::1.104*0.97] collected minimizers [M::mm_idx_gen::1.497*1.44] sorted minimizers [M::main::1.497*1.44] loaded/built the index for 103 target sequence(s) [M::mm_mapopt_update::1.571*1.42] mid_occ = 36 [M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 103 [M::mm_idx_stat::1.623*1.41] distinct minimizers: 5111506 (89.06% are singletons); average occurrences: 1.235; average spacing: 5.336; total length: 33694841 -/bin/bash: line 2: 3479709 Killed minimap2 -t 4 -ax map-ont H149_P1_200723/sorted_contigs/H149_P1_200723.fasta /pasteur/zeus/projets/p02/BiomicsData/2023/DryLab/B17420/20230720_1448_MN18671_FAV94301_57a76bad/nanomerge/B17420/H149_P1_200723.fastq.gz +/bin/bash: line 2: 3479709 Killed minimap2 -t 4 -ax map-ont H149_P1_200723/sorted_contigs/H149_P1_200723.fasta /Data/2023/DryLab/B17420/20230720_1448_MN18671_FAV94301_57a76bad/nanomerge/B17420/H149_P1_200723.fastq.gz 3479710 Done | samtools sort -@ $((4 - 1)) -o H149_P1_200723/minimap2/H149_P1_200723.bam [Sat Jan 6 02:47:31 2024] Error in rule minimap2_and_genomecov: jobid: 0 - input: /pasteur/zeus/projets/p02/BiomicsData/2023/DryLab/B17420/20230720_1448_MN18671_FAV94301_57a76bad/nanomerge/B17420/H149_P1_200723.fastq.gz, H149_P1_200723/sorted_contigs/H149_P1_200723.fasta + input: /Data/2023/DryLab/B17420/20230720_1448_MN18671_FAV94301_57a76bad/nanomerge/B17420/H149_P1_200723.fastq.gz, H149_P1_200723/sorted_contigs/H149_P1_200723.fasta output: H149_P1_200723/minimap2/H149_P1_200723.bam, H149_P1_200723/minimap2/H149_P1_200723.bed shell: - minimap2 -t 4 -ax map-ont H149_P1_200723/sorted_contigs/H149_P1_200723.fasta /pasteur/zeus/projets/p02/BiomicsData/2023/DryLab/B17420/20230720_1448_MN18671_FAV94301_57a76bad/nanomerge/B17420/H149_P1_200723.fastq.gz | samtools sort -@ $((4 - 1)) -o H149_P1_200723/minimap2/H149_P1_200723.bam && samtools index H149_P1_200723/minimap2/H149_P1_200723.bam && samtools depth -aa H149_P1_200723/minimap2/H149_P1_200723.bam > H149_P1_200723/minimap2/H149_P1_200723.bed + minimap2 -t 4 -ax map-ont H149_P1_200723/sorted_contigs/H149_P1_200723.fasta /Data/2023/DryLab/B17420/20230720_1448_MN18671_FAV94301_57a76bad/nanomerge/B17420/H149_P1_200723.fastq.gz | samtools sort -@ $((4 - 1)) -o H149_P1_200723/minimap2/H149_P1_200723.bam && samtools index H149_P1_200723/minimap2/H149_P1_200723.bam && samtools depth -aa H149_P1_200723/minimap2/H149_P1_200723.bam > H149_P1_200723/minimap2/H149_P1_200723.bed (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!) diff --git a/tests/snaketools/data/slurm_error1/logs/minimap2_and_genomecov/minimap2_and_genomecov-sample=H149_P1_200723-slurm-54474584.out b/tests/snaketools/data/slurm_error1/logs/minimap2_and_genomecov/minimap2_and_genomecov-sample=H149_P1_200723-slurm-54474584.out index db2de4d..0f2f58c 100755 --- a/tests/snaketools/data/slurm_error1/logs/minimap2_and_genomecov/minimap2_and_genomecov-sample=H149_P1_200723-slurm-54474584.out +++ b/tests/snaketools/data/slurm_error1/logs/minimap2_and_genomecov/minimap2_and_genomecov-sample=H149_P1_200723-slurm-54474584.out @@ -8,7 +8,7 @@ Select jobs to execute... [Sat Jan 6 08:12:45 2024] rule minimap2_and_genomecov: - input: /pasteur/zeus/projets/p02/BiomicsData/2023/DryLab/B17420/20230720_1448_MN18671_FAV94301_57a76bad/nanomerge/B17420/H149_P1_200723.fastq.gz, H149_P1_200723/sorted_contigs/H149_P1_200723.fasta + input: /Data/2023/DryLab/B17420/20230720_1448_MN18671_FAV94301_57a76bad/nanomerge/B17420/H149_P1_200723.fastq.gz, H149_P1_200723/sorted_contigs/H149_P1_200723.fasta output: H149_P1_200723/minimap2/H149_P1_200723.bam, H149_P1_200723/minimap2/H149_P1_200723.bed jobid: 0 reason: Forced execution @@ -17,9 +17,9 @@ rule minimap2_and_genomecov: resources: mem_mb=16000, mem_mib=6612, disk_mb=6933, disk_mib=6612, tmpdir=/local/scratch/tmp, partition=common, qos=normal, mem=16G, gres= - minimap2 -t 4 -ax map-ont H149_P1_200723/sorted_contigs/H149_P1_200723.fasta /pasteur/zeus/projets/p02/BiomicsData/2023/DryLab/B17420/20230720_1448_MN18671_FAV94301_57a76bad/nanomerge/B17420/H149_P1_200723.fastq.gz | samtools sort -@ $((4 - 1)) -o H149_P1_200723/minimap2/H149_P1_200723.bam && samtools index H149_P1_200723/minimap2/H149_P1_200723.bam && samtools depth -aa H149_P1_200723/minimap2/H149_P1_200723.bam > H149_P1_200723/minimap2/H149_P1_200723.bed + minimap2 -t 4 -ax map-ont H149_P1_200723/sorted_contigs/H149_P1_200723.fasta /Data/2023/DryLab/B17420/20230720_1448_MN18671_FAV94301_57a76bad/nanomerge/B17420/H149_P1_200723.fastq.gz | samtools sort -@ $((4 - 1)) -o H149_P1_200723/minimap2/H149_P1_200723.bam && samtools index H149_P1_200723/minimap2/H149_P1_200723.bam && samtools depth -aa H149_P1_200723/minimap2/H149_P1_200723.bam > H149_P1_200723/minimap2/H149_P1_200723.bed -Activating singularity image /pasteur/zeus/projets/p02/Biomics/resources/condas/sequana_0.16/apptainers/c3e4a8244ce7b65fa873ebda134fea7f.simg +Activating singularity image //resources/condas/sequana_0.16/apptainers/c3e4a8244ce7b65fa873ebda134fea7f.simg [M::mm_idx_gen::1.466*0.80] collected minimizers [M::mm_idx_gen::1.713*1.17] sorted minimizers [M::main::1.713*1.17] loaded/built the index for 103 target sequence(s) @@ -35,7 +35,7 @@ Activating singularity image /pasteur/zeus/projets/p02/Biomics/resources/condas/ [M::worker_pipeline::759.447*3.76] mapped 66000 sequences [M::worker_pipeline::830.013*3.77] mapped 54006 sequences [M::main] Version: 2.24-r1122 -[M::main] CMD: minimap2 -t 4 -ax map-ont H149_P1_200723/sorted_contigs/H149_P1_200723.fasta /pasteur/zeus/projets/p02/BiomicsData/2023/DryLab/B17420/20230720_1448_MN18671_FAV94301_57a76bad/nanomerge/B17420/H149_P1_200723.fastq.gz +[M::main] CMD: minimap2 -t 4 -ax map-ont H149_P1_200723/sorted_contigs/H149_P1_200723.fasta /Data/2023/DryLab/B17420/20230720_1448_MN18671_FAV94301_57a76bad/nanomerge/B17420/H149_P1_200723.fastq.gz [M::main] Real time: 830.103 sec; CPU: 3128.051 sec; Peak RSS: 4.470 GB [bam_sort_core] merging from 2 files and 3 in-memory blocks... [Sat Jan 6 08:30:29 2024] diff --git a/tests/snaketools/data/slurm_error1/logs/minimap2_and_genomecov/minimap2_and_genomecov-sample=H149_P1_200723_1000-slurm-54467581.out b/tests/snaketools/data/slurm_error1/logs/minimap2_and_genomecov/minimap2_and_genomecov-sample=H149_P1_200723_1000-slurm-54467581.out index ae91a5f..dce6210 100755 --- a/tests/snaketools/data/slurm_error1/logs/minimap2_and_genomecov/minimap2_and_genomecov-sample=H149_P1_200723_1000-slurm-54467581.out +++ b/tests/snaketools/data/slurm_error1/logs/minimap2_and_genomecov/minimap2_and_genomecov-sample=H149_P1_200723_1000-slurm-54467581.out @@ -8,7 +8,7 @@ Select jobs to execute... [Sat Jan 6 02:48:15 2024] rule minimap2_and_genomecov: - input: /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/data/filters/H149_P1_200723_1000.fastq.gz, H149_P1_200723_1000/sorted_contigs/H149_P1_200723_1000.fasta + input: //Drylab/InProgress/B17420/data/filters/H149_P1_200723_1000.fastq.gz, H149_P1_200723_1000/sorted_contigs/H149_P1_200723_1000.fasta output: H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bam, H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bed jobid: 0 reason: Forced execution @@ -17,9 +17,9 @@ rule minimap2_and_genomecov: resources: mem_mb=4000, mem_mib=3815, disk_mb=6823, disk_mib=6507, tmpdir=/local/scratch/tmp, partition=common, qos=normal, mem=4G, gres= - minimap2 -t 4 -ax map-ont H149_P1_200723_1000/sorted_contigs/H149_P1_200723_1000.fasta /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/data/filters/H149_P1_200723_1000.fastq.gz | samtools sort -@ $((4 - 1)) -o H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bam && samtools index H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bam && samtools depth -aa H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bam > H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bed + minimap2 -t 4 -ax map-ont H149_P1_200723_1000/sorted_contigs/H149_P1_200723_1000.fasta //Drylab/InProgress/B17420/data/filters/H149_P1_200723_1000.fastq.gz | samtools sort -@ $((4 - 1)) -o H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bam && samtools index H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bam && samtools depth -aa H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bam > H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bed -Activating singularity image /pasteur/zeus/projets/p02/Biomics/resources/condas/sequana_0.16/apptainers/c3e4a8244ce7b65fa873ebda134fea7f.simg +Activating singularity image //resources/condas/sequana_0.16/apptainers/c3e4a8244ce7b65fa873ebda134fea7f.simg [M::mm_idx_gen::1.158*0.99] collected minimizers [M::mm_idx_gen::1.469*1.48] sorted minimizers [M::main::1.469*1.48] loaded/built the index for 98 target sequence(s) @@ -28,16 +28,16 @@ Activating singularity image /pasteur/zeus/projets/p02/Biomics/resources/condas/ [M::mm_idx_stat::1.590*1.45] distinct minimizers: 5111798 (89.08% are singletons); average occurrences: 1.236; average spacing: 5.337; total length: 33703652 [M::worker_pipeline::123.711*3.74] mapped 54811 sequences [bam_sort_core] merging from 0 files and 3 in-memory blocks... -/bin/bash: line 2: 896648 Killed minimap2 -t 4 -ax map-ont H149_P1_200723_1000/sorted_contigs/H149_P1_200723_1000.fasta /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/data/filters/H149_P1_200723_1000.fastq.gz +/bin/bash: line 2: 896648 Killed minimap2 -t 4 -ax map-ont H149_P1_200723_1000/sorted_contigs/H149_P1_200723_1000.fasta //Drylab/InProgress/B17420/data/filters/H149_P1_200723_1000.fastq.gz 896649 Done | samtools sort -@ $((4 - 1)) -o H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bam [Sat Jan 6 02:50:51 2024] Error in rule minimap2_and_genomecov: jobid: 0 - input: /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/data/filters/H149_P1_200723_1000.fastq.gz, H149_P1_200723_1000/sorted_contigs/H149_P1_200723_1000.fasta + input: //Drylab/InProgress/B17420/data/filters/H149_P1_200723_1000.fastq.gz, H149_P1_200723_1000/sorted_contigs/H149_P1_200723_1000.fasta output: H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bam, H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bed shell: - minimap2 -t 4 -ax map-ont H149_P1_200723_1000/sorted_contigs/H149_P1_200723_1000.fasta /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/data/filters/H149_P1_200723_1000.fastq.gz | samtools sort -@ $((4 - 1)) -o H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bam && samtools index H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bam && samtools depth -aa H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bam > H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bed + minimap2 -t 4 -ax map-ont H149_P1_200723_1000/sorted_contigs/H149_P1_200723_1000.fasta //Drylab/InProgress/B17420/data/filters/H149_P1_200723_1000.fastq.gz | samtools sort -@ $((4 - 1)) -o H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bam && samtools index H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bam && samtools depth -aa H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bam > H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bed (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!) diff --git a/tests/snaketools/data/slurm_error1/logs/minimap2_and_genomecov/minimap2_and_genomecov-sample=H149_P1_200723_1000-slurm-54474583.out b/tests/snaketools/data/slurm_error1/logs/minimap2_and_genomecov/minimap2_and_genomecov-sample=H149_P1_200723_1000-slurm-54474583.out index 03393bb..2579b27 100755 --- a/tests/snaketools/data/slurm_error1/logs/minimap2_and_genomecov/minimap2_and_genomecov-sample=H149_P1_200723_1000-slurm-54474583.out +++ b/tests/snaketools/data/slurm_error1/logs/minimap2_and_genomecov/minimap2_and_genomecov-sample=H149_P1_200723_1000-slurm-54474583.out @@ -8,7 +8,7 @@ Select jobs to execute... [Sat Jan 6 08:12:32 2024] rule minimap2_and_genomecov: - input: /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/data/filters/H149_P1_200723_1000.fastq.gz, H149_P1_200723_1000/sorted_contigs/H149_P1_200723_1000.fasta + input: //Drylab/InProgress/B17420/data/filters/H149_P1_200723_1000.fastq.gz, H149_P1_200723_1000/sorted_contigs/H149_P1_200723_1000.fasta output: H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bam, H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bed jobid: 0 reason: Forced execution @@ -17,9 +17,9 @@ rule minimap2_and_genomecov: resources: mem_mb=16000, mem_mib=6507, disk_mb=6823, disk_mib=6507, tmpdir=/local/scratch/tmp, partition=common, qos=normal, mem=16G, gres= - minimap2 -t 4 -ax map-ont H149_P1_200723_1000/sorted_contigs/H149_P1_200723_1000.fasta /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/data/filters/H149_P1_200723_1000.fastq.gz | samtools sort -@ $((4 - 1)) -o H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bam && samtools index H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bam && samtools depth -aa H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bam > H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bed + minimap2 -t 4 -ax map-ont H149_P1_200723_1000/sorted_contigs/H149_P1_200723_1000.fasta //Drylab/InProgress/B17420/data/filters/H149_P1_200723_1000.fastq.gz | samtools sort -@ $((4 - 1)) -o H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bam && samtools index H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bam && samtools depth -aa H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bam > H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bed -Activating singularity image /pasteur/zeus/projets/p02/Biomics/resources/condas/sequana_0.16/apptainers/c3e4a8244ce7b65fa873ebda134fea7f.simg +Activating singularity image //resources/condas/sequana_0.16/apptainers/c3e4a8244ce7b65fa873ebda134fea7f.simg [M::mm_idx_gen::1.537*0.72] collected minimizers [M::mm_idx_gen::1.792*1.09] sorted minimizers [M::main::1.792*1.09] loaded/built the index for 98 target sequence(s) @@ -35,7 +35,7 @@ Activating singularity image /pasteur/zeus/projets/p02/Biomics/resources/condas/ [M::worker_pipeline::766.288*3.77] mapped 54700 sequences [M::worker_pipeline::826.454*3.78] mapped 33228 sequences [M::main] Version: 2.24-r1122 -[M::main] CMD: minimap2 -t 4 -ax map-ont H149_P1_200723_1000/sorted_contigs/H149_P1_200723_1000.fasta /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/data/filters/H149_P1_200723_1000.fastq.gz +[M::main] CMD: minimap2 -t 4 -ax map-ont H149_P1_200723_1000/sorted_contigs/H149_P1_200723_1000.fasta //Drylab/InProgress/B17420/data/filters/H149_P1_200723_1000.fastq.gz [M::main] Real time: 826.537 sec; CPU: 3122.557 sec; Peak RSS: 4.578 GB [bam_sort_core] merging from 2 files and 3 in-memory blocks... [Sat Jan 6 08:30:12 2024] diff --git a/tests/snaketools/data/slurm_error1/logs/minimap2_and_genomecov/minimap2_and_genomecov-sample=H149_P1_200723_500-slurm-54466792.out b/tests/snaketools/data/slurm_error1/logs/minimap2_and_genomecov/minimap2_and_genomecov-sample=H149_P1_200723_500-slurm-54466792.out index b1c1a30..0c71a69 100755 --- a/tests/snaketools/data/slurm_error1/logs/minimap2_and_genomecov/minimap2_and_genomecov-sample=H149_P1_200723_500-slurm-54466792.out +++ b/tests/snaketools/data/slurm_error1/logs/minimap2_and_genomecov/minimap2_and_genomecov-sample=H149_P1_200723_500-slurm-54466792.out @@ -8,7 +8,7 @@ Select jobs to execute... [Sat Jan 6 02:42:27 2024] rule minimap2_and_genomecov: - input: /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/data/filters/H149_P1_200723_500.fastq.gz, H149_P1_200723_500/sorted_contigs/H149_P1_200723_500.fasta + input: //Drylab/InProgress/B17420/data/filters/H149_P1_200723_500.fastq.gz, H149_P1_200723_500/sorted_contigs/H149_P1_200723_500.fasta output: H149_P1_200723_500/minimap2/H149_P1_200723_500.bam, H149_P1_200723_500/minimap2/H149_P1_200723_500.bed jobid: 0 reason: Forced execution @@ -17,25 +17,25 @@ rule minimap2_and_genomecov: resources: mem_mb=4000, mem_mib=3815, disk_mb=6903, disk_mib=6584, tmpdir=/local/scratch/tmp, partition=common, qos=normal, mem=4G, gres= - minimap2 -t 4 -ax map-ont H149_P1_200723_500/sorted_contigs/H149_P1_200723_500.fasta /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/data/filters/H149_P1_200723_500.fastq.gz | samtools sort -@ $((4 - 1)) -o H149_P1_200723_500/minimap2/H149_P1_200723_500.bam && samtools index H149_P1_200723_500/minimap2/H149_P1_200723_500.bam && samtools depth -aa H149_P1_200723_500/minimap2/H149_P1_200723_500.bam > H149_P1_200723_500/minimap2/H149_P1_200723_500.bed + minimap2 -t 4 -ax map-ont H149_P1_200723_500/sorted_contigs/H149_P1_200723_500.fasta //Drylab/InProgress/B17420/data/filters/H149_P1_200723_500.fastq.gz | samtools sort -@ $((4 - 1)) -o H149_P1_200723_500/minimap2/H149_P1_200723_500.bam && samtools index H149_P1_200723_500/minimap2/H149_P1_200723_500.bam && samtools depth -aa H149_P1_200723_500/minimap2/H149_P1_200723_500.bam > H149_P1_200723_500/minimap2/H149_P1_200723_500.bed -Activating singularity image /pasteur/zeus/projets/p02/Biomics/resources/condas/sequana_0.16/apptainers/c3e4a8244ce7b65fa873ebda134fea7f.simg +Activating singularity image //resources/condas/sequana_0.16/apptainers/c3e4a8244ce7b65fa873ebda134fea7f.simg [M::mm_idx_gen::1.084*0.99] collected minimizers [M::mm_idx_gen::1.324*1.43] sorted minimizers [M::main::1.324*1.43] loaded/built the index for 101 target sequence(s) [M::mm_mapopt_update::1.397*1.41] mid_occ = 37 [M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 101 [M::mm_idx_stat::1.445*1.39] distinct minimizers: 5111260 (89.08% are singletons); average occurrences: 1.236; average spacing: 5.337; total length: 33723594 -/bin/bash: line 2: 2229689 Killed minimap2 -t 4 -ax map-ont H149_P1_200723_500/sorted_contigs/H149_P1_200723_500.fasta /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/data/filters/H149_P1_200723_500.fastq.gz +/bin/bash: line 2: 2229689 Killed minimap2 -t 4 -ax map-ont H149_P1_200723_500/sorted_contigs/H149_P1_200723_500.fasta //Drylab/InProgress/B17420/data/filters/H149_P1_200723_500.fastq.gz 2229690 Done | samtools sort -@ $((4 - 1)) -o H149_P1_200723_500/minimap2/H149_P1_200723_500.bam [Sat Jan 6 02:43:32 2024] Error in rule minimap2_and_genomecov: jobid: 0 - input: /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/data/filters/H149_P1_200723_500.fastq.gz, H149_P1_200723_500/sorted_contigs/H149_P1_200723_500.fasta + input: //Drylab/InProgress/B17420/data/filters/H149_P1_200723_500.fastq.gz, H149_P1_200723_500/sorted_contigs/H149_P1_200723_500.fasta output: H149_P1_200723_500/minimap2/H149_P1_200723_500.bam, H149_P1_200723_500/minimap2/H149_P1_200723_500.bed shell: - minimap2 -t 4 -ax map-ont H149_P1_200723_500/sorted_contigs/H149_P1_200723_500.fasta /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/data/filters/H149_P1_200723_500.fastq.gz | samtools sort -@ $((4 - 1)) -o H149_P1_200723_500/minimap2/H149_P1_200723_500.bam && samtools index H149_P1_200723_500/minimap2/H149_P1_200723_500.bam && samtools depth -aa H149_P1_200723_500/minimap2/H149_P1_200723_500.bam > H149_P1_200723_500/minimap2/H149_P1_200723_500.bed + minimap2 -t 4 -ax map-ont H149_P1_200723_500/sorted_contigs/H149_P1_200723_500.fasta //Drylab/InProgress/B17420/data/filters/H149_P1_200723_500.fastq.gz | samtools sort -@ $((4 - 1)) -o H149_P1_200723_500/minimap2/H149_P1_200723_500.bam && samtools index H149_P1_200723_500/minimap2/H149_P1_200723_500.bam && samtools depth -aa H149_P1_200723_500/minimap2/H149_P1_200723_500.bam > H149_P1_200723_500/minimap2/H149_P1_200723_500.bed (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!) diff --git a/tests/snaketools/data/slurm_error1/logs/minimap2_and_genomecov/minimap2_and_genomecov-sample=H149_P1_200723_500-slurm-54474582.out b/tests/snaketools/data/slurm_error1/logs/minimap2_and_genomecov/minimap2_and_genomecov-sample=H149_P1_200723_500-slurm-54474582.out index 2b51a4d..4a23726 100755 --- a/tests/snaketools/data/slurm_error1/logs/minimap2_and_genomecov/minimap2_and_genomecov-sample=H149_P1_200723_500-slurm-54474582.out +++ b/tests/snaketools/data/slurm_error1/logs/minimap2_and_genomecov/minimap2_and_genomecov-sample=H149_P1_200723_500-slurm-54474582.out @@ -8,7 +8,7 @@ Select jobs to execute... [Sat Jan 6 08:12:33 2024] rule minimap2_and_genomecov: - input: /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/data/filters/H149_P1_200723_500.fastq.gz, H149_P1_200723_500/sorted_contigs/H149_P1_200723_500.fasta + input: //Drylab/InProgress/B17420/data/filters/H149_P1_200723_500.fastq.gz, H149_P1_200723_500/sorted_contigs/H149_P1_200723_500.fasta output: H149_P1_200723_500/minimap2/H149_P1_200723_500.bam, H149_P1_200723_500/minimap2/H149_P1_200723_500.bed jobid: 0 reason: Forced execution @@ -17,9 +17,9 @@ rule minimap2_and_genomecov: resources: mem_mb=16000, mem_mib=6584, disk_mb=6903, disk_mib=6584, tmpdir=/local/scratch/tmp, partition=common, qos=normal, mem=16G, gres= - minimap2 -t 4 -ax map-ont H149_P1_200723_500/sorted_contigs/H149_P1_200723_500.fasta /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/data/filters/H149_P1_200723_500.fastq.gz | samtools sort -@ $((4 - 1)) -o H149_P1_200723_500/minimap2/H149_P1_200723_500.bam && samtools index H149_P1_200723_500/minimap2/H149_P1_200723_500.bam && samtools depth -aa H149_P1_200723_500/minimap2/H149_P1_200723_500.bam > H149_P1_200723_500/minimap2/H149_P1_200723_500.bed + minimap2 -t 4 -ax map-ont H149_P1_200723_500/sorted_contigs/H149_P1_200723_500.fasta //Drylab/InProgress/B17420/data/filters/H149_P1_200723_500.fastq.gz | samtools sort -@ $((4 - 1)) -o H149_P1_200723_500/minimap2/H149_P1_200723_500.bam && samtools index H149_P1_200723_500/minimap2/H149_P1_200723_500.bam && samtools depth -aa H149_P1_200723_500/minimap2/H149_P1_200723_500.bam > H149_P1_200723_500/minimap2/H149_P1_200723_500.bed -Activating singularity image /pasteur/zeus/projets/p02/Biomics/resources/condas/sequana_0.16/apptainers/c3e4a8244ce7b65fa873ebda134fea7f.simg +Activating singularity image //resources/condas/sequana_0.16/apptainers/c3e4a8244ce7b65fa873ebda134fea7f.simg [M::mm_idx_gen::1.476*0.74] collected minimizers [M::mm_idx_gen::1.721*1.11] sorted minimizers [M::main::1.721*1.11] loaded/built the index for 101 target sequence(s) @@ -35,7 +35,7 @@ Activating singularity image /pasteur/zeus/projets/p02/Biomics/resources/condas/ [M::worker_pipeline::778.489*3.78] mapped 61261 sequences [M::worker_pipeline::848.696*3.79] mapped 44951 sequences [M::main] Version: 2.24-r1122 -[M::main] CMD: minimap2 -t 4 -ax map-ont H149_P1_200723_500/sorted_contigs/H149_P1_200723_500.fasta /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/data/filters/H149_P1_200723_500.fastq.gz +[M::main] CMD: minimap2 -t 4 -ax map-ont H149_P1_200723_500/sorted_contigs/H149_P1_200723_500.fasta //Drylab/InProgress/B17420/data/filters/H149_P1_200723_500.fastq.gz [M::main] Real time: 848.775 sec; CPU: 3215.568 sec; Peak RSS: 4.587 GB [bam_sort_core] merging from 2 files and 3 in-memory blocks... [Sat Jan 6 08:30:41 2024] diff --git a/tests/snaketools/data/slurm_error1/logs/quast/quast-sample=H149_P1_200723-slurm-54467145.out b/tests/snaketools/data/slurm_error1/logs/quast/quast-sample=H149_P1_200723-slurm-54467145.out index 500ad20..483635a 100755 --- a/tests/snaketools/data/slurm_error1/logs/quast/quast-sample=H149_P1_200723-slurm-54467145.out +++ b/tests/snaketools/data/slurm_error1/logs/quast/quast-sample=H149_P1_200723-slurm-54467145.out @@ -8,7 +8,7 @@ Select jobs to execute... [Sat Jan 6 02:46:31 2024] rule quast: - input: /pasteur/zeus/projets/p02/BiomicsData/2023/DryLab/B17420/20230720_1448_MN18671_FAV94301_57a76bad/nanomerge/B17420/H149_P1_200723.fastq.gz, H149_P1_200723/sorted_contigs/H149_P1_200723.fasta + input: /Data/2023/DryLab/B17420/20230720_1448_MN18671_FAV94301_57a76bad/nanomerge/B17420/H149_P1_200723.fastq.gz, H149_P1_200723/sorted_contigs/H149_P1_200723.fasta output: H149_P1_200723/quast/quast.done jobid: 0 reason: Forced execution @@ -17,11 +17,11 @@ rule quast: resources: mem_mb=4000, mem_mib=3815, disk_mb=6933, disk_mib=6612, tmpdir=/local/scratch/tmp, partition=common, qos=normal, mem=4G, gres= - quast.py -e --rna-finding -t 4 H149_P1_200723/sorted_contigs/H149_P1_200723.fasta --nanopore /pasteur/zeus/projets/p02/BiomicsData/2023/DryLab/B17420/20230720_1448_MN18671_FAV94301_57a76bad/nanomerge/B17420/H149_P1_200723.fastq.gz -o H149_P1_200723/quast && touch H149_P1_200723/quast/quast.done + quast.py -e --rna-finding -t 4 H149_P1_200723/sorted_contigs/H149_P1_200723.fasta --nanopore /Data/2023/DryLab/B17420/20230720_1448_MN18671_FAV94301_57a76bad/nanomerge/B17420/H149_P1_200723.fastq.gz -o H149_P1_200723/quast && touch H149_P1_200723/quast/quast.done -Activating singularity image /pasteur/zeus/projets/p02/Biomics/resources/condas/sequana_0.16/apptainers/847f147b6a61a6684f5106b4bc25a0c7.simg +Activating singularity image //resources/condas/sequana_0.16/apptainers/847f147b6a61a6684f5106b4bc25a0c7.simg WARNING: Python locale settings can't be changed -/usr/local/bin/quast.py -e --rna-finding -t 4 H149_P1_200723/sorted_contigs/H149_P1_200723.fasta --nanopore /pasteur/zeus/projets/p02/BiomicsData/2023/DryLab/B17420/20230720_1448_MN18671_FAV94301_57a76bad/nanomerge/B17420/H149_P1_200723.fastq.gz -o H149_P1_200723/quast +/usr/local/bin/quast.py -e --rna-finding -t 4 H149_P1_200723/sorted_contigs/H149_P1_200723.fasta --nanopore /Data/2023/DryLab/B17420/20230720_1448_MN18671_FAV94301_57a76bad/nanomerge/B17420/H149_P1_200723.fastq.gz -o H149_P1_200723/quast Version: 5.2.0 @@ -32,9 +32,9 @@ System information: Started: 2024-01-06 02:46:32 -Logging to /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723/quast/quast.log +Logging to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723/quast/quast.log -CWD: /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate +CWD: //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate Main parameters: MODE: default, threads: 4, eukaryotic: true, min contig length: 500, min alignment length: 65, \ min alignment IDY: 95.0, ambiguity: one, min local misassembly length: 200, min extensive misassembly length: 1000 @@ -45,14 +45,14 @@ Contigs: 2024-01-06 02:46:34 Running Reads analyzer... - Logging to files /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723/quast/reads_stats/reads_stats.log and /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723/quast/reads_stats/reads_stats.err... + Logging to files //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723/quast/reads_stats/reads_stats.log and //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723/quast/reads_stats/reads_stats.err... Pre-processing reads... Running BWA... Done. Sorting SAM-file... Analysis is finished. Creating total report... - saved to /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723/quast/reads_stats/reads_report.txt, reads_report.tsv, and reads_report.tex + saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723/quast/reads_stats/reads_report.txt, reads_report.tsv, and reads_report.tex Done. 2024-01-06 03:29:17 @@ -62,22 +62,22 @@ Running Basic statistics processor... Calculating N50 and L50... H149_P1_200723, N50 = 787511, L50 = 13, auN = 1015747.0, Total length = 33694841, GC % = 59.64, # N's per 100 kbp = 0.00 Drawing Nx plot... - saved to /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723/quast/basic_stats/Nx_plot.pdf + saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723/quast/basic_stats/Nx_plot.pdf Drawing cumulative plot... - saved to /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723/quast/basic_stats/cumulative_plot.pdf + saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723/quast/basic_stats/cumulative_plot.pdf Drawing GC content plot... - saved to /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723/quast/basic_stats/GC_content_plot.pdf + saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723/quast/basic_stats/GC_content_plot.pdf Drawing H149_P1_200723 GC content plot... - saved to /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723/quast/basic_stats/H149_P1_200723_GC_content_plot.pdf + saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723/quast/basic_stats/H149_P1_200723_GC_content_plot.pdf Done. NOTICE: Genes are not predicted by default. Use --gene-finding or --glimmer option to enable it. 2024-01-06 03:29:20 Running Barrnap... -Logging to /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723/quast/predicted_genes/barrnap.log... +Logging to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723/quast/predicted_genes/barrnap.log... Ribosomal RNA genes = 15 - Predicted genes (GFF): /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723/quast/predicted_genes/H149_P1_200723.rna.gff + Predicted genes (GFF): //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723/quast/predicted_genes/H149_P1_200723.rna.gff Done. 2024-01-06 03:29:26 @@ -89,12 +89,12 @@ Done 2024-01-06 03:29:27 RESULTS: - Text versions of total report are saved to /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723/quast/report.txt, report.tsv, and report.tex - Text versions of transposed total report are saved to /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723/quast/transposed_report.txt, transposed_report.tsv, and transposed_report.tex - HTML version (interactive tables and plots) is saved to /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723/quast/report.html - PDF version (tables and plots) is saved to /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723/quast/report.pdf - Icarus (contig browser) is saved to /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723/quast/icarus.html - Log is saved to /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723/quast/quast.log + Text versions of total report are saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723/quast/report.txt, report.tsv, and report.tex + Text versions of transposed total report are saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723/quast/transposed_report.txt, transposed_report.tsv, and transposed_report.tex + HTML version (interactive tables and plots) is saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723/quast/report.html + PDF version (tables and plots) is saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723/quast/report.pdf + Icarus (contig browser) is saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723/quast/icarus.html + Log is saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723/quast/quast.log Finished: 2024-01-06 03:29:27 Elapsed time: 0:42:54.840139 diff --git a/tests/snaketools/data/slurm_error1/logs/quast/quast-sample=H149_P1_200723_1000-slurm-54467578.out b/tests/snaketools/data/slurm_error1/logs/quast/quast-sample=H149_P1_200723_1000-slurm-54467578.out index 318c830..884dc32 100755 --- a/tests/snaketools/data/slurm_error1/logs/quast/quast-sample=H149_P1_200723_1000-slurm-54467578.out +++ b/tests/snaketools/data/slurm_error1/logs/quast/quast-sample=H149_P1_200723_1000-slurm-54467578.out @@ -8,7 +8,7 @@ Select jobs to execute... [Sat Jan 6 02:48:07 2024] rule quast: - input: /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/data/filters/H149_P1_200723_1000.fastq.gz, H149_P1_200723_1000/sorted_contigs/H149_P1_200723_1000.fasta + input: //Drylab/InProgress/B17420/data/filters/H149_P1_200723_1000.fastq.gz, H149_P1_200723_1000/sorted_contigs/H149_P1_200723_1000.fasta output: H149_P1_200723_1000/quast/quast.done jobid: 0 reason: Forced execution @@ -17,11 +17,11 @@ rule quast: resources: mem_mb=4000, mem_mib=3815, disk_mb=6823, disk_mib=6507, tmpdir=/local/scratch/tmp, partition=common, qos=normal, mem=4G, gres= - quast.py -e --rna-finding -t 4 H149_P1_200723_1000/sorted_contigs/H149_P1_200723_1000.fasta --nanopore /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/data/filters/H149_P1_200723_1000.fastq.gz -o H149_P1_200723_1000/quast && touch H149_P1_200723_1000/quast/quast.done + quast.py -e --rna-finding -t 4 H149_P1_200723_1000/sorted_contigs/H149_P1_200723_1000.fasta --nanopore //Drylab/InProgress/B17420/data/filters/H149_P1_200723_1000.fastq.gz -o H149_P1_200723_1000/quast && touch H149_P1_200723_1000/quast/quast.done -Activating singularity image /pasteur/zeus/projets/p02/Biomics/resources/condas/sequana_0.16/apptainers/847f147b6a61a6684f5106b4bc25a0c7.simg +Activating singularity image //resources/condas/sequana_0.16/apptainers/847f147b6a61a6684f5106b4bc25a0c7.simg WARNING: Python locale settings can't be changed -/usr/local/bin/quast.py -e --rna-finding -t 4 H149_P1_200723_1000/sorted_contigs/H149_P1_200723_1000.fasta --nanopore /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/data/filters/H149_P1_200723_1000.fastq.gz -o H149_P1_200723_1000/quast +/usr/local/bin/quast.py -e --rna-finding -t 4 H149_P1_200723_1000/sorted_contigs/H149_P1_200723_1000.fasta --nanopore //Drylab/InProgress/B17420/data/filters/H149_P1_200723_1000.fastq.gz -o H149_P1_200723_1000/quast Version: 5.2.0 @@ -32,9 +32,9 @@ System information: Started: 2024-01-06 02:48:08 -Logging to /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_1000/quast/quast.log +Logging to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_1000/quast/quast.log -CWD: /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate +CWD: //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate Main parameters: MODE: default, threads: 4, eukaryotic: true, min contig length: 500, min alignment length: 65, \ min alignment IDY: 95.0, ambiguity: one, min local misassembly length: 200, min extensive misassembly length: 1000 @@ -45,14 +45,14 @@ Contigs: 2024-01-06 02:48:10 Running Reads analyzer... - Logging to files /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_1000/quast/reads_stats/reads_stats.log and /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_1000/quast/reads_stats/reads_stats.err... + Logging to files //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_1000/quast/reads_stats/reads_stats.log and //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_1000/quast/reads_stats/reads_stats.err... Pre-processing reads... Running BWA... Done. Sorting SAM-file... Analysis is finished. Creating total report... - saved to /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_1000/quast/reads_stats/reads_report.txt, reads_report.tsv, and reads_report.tex + saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_1000/quast/reads_stats/reads_report.txt, reads_report.tsv, and reads_report.tex Done. 2024-01-06 03:29:26 @@ -62,22 +62,22 @@ Running Basic statistics processor... Calculating N50 and L50... H149_P1_200723_1000, N50 = 787408, L50 = 12, auN = 1066532.2, Total length = 33703652, GC % = 59.64, # N's per 100 kbp = 0.00 Drawing Nx plot... - saved to /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_1000/quast/basic_stats/Nx_plot.pdf + saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_1000/quast/basic_stats/Nx_plot.pdf Drawing cumulative plot... - saved to /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_1000/quast/basic_stats/cumulative_plot.pdf + saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_1000/quast/basic_stats/cumulative_plot.pdf Drawing GC content plot... - saved to /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_1000/quast/basic_stats/GC_content_plot.pdf + saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_1000/quast/basic_stats/GC_content_plot.pdf Drawing H149_P1_200723_1000 GC content plot... - saved to /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_1000/quast/basic_stats/H149_P1_200723_1000_GC_content_plot.pdf + saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_1000/quast/basic_stats/H149_P1_200723_1000_GC_content_plot.pdf Done. NOTICE: Genes are not predicted by default. Use --gene-finding or --glimmer option to enable it. 2024-01-06 03:29:28 Running Barrnap... -Logging to /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_1000/quast/predicted_genes/barrnap.log... +Logging to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_1000/quast/predicted_genes/barrnap.log... Ribosomal RNA genes = 15 - Predicted genes (GFF): /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_1000/quast/predicted_genes/H149_P1_200723_1000.rna.gff + Predicted genes (GFF): //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_1000/quast/predicted_genes/H149_P1_200723_1000.rna.gff Done. 2024-01-06 03:29:34 @@ -89,12 +89,12 @@ Done 2024-01-06 03:29:35 RESULTS: - Text versions of total report are saved to /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_1000/quast/report.txt, report.tsv, and report.tex - Text versions of transposed total report are saved to /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_1000/quast/transposed_report.txt, transposed_report.tsv, and transposed_report.tex - HTML version (interactive tables and plots) is saved to /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_1000/quast/report.html - PDF version (tables and plots) is saved to /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_1000/quast/report.pdf - Icarus (contig browser) is saved to /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_1000/quast/icarus.html - Log is saved to /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_1000/quast/quast.log + Text versions of total report are saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_1000/quast/report.txt, report.tsv, and report.tex + Text versions of transposed total report are saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_1000/quast/transposed_report.txt, transposed_report.tsv, and transposed_report.tex + HTML version (interactive tables and plots) is saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_1000/quast/report.html + PDF version (tables and plots) is saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_1000/quast/report.pdf + Icarus (contig browser) is saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_1000/quast/icarus.html + Log is saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_1000/quast/quast.log Finished: 2024-01-06 03:29:35 Elapsed time: 0:41:27.307706 diff --git a/tests/snaketools/data/slurm_error1/logs/quast/quast-sample=H149_P1_200723_500-slurm-54466795.out b/tests/snaketools/data/slurm_error1/logs/quast/quast-sample=H149_P1_200723_500-slurm-54466795.out index c635ca1..4163d91 100755 --- a/tests/snaketools/data/slurm_error1/logs/quast/quast-sample=H149_P1_200723_500-slurm-54466795.out +++ b/tests/snaketools/data/slurm_error1/logs/quast/quast-sample=H149_P1_200723_500-slurm-54466795.out @@ -8,7 +8,7 @@ Select jobs to execute... [Sat Jan 6 02:42:28 2024] rule quast: - input: /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/data/filters/H149_P1_200723_500.fastq.gz, H149_P1_200723_500/sorted_contigs/H149_P1_200723_500.fasta + input: //Drylab/InProgress/B17420/data/filters/H149_P1_200723_500.fastq.gz, H149_P1_200723_500/sorted_contigs/H149_P1_200723_500.fasta output: H149_P1_200723_500/quast/quast.done jobid: 0 reason: Forced execution @@ -17,11 +17,11 @@ rule quast: resources: mem_mb=4000, mem_mib=3815, disk_mb=6903, disk_mib=6584, tmpdir=/local/scratch/tmp, partition=common, qos=normal, mem=4G, gres= - quast.py -e --rna-finding -t 4 H149_P1_200723_500/sorted_contigs/H149_P1_200723_500.fasta --nanopore /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/data/filters/H149_P1_200723_500.fastq.gz -o H149_P1_200723_500/quast && touch H149_P1_200723_500/quast/quast.done + quast.py -e --rna-finding -t 4 H149_P1_200723_500/sorted_contigs/H149_P1_200723_500.fasta --nanopore //Drylab/InProgress/B17420/data/filters/H149_P1_200723_500.fastq.gz -o H149_P1_200723_500/quast && touch H149_P1_200723_500/quast/quast.done -Activating singularity image /pasteur/zeus/projets/p02/Biomics/resources/condas/sequana_0.16/apptainers/847f147b6a61a6684f5106b4bc25a0c7.simg +Activating singularity image //resources/condas/sequana_0.16/apptainers/847f147b6a61a6684f5106b4bc25a0c7.simg WARNING: Python locale settings can't be changed -/usr/local/bin/quast.py -e --rna-finding -t 4 H149_P1_200723_500/sorted_contigs/H149_P1_200723_500.fasta --nanopore /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/data/filters/H149_P1_200723_500.fastq.gz -o H149_P1_200723_500/quast +/usr/local/bin/quast.py -e --rna-finding -t 4 H149_P1_200723_500/sorted_contigs/H149_P1_200723_500.fasta --nanopore //Drylab/InProgress/B17420/data/filters/H149_P1_200723_500.fastq.gz -o H149_P1_200723_500/quast Version: 5.2.0 @@ -32,9 +32,9 @@ System information: Started: 2024-01-06 02:42:30 -Logging to /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_500/quast/quast.log +Logging to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_500/quast/quast.log -CWD: /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate +CWD: //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate Main parameters: MODE: default, threads: 4, eukaryotic: true, min contig length: 500, min alignment length: 65, \ min alignment IDY: 95.0, ambiguity: one, min local misassembly length: 200, min extensive misassembly length: 1000 @@ -45,14 +45,14 @@ Contigs: 2024-01-06 02:42:32 Running Reads analyzer... - Logging to files /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_500/quast/reads_stats/reads_stats.log and /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_500/quast/reads_stats/reads_stats.err... + Logging to files //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_500/quast/reads_stats/reads_stats.log and //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_500/quast/reads_stats/reads_stats.err... Pre-processing reads... Running BWA... Done. Sorting SAM-file... Analysis is finished. Creating total report... - saved to /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_500/quast/reads_stats/reads_report.txt, reads_report.tsv, and reads_report.tex + saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_500/quast/reads_stats/reads_report.txt, reads_report.tsv, and reads_report.tex Done. 2024-01-06 03:25:10 @@ -62,22 +62,22 @@ Running Basic statistics processor... Calculating N50 and L50... H149_P1_200723_500, N50 = 787466, L50 = 12, auN = 1062323.0, Total length = 33723594, GC % = 59.65, # N's per 100 kbp = 0.00 Drawing Nx plot... - saved to /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_500/quast/basic_stats/Nx_plot.pdf + saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_500/quast/basic_stats/Nx_plot.pdf Drawing cumulative plot... - saved to /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_500/quast/basic_stats/cumulative_plot.pdf + saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_500/quast/basic_stats/cumulative_plot.pdf Drawing GC content plot... - saved to /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_500/quast/basic_stats/GC_content_plot.pdf + saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_500/quast/basic_stats/GC_content_plot.pdf Drawing H149_P1_200723_500 GC content plot... - saved to /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_500/quast/basic_stats/H149_P1_200723_500_GC_content_plot.pdf + saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_500/quast/basic_stats/H149_P1_200723_500_GC_content_plot.pdf Done. NOTICE: Genes are not predicted by default. Use --gene-finding or --glimmer option to enable it. 2024-01-06 03:25:13 Running Barrnap... -Logging to /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_500/quast/predicted_genes/barrnap.log... +Logging to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_500/quast/predicted_genes/barrnap.log... Ribosomal RNA genes = 15 - Predicted genes (GFF): /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_500/quast/predicted_genes/H149_P1_200723_500.rna.gff + Predicted genes (GFF): //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_500/quast/predicted_genes/H149_P1_200723_500.rna.gff Done. 2024-01-06 03:25:19 @@ -89,12 +89,12 @@ Done 2024-01-06 03:25:20 RESULTS: - Text versions of total report are saved to /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_500/quast/report.txt, report.tsv, and report.tex - Text versions of transposed total report are saved to /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_500/quast/transposed_report.txt, transposed_report.tsv, and transposed_report.tex - HTML version (interactive tables and plots) is saved to /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_500/quast/report.html - PDF version (tables and plots) is saved to /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_500/quast/report.pdf - Icarus (contig browser) is saved to /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_500/quast/icarus.html - Log is saved to /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_500/quast/quast.log + Text versions of total report are saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_500/quast/report.txt, report.tsv, and report.tex + Text versions of transposed total report are saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_500/quast/transposed_report.txt, transposed_report.tsv, and transposed_report.tex + HTML version (interactive tables and plots) is saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_500/quast/report.html + PDF version (tables and plots) is saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_500/quast/report.pdf + Icarus (contig browser) is saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_500/quast/icarus.html + Log is saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_500/quast/quast.log Finished: 2024-01-06 03:25:20 Elapsed time: 0:42:50.432404 diff --git a/tests/snaketools/data/slurm_error1/slurm-54463807.out b/tests/snaketools/data/slurm_error1/slurm-54463807.out index 4f4a0f1..eead520 100755 --- a/tests/snaketools/data/slurm_error1/slurm-54463807.out +++ b/tests/snaketools/data/slurm_error1/slurm-54463807.out @@ -1,10 +1,5 @@ Using profile .sequana/profile_slurm for setting default command line arguments. Building DAG of jobs... -File path git+file:///pasteur/zeus/projets/p02/Biomics/resources/condas/sequana_0.16/github/sequana-wrappers/wrappers/rulegraph/environment.yaml@main contains double '/'. This is likely unintended. It can also lead to inconsistent results of the file-matching approach used by Snakemake. -File path git+file:///pasteur/zeus/projets/p02/Biomics/resources/condas/sequana_0.16/github/sequana-wrappers/wrappers/multiqc/environment.yaml@v23.11.28 contains double '/'. This is likely unintended. It can also lead to inconsistent results of the file-matching approach used by Snakemake. -File path git+file:///pasteur/zeus/projets/p02/Biomics/resources/condas/sequana_0.16/github/sequana-wrappers/wrappers/busco/environment.yaml@v23.11.28 contains double '/'. This is likely unintended. It can also lead to inconsistent results of the file-matching approach used by Snakemake. -File path git+file:///pasteur/zeus/projets/p02/Biomics/resources/condas/sequana_0.16/github/sequana-wrappers/wrappers/busco/environment.yaml@v23.11.28 contains double '/'. This is likely unintended. It can also lead to inconsistent results of the file-matching approach used by Snakemake. -File path git+file:///pasteur/zeus/projets/p02/Biomics/resources/condas/sequana_0.16/github/sequana-wrappers/wrappers/busco/environment.yaml@v23.11.28 contains double '/'. This is likely unintended. It can also lead to inconsistent results of the file-matching approach used by Snakemake. Using shell: /usr/bin/bash Provided cluster nodes: 40 Conda environments: ignored @@ -29,7 +24,7 @@ Select jobs to execute... [Fri Jan 5 20:40:37 2024] localrule rulegraph: - input: /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/lora.rules + input: lora.rules output: rulegraph/rg.ann.dot jobid: 2 reason: Missing output files: rulegraph/rg.ann.dot @@ -37,7 +32,7 @@ localrule rulegraph: [Fri Jan 5 20:40:37 2024] rule flye: - input: /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/data/filters/H149_P1_200723_500.fastq.gz + input: H149_P1_200723_500.fastq.gz output: H149_P1_200723_500/flye/H149_P1_200723_500.contigs.fasta jobid: 9 reason: Missing output files: H149_P1_200723_500/flye/H149_P1_200723_500.contigs.fasta @@ -48,13 +43,13 @@ rule flye: outdir="$(dirname "H149_P1_200723_500/flye/H149_P1_200723_500.contigs.fasta")" - flye --nano-raw /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/data/filters/H149_P1_200723_500.fastq.gz --out-dir ${outdir} --threads 4 && mv ${outdir}/assembly.fasta H149_P1_200723_500/flye/H149_P1_200723_500.contigs.fasta + flye --nano-raw H149_P1_200723_500.fastq.gz --out-dir ${outdir} --threads 4 && mv ${outdir}/assembly.fasta H149_P1_200723_500/flye/H149_P1_200723_500.contigs.fasta Submitted job 9 with external jobid 'Submitted batch job 54463808'. [Fri Jan 5 20:40:37 2024] rule flye: - input: /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/data/filters/H149_P1_200723_1000.fastq.gz + input: H149_P1_200723_1000.fastq.gz output: H149_P1_200723_1000/flye/H149_P1_200723_1000.contigs.fasta jobid: 12 reason: Missing output files: H149_P1_200723_1000/flye/H149_P1_200723_1000.contigs.fasta @@ -65,13 +60,13 @@ rule flye: outdir="$(dirname "H149_P1_200723_1000/flye/H149_P1_200723_1000.contigs.fasta")" - flye --nano-raw /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/data/filters/H149_P1_200723_1000.fastq.gz --out-dir ${outdir} --threads 4 && mv ${outdir}/assembly.fasta H149_P1_200723_1000/flye/H149_P1_200723_1000.contigs.fasta + flye --nano-raw //Drylab/InProgress/B17420/data/filters/H149_P1_200723_1000.fastq.gz --out-dir ${outdir} --threads 4 && mv ${outdir}/assembly.fasta H149_P1_200723_1000/flye/H149_P1_200723_1000.contigs.fasta Submitted job 12 with external jobid 'Submitted batch job 54463809'. [Fri Jan 5 20:40:37 2024] rule flye: - input: /pasteur/zeus/projets/p02/BiomicsData/2023/DryLab/B17420/20230720_1448_MN18671_FAV94301_57a76bad/nanomerge/B17420/H149_P1_200723.fastq.gz + input: //2023/DryLab/B17420/20230720_1448_MN18671_FAV94301_57a76bad/nanomerge/B17420/H149_P1_200723.fastq.gz output: H149_P1_200723/flye/H149_P1_200723.contigs.fasta jobid: 6 reason: Missing output files: H149_P1_200723/flye/H149_P1_200723.contigs.fasta @@ -82,11 +77,11 @@ rule flye: outdir="$(dirname "H149_P1_200723/flye/H149_P1_200723.contigs.fasta")" - flye --nano-raw /pasteur/zeus/projets/p02/BiomicsData/2023/DryLab/B17420/20230720_1448_MN18671_FAV94301_57a76bad/nanomerge/B17420/H149_P1_200723.fastq.gz --out-dir ${outdir} --threads 4 && mv ${outdir}/assembly.fasta H149_P1_200723/flye/H149_P1_200723.contigs.fasta + flye --nano-raw //2023/DryLab/B17420/20230720_1448_MN18671_FAV94301_57a76bad/nanomerge/B17420/H149_P1_200723.fastq.gz --out-dir ${outdir} --threads 4 && mv ${outdir}/assembly.fasta H149_P1_200723/flye/H149_P1_200723.contigs.fasta Submitted job 6 with external jobid 'Submitted batch job 54463810'. -/pasteur/zeus/projets/p02/Biomics/resources/condas/sequana_0.16/bin/python3.1 /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/.snakemake/scripts/tmpal3rxj_z.wrapper.py -cd rulegraph && snakemake -s "/pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/lora.rules" --rulegraph --nolock > rg.dot; cd .. +//resources/condas/sequana_0.16/bin/python3.1 //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/.snakemake/scripts/tmpal3rxj_z.wrapper.py +cd rulegraph && snakemake -s "//Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/lora.rules" --rulegraph --nolock > rg.dot; cd .. Building DAG of jobs... [Fri Jan 5 20:40:41 2024] Finished job 2. @@ -104,7 +99,7 @@ localrule create_svg: dot -Tsvg rulegraph/rg.ann.dot -o .sequana/rulegraph.svg -Activating singularity image /pasteur/zeus/projets/p02/Biomics/resources/condas/sequana_0.16/apptainers/423bed3d162e90bd043c210525c5d23e.simg +Activating singularity image //resources/condas/sequana_0.16/apptainers/423bed3d162e90bd043c210525c5d23e.simg [Fri Jan 5 20:40:42 2024] Finished job 1. 2 of 28 steps (7%) done @@ -135,7 +130,7 @@ Select jobs to execute... [Sat Jan 6 02:42:23 2024] rule minimap2_and_genomecov: - input: /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/data/filters/H149_P1_200723_500.fastq.gz, H149_P1_200723_500/sorted_contigs/H149_P1_200723_500.fasta + input: //Drylab/InProgress/B17420/data/filters/H149_P1_200723_500.fastq.gz, H149_P1_200723_500/sorted_contigs/H149_P1_200723_500.fasta output: H149_P1_200723_500/minimap2/H149_P1_200723_500.bam, H149_P1_200723_500/minimap2/H149_P1_200723_500.bed jobid: 16 reason: Missing output files: H149_P1_200723_500/minimap2/H149_P1_200723_500.bed; Input files updated by another job: H149_P1_200723_500/sorted_contigs/H149_P1_200723_500.fasta @@ -144,7 +139,7 @@ rule minimap2_and_genomecov: resources: mem_mb=4000, mem_mib=6584, disk_mb=6903, disk_mib=6584, tmpdir=, partition=common, qos=normal, mem=4G, gres= - minimap2 -t 4 -ax map-ont H149_P1_200723_500/sorted_contigs/H149_P1_200723_500.fasta /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/data/filters/H149_P1_200723_500.fastq.gz | samtools sort -@ $((4 - 1)) -o H149_P1_200723_500/minimap2/H149_P1_200723_500.bam && samtools index H149_P1_200723_500/minimap2/H149_P1_200723_500.bam && samtools depth -aa H149_P1_200723_500/minimap2/H149_P1_200723_500.bam > H149_P1_200723_500/minimap2/H149_P1_200723_500.bed + minimap2 -t 4 -ax map-ont H149_P1_200723_500/sorted_contigs/H149_P1_200723_500.fasta //Drylab/InProgress/B17420/data/filters/H149_P1_200723_500.fastq.gz | samtools sort -@ $((4 - 1)) -o H149_P1_200723_500/minimap2/H149_P1_200723_500.bam && samtools index H149_P1_200723_500/minimap2/H149_P1_200723_500.bam && samtools depth -aa H149_P1_200723_500/minimap2/H149_P1_200723_500.bam > H149_P1_200723_500/minimap2/H149_P1_200723_500.bed Submitted job 16 with external jobid 'Submitted batch job 54466792'. @@ -177,7 +172,7 @@ Submitted job 20 with external jobid 'Submitted batch job 54466794'. [Sat Jan 6 02:42:23 2024] rule quast: - input: /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/data/filters/H149_P1_200723_500.fastq.gz, H149_P1_200723_500/sorted_contigs/H149_P1_200723_500.fasta + input: //Drylab/InProgress/B17420/data/filters/H149_P1_200723_500.fastq.gz, H149_P1_200723_500/sorted_contigs/H149_P1_200723_500.fasta output: H149_P1_200723_500/quast/quast.done jobid: 7 reason: Missing output files: H149_P1_200723_500/quast/quast.done; Input files updated by another job: H149_P1_200723_500/sorted_contigs/H149_P1_200723_500.fasta @@ -186,7 +181,7 @@ rule quast: resources: mem_mb=4000, mem_mib=6584, disk_mb=6903, disk_mib=6584, tmpdir=, partition=common, qos=normal, mem=4G, gres= - quast.py -e --rna-finding -t 4 H149_P1_200723_500/sorted_contigs/H149_P1_200723_500.fasta --nanopore /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/data/filters/H149_P1_200723_500.fastq.gz -o H149_P1_200723_500/quast && touch H149_P1_200723_500/quast/quast.done + quast.py -e --rna-finding -t 4 H149_P1_200723_500/sorted_contigs/H149_P1_200723_500.fasta --nanopore //Drylab/InProgress/B17420/data/filters/H149_P1_200723_500.fastq.gz -o H149_P1_200723_500/quast && touch H149_P1_200723_500/quast/quast.done Submitted job 7 with external jobid 'Submitted batch job 54466795'. [Sat Jan 6 02:42:33 2024] @@ -205,23 +200,23 @@ rule blast: resources: mem_mb=12000, mem_mib=954, disk_mb=1000, disk_mib=954, tmpdir=, partition=common, qos=normal, mem=12G, gres= - export BLASTDB=/pasteur/zeus/projets/p02/Biomics/resources/blast/ + export BLASTDB=//resources/blast/ blastn -query H149_P1_200723_500/subset_contigs/H149_P1_200723_500.subset.fasta -db nt -evalue 1e-10 -out H149_P1_200723_500/blast/H149_P1_200723_500.tsv -num_threads 4 -max_target_seqs 5 -outfmt "6 qseqid sseqid pident length mismatch gapopen qstart qend sstart send evalue bitscore staxids stitle" Submitted job 24 with external jobid 'Submitted batch job 54466796'. [Sat Jan 6 02:43:33 2024] Error in rule minimap2_and_genomecov: jobid: 16 - input: /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/data/filters/H149_P1_200723_500.fastq.gz, H149_P1_200723_500/sorted_contigs/H149_P1_200723_500.fasta + input: //Drylab/InProgress/B17420/data/filters/H149_P1_200723_500.fastq.gz, H149_P1_200723_500/sorted_contigs/H149_P1_200723_500.fasta output: H149_P1_200723_500/minimap2/H149_P1_200723_500.bam, H149_P1_200723_500/minimap2/H149_P1_200723_500.bed shell: - minimap2 -t 4 -ax map-ont H149_P1_200723_500/sorted_contigs/H149_P1_200723_500.fasta /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/data/filters/H149_P1_200723_500.fastq.gz | samtools sort -@ $((4 - 1)) -o H149_P1_200723_500/minimap2/H149_P1_200723_500.bam && samtools index H149_P1_200723_500/minimap2/H149_P1_200723_500.bam && samtools depth -aa H149_P1_200723_500/minimap2/H149_P1_200723_500.bam > H149_P1_200723_500/minimap2/H149_P1_200723_500.bed + minimap2 -t 4 -ax map-ont H149_P1_200723_500/sorted_contigs/H149_P1_200723_500.fasta //Drylab/InProgress/B17420/data/filters/H149_P1_200723_500.fastq.gz | samtools sort -@ $((4 - 1)) -o H149_P1_200723_500/minimap2/H149_P1_200723_500.bam && samtools index H149_P1_200723_500/minimap2/H149_P1_200723_500.bam && samtools depth -aa H149_P1_200723_500/minimap2/H149_P1_200723_500.bam > H149_P1_200723_500/minimap2/H149_P1_200723_500.bed (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!) cluster_jobid: Submitted batch job 54466792 -Error executing rule minimap2_and_genomecov on cluster (jobid: 16, external: Submitted batch job 54466792, jobscript: /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/.snakemake/tmp.uxjuhhze/snakejob.minimap2_and_genomecov.16.sh). For error details see the cluster log and the log files of the involved rule(s). +Error executing rule minimap2_and_genomecov on cluster (jobid: 16, external: Submitted batch job 54466792, jobscript: //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/.snakemake/tmp.uxjuhhze/snakejob.minimap2_and_genomecov.16.sh). For error details see the cluster log and the log files of the involved rule(s). [Sat Jan 6 02:44:03 2024] Finished job 20. 6 of 28 steps (21%) done @@ -264,7 +259,7 @@ Submitted job 19 with external jobid 'Submitted batch job 54467144'. [Sat Jan 6 02:45:53 2024] rule quast: - input: /pasteur/zeus/projets/p02/BiomicsData/2023/DryLab/B17420/20230720_1448_MN18671_FAV94301_57a76bad/nanomerge/B17420/H149_P1_200723.fastq.gz, H149_P1_200723/sorted_contigs/H149_P1_200723.fasta + input: //2023/DryLab/B17420/20230720_1448_MN18671_FAV94301_57a76bad/nanomerge/B17420/H149_P1_200723.fastq.gz, H149_P1_200723/sorted_contigs/H149_P1_200723.fasta output: H149_P1_200723/quast/quast.done jobid: 4 reason: Missing output files: H149_P1_200723/quast/quast.done; Input files updated by another job: H149_P1_200723/sorted_contigs/H149_P1_200723.fasta @@ -273,13 +268,13 @@ rule quast: resources: mem_mb=4000, mem_mib=6612, disk_mb=6933, disk_mib=6612, tmpdir=, partition=common, qos=normal, mem=4G, gres= - quast.py -e --rna-finding -t 4 H149_P1_200723/sorted_contigs/H149_P1_200723.fasta --nanopore /pasteur/zeus/projets/p02/BiomicsData/2023/DryLab/B17420/20230720_1448_MN18671_FAV94301_57a76bad/nanomerge/B17420/H149_P1_200723.fastq.gz -o H149_P1_200723/quast && touch H149_P1_200723/quast/quast.done + quast.py -e --rna-finding -t 4 H149_P1_200723/sorted_contigs/H149_P1_200723.fasta --nanopore //2023/DryLab/B17420/20230720_1448_MN18671_FAV94301_57a76bad/nanomerge/B17420/H149_P1_200723.fastq.gz -o H149_P1_200723/quast && touch H149_P1_200723/quast/quast.done Submitted job 4 with external jobid 'Submitted batch job 54467145'. [Sat Jan 6 02:45:53 2024] rule minimap2_and_genomecov: - input: /pasteur/zeus/projets/p02/BiomicsData/2023/DryLab/B17420/20230720_1448_MN18671_FAV94301_57a76bad/nanomerge/B17420/H149_P1_200723.fastq.gz, H149_P1_200723/sorted_contigs/H149_P1_200723.fasta + input: //2023/DryLab/B17420/20230720_1448_MN18671_FAV94301_57a76bad/nanomerge/B17420/H149_P1_200723.fastq.gz, H149_P1_200723/sorted_contigs/H149_P1_200723.fasta output: H149_P1_200723/minimap2/H149_P1_200723.bam, H149_P1_200723/minimap2/H149_P1_200723.bed jobid: 14 reason: Missing output files: H149_P1_200723/minimap2/H149_P1_200723.bed; Input files updated by another job: H149_P1_200723/sorted_contigs/H149_P1_200723.fasta @@ -288,7 +283,7 @@ rule minimap2_and_genomecov: resources: mem_mb=4000, mem_mib=6612, disk_mb=6933, disk_mib=6612, tmpdir=, partition=common, qos=normal, mem=4G, gres= - minimap2 -t 4 -ax map-ont H149_P1_200723/sorted_contigs/H149_P1_200723.fasta /pasteur/zeus/projets/p02/BiomicsData/2023/DryLab/B17420/20230720_1448_MN18671_FAV94301_57a76bad/nanomerge/B17420/H149_P1_200723.fastq.gz | samtools sort -@ $((4 - 1)) -o H149_P1_200723/minimap2/H149_P1_200723.bam && samtools index H149_P1_200723/minimap2/H149_P1_200723.bam && samtools depth -aa H149_P1_200723/minimap2/H149_P1_200723.bam > H149_P1_200723/minimap2/H149_P1_200723.bed + minimap2 -t 4 -ax map-ont H149_P1_200723/sorted_contigs/H149_P1_200723.fasta //2023/DryLab/B17420/20230720_1448_MN18671_FAV94301_57a76bad/nanomerge/B17420/H149_P1_200723.fastq.gz | samtools sort -@ $((4 - 1)) -o H149_P1_200723/minimap2/H149_P1_200723.bam && samtools index H149_P1_200723/minimap2/H149_P1_200723.bam && samtools depth -aa H149_P1_200723/minimap2/H149_P1_200723.bam > H149_P1_200723/minimap2/H149_P1_200723.bed Submitted job 14 with external jobid 'Submitted batch job 54467146'. @@ -321,7 +316,7 @@ rule blast: resources: mem_mb=12000, mem_mib=954, disk_mb=1000, disk_mib=954, tmpdir=, partition=common, qos=normal, mem=12G, gres= - export BLASTDB=/pasteur/zeus/projets/p02/Biomics/resources/blast/ + export BLASTDB=//resources/blast/ blastn -query H149_P1_200723/subset_contigs/H149_P1_200723.subset.fasta -db nt -evalue 1e-10 -out H149_P1_200723/blast/H149_P1_200723.tsv -num_threads 4 -max_target_seqs 5 -outfmt "6 qseqid sseqid pident length mismatch gapopen qstart qend sstart send evalue bitscore staxids stitle" Submitted job 22 with external jobid 'Submitted batch job 54467194'. @@ -331,16 +326,16 @@ Finished job 19. [Sat Jan 6 02:47:33 2024] Error in rule minimap2_and_genomecov: jobid: 14 - input: /pasteur/zeus/projets/p02/BiomicsData/2023/DryLab/B17420/20230720_1448_MN18671_FAV94301_57a76bad/nanomerge/B17420/H149_P1_200723.fastq.gz, H149_P1_200723/sorted_contigs/H149_P1_200723.fasta + input: //2023/DryLab/B17420/20230720_1448_MN18671_FAV94301_57a76bad/nanomerge/B17420/H149_P1_200723.fastq.gz, H149_P1_200723/sorted_contigs/H149_P1_200723.fasta output: H149_P1_200723/minimap2/H149_P1_200723.bam, H149_P1_200723/minimap2/H149_P1_200723.bed shell: - minimap2 -t 4 -ax map-ont H149_P1_200723/sorted_contigs/H149_P1_200723.fasta /pasteur/zeus/projets/p02/BiomicsData/2023/DryLab/B17420/20230720_1448_MN18671_FAV94301_57a76bad/nanomerge/B17420/H149_P1_200723.fastq.gz | samtools sort -@ $((4 - 1)) -o H149_P1_200723/minimap2/H149_P1_200723.bam && samtools index H149_P1_200723/minimap2/H149_P1_200723.bam && samtools depth -aa H149_P1_200723/minimap2/H149_P1_200723.bam > H149_P1_200723/minimap2/H149_P1_200723.bed + minimap2 -t 4 -ax map-ont H149_P1_200723/sorted_contigs/H149_P1_200723.fasta //2023/DryLab/B17420/20230720_1448_MN18671_FAV94301_57a76bad/nanomerge/B17420/H149_P1_200723.fastq.gz | samtools sort -@ $((4 - 1)) -o H149_P1_200723/minimap2/H149_P1_200723.bam && samtools index H149_P1_200723/minimap2/H149_P1_200723.bam && samtools depth -aa H149_P1_200723/minimap2/H149_P1_200723.bam > H149_P1_200723/minimap2/H149_P1_200723.bed (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!) cluster_jobid: Submitted batch job 54467146 -Error executing rule minimap2_and_genomecov on cluster (jobid: 14, external: Submitted batch job 54467146, jobscript: /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/.snakemake/tmp.uxjuhhze/snakejob.minimap2_and_genomecov.14.sh). For error details see the cluster log and the log files of the involved rule(s). +Error executing rule minimap2_and_genomecov on cluster (jobid: 14, external: Submitted batch job 54467146, jobscript: //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/.snakemake/tmp.uxjuhhze/snakejob.minimap2_and_genomecov.14.sh). For error details see the cluster log and the log files of the involved rule(s). [Sat Jan 6 02:47:53 2024] Finished job 12. 11 of 28 steps (39%) done @@ -367,7 +362,7 @@ Select jobs to execute... [Sat Jan 6 02:48:03 2024] rule quast: - input: /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/data/filters/H149_P1_200723_1000.fastq.gz, H149_P1_200723_1000/sorted_contigs/H149_P1_200723_1000.fasta + input: //Drylab/InProgress/B17420/data/filters/H149_P1_200723_1000.fastq.gz, H149_P1_200723_1000/sorted_contigs/H149_P1_200723_1000.fasta output: H149_P1_200723_1000/quast/quast.done jobid: 10 reason: Missing output files: H149_P1_200723_1000/quast/quast.done; Input files updated by another job: H149_P1_200723_1000/sorted_contigs/H149_P1_200723_1000.fasta @@ -376,7 +371,7 @@ rule quast: resources: mem_mb=4000, mem_mib=6507, disk_mb=6823, disk_mib=6507, tmpdir=, partition=common, qos=normal, mem=4G, gres= - quast.py -e --rna-finding -t 4 H149_P1_200723_1000/sorted_contigs/H149_P1_200723_1000.fasta --nanopore /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/data/filters/H149_P1_200723_1000.fastq.gz -o H149_P1_200723_1000/quast && touch H149_P1_200723_1000/quast/quast.done + quast.py -e --rna-finding -t 4 H149_P1_200723_1000/sorted_contigs/H149_P1_200723_1000.fasta --nanopore //Drylab/InProgress/B17420/data/filters/H149_P1_200723_1000.fastq.gz -o H149_P1_200723_1000/quast && touch H149_P1_200723_1000/quast/quast.done Submitted job 10 with external jobid 'Submitted batch job 54467578'. @@ -409,7 +404,7 @@ Submitted job 21 with external jobid 'Submitted batch job 54467580'. [Sat Jan 6 02:48:04 2024] rule minimap2_and_genomecov: - input: /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/data/filters/H149_P1_200723_1000.fastq.gz, H149_P1_200723_1000/sorted_contigs/H149_P1_200723_1000.fasta + input: //Drylab/InProgress/B17420/data/filters/H149_P1_200723_1000.fastq.gz, H149_P1_200723_1000/sorted_contigs/H149_P1_200723_1000.fasta output: H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bam, H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bed jobid: 18 reason: Missing output files: H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bed; Input files updated by another job: H149_P1_200723_1000/sorted_contigs/H149_P1_200723_1000.fasta @@ -418,7 +413,7 @@ rule minimap2_and_genomecov: resources: mem_mb=4000, mem_mib=6507, disk_mb=6823, disk_mib=6507, tmpdir=, partition=common, qos=normal, mem=4G, gres= - minimap2 -t 4 -ax map-ont H149_P1_200723_1000/sorted_contigs/H149_P1_200723_1000.fasta /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/data/filters/H149_P1_200723_1000.fastq.gz | samtools sort -@ $((4 - 1)) -o H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bam && samtools index H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bam && samtools depth -aa H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bam > H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bed + minimap2 -t 4 -ax map-ont H149_P1_200723_1000/sorted_contigs/H149_P1_200723_1000.fasta //Drylab/InProgress/B17420/data/filters/H149_P1_200723_1000.fastq.gz | samtools sort -@ $((4 - 1)) -o H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bam && samtools index H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bam && samtools depth -aa H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bam > H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bed Submitted job 18 with external jobid 'Submitted batch job 54467581'. [Sat Jan 6 02:48:13 2024] @@ -437,7 +432,7 @@ rule blast: resources: mem_mb=12000, mem_mib=954, disk_mb=1000, disk_mib=954, tmpdir=, partition=common, qos=normal, mem=12G, gres= - export BLASTDB=/pasteur/zeus/projets/p02/Biomics/resources/blast/ + export BLASTDB=//resources/blast/ blastn -query H149_P1_200723_1000/subset_contigs/H149_P1_200723_1000.subset.fasta -db nt -evalue 1e-10 -out H149_P1_200723_1000/blast/H149_P1_200723_1000.tsv -num_threads 4 -max_target_seqs 5 -outfmt "6 qseqid sseqid pident length mismatch gapopen qstart qend sstart send evalue bitscore staxids stitle" Submitted job 26 with external jobid 'Submitted batch job 54467619'. @@ -447,16 +442,16 @@ Finished job 21. [Sat Jan 6 02:50:54 2024] Error in rule minimap2_and_genomecov: jobid: 18 - input: /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/data/filters/H149_P1_200723_1000.fastq.gz, H149_P1_200723_1000/sorted_contigs/H149_P1_200723_1000.fasta + input: //Drylab/InProgress/B17420/data/filters/H149_P1_200723_1000.fastq.gz, H149_P1_200723_1000/sorted_contigs/H149_P1_200723_1000.fasta output: H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bam, H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bed shell: - minimap2 -t 4 -ax map-ont H149_P1_200723_1000/sorted_contigs/H149_P1_200723_1000.fasta /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/data/filters/H149_P1_200723_1000.fastq.gz | samtools sort -@ $((4 - 1)) -o H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bam && samtools index H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bam && samtools depth -aa H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bam > H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bed + minimap2 -t 4 -ax map-ont H149_P1_200723_1000/sorted_contigs/H149_P1_200723_1000.fasta //Drylab/InProgress/B17420/data/filters/H149_P1_200723_1000.fastq.gz | samtools sort -@ $((4 - 1)) -o H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bam && samtools index H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bam && samtools depth -aa H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bam > H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bed (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!) cluster_jobid: Submitted batch job 54467581 -Error executing rule minimap2_and_genomecov on cluster (jobid: 18, external: Submitted batch job 54467581, jobscript: /pasteur/zeus/projets/p02/Biomics/Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/.snakemake/tmp.uxjuhhze/snakejob.minimap2_and_genomecov.18.sh). For error details see the cluster log and the log files of the involved rule(s). +Error executing rule minimap2_and_genomecov on cluster (jobid: 18, external: Submitted batch job 54467581, jobscript: //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/.snakemake/tmp.uxjuhhze/snakejob.minimap2_and_genomecov.18.sh). For error details see the cluster log and the log files of the involved rule(s). [Sat Jan 6 03:25:26 2024] Finished job 7. 15 of 28 steps (54%) done diff --git a/tests/snaketools/data/slurm_error_no_master/__init__.py b/tests/snaketools/data/slurm_error_no_master/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/tests/snaketools/data/slurm_error_no_master/logs/__init__.py b/tests/snaketools/data/slurm_error_no_master/logs/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/tests/snaketools/data/slurm_error_no_master/logs/minimap2_and_genomecov/minimap2_and_genomecov-sample=H149_P1_200723-slurm-54467146.out b/tests/snaketools/data/slurm_error_no_master/logs/minimap2_and_genomecov/minimap2_and_genomecov-sample=H149_P1_200723-slurm-54467146.out new file mode 100755 index 0000000..d998249 --- /dev/null +++ b/tests/snaketools/data/slurm_error_no_master/logs/minimap2_and_genomecov/minimap2_and_genomecov-sample=H149_P1_200723-slurm-54467146.out @@ -0,0 +1,46 @@ +Building DAG of jobs... +Using shell: /usr/bin/bash +Provided cores: 4 +Rules claiming more threads will be scaled down. +Provided resources: mem_mb=4000, mem_mib=6612, disk_mb=6933, disk_mib=6612 +Conda environments: ignored +Select jobs to execute... + +[Sat Jan 6 02:46:49 2024] +rule minimap2_and_genomecov: + input: /Data/2023/DryLab/B17420/20230720_1448_MN18671_FAV94301_57a76bad/nanomerge/B17420/H149_P1_200723.fastq.gz, H149_P1_200723/sorted_contigs/H149_P1_200723.fasta + output: H149_P1_200723/minimap2/H149_P1_200723.bam, H149_P1_200723/minimap2/H149_P1_200723.bed + jobid: 0 + reason: Forced execution + wildcards: sample=H149_P1_200723 + threads: 4 + resources: mem_mb=4000, mem_mib=3815, disk_mb=6933, disk_mib=6612, tmpdir=/local/scratch/tmp, partition=common, qos=normal, mem=4G, gres= + + + minimap2 -t 4 -ax map-ont H149_P1_200723/sorted_contigs/H149_P1_200723.fasta /Data/2023/DryLab/B17420/20230720_1448_MN18671_FAV94301_57a76bad/nanomerge/B17420/H149_P1_200723.fastq.gz | samtools sort -@ $((4 - 1)) -o H149_P1_200723/minimap2/H149_P1_200723.bam && samtools index H149_P1_200723/minimap2/H149_P1_200723.bam && samtools depth -aa H149_P1_200723/minimap2/H149_P1_200723.bam > H149_P1_200723/minimap2/H149_P1_200723.bed + +Activating singularity image //resources/condas/sequana_0.16/apptainers/c3e4a8244ce7b65fa873ebda134fea7f.simg +[M::mm_idx_gen::1.104*0.97] collected minimizers +[M::mm_idx_gen::1.497*1.44] sorted minimizers +[M::main::1.497*1.44] loaded/built the index for 103 target sequence(s) +[M::mm_mapopt_update::1.571*1.42] mid_occ = 36 +[M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 103 +[M::mm_idx_stat::1.623*1.41] distinct minimizers: 5111506 (89.06% are singletons); average occurrences: 1.235; average spacing: 5.336; total length: 33694841 +/bin/bash: line 2: 3479709 Killed minimap2 -t 4 -ax map-ont H149_P1_200723/sorted_contigs/H149_P1_200723.fasta /Data/2023/DryLab/B17420/20230720_1448_MN18671_FAV94301_57a76bad/nanomerge/B17420/H149_P1_200723.fastq.gz + 3479710 Done | samtools sort -@ $((4 - 1)) -o H149_P1_200723/minimap2/H149_P1_200723.bam +[Sat Jan 6 02:47:31 2024] +Error in rule minimap2_and_genomecov: + jobid: 0 + input: /Data/2023/DryLab/B17420/20230720_1448_MN18671_FAV94301_57a76bad/nanomerge/B17420/H149_P1_200723.fastq.gz, H149_P1_200723/sorted_contigs/H149_P1_200723.fasta + output: H149_P1_200723/minimap2/H149_P1_200723.bam, H149_P1_200723/minimap2/H149_P1_200723.bed + shell: + + minimap2 -t 4 -ax map-ont H149_P1_200723/sorted_contigs/H149_P1_200723.fasta /Data/2023/DryLab/B17420/20230720_1448_MN18671_FAV94301_57a76bad/nanomerge/B17420/H149_P1_200723.fastq.gz | samtools sort -@ $((4 - 1)) -o H149_P1_200723/minimap2/H149_P1_200723.bam && samtools index H149_P1_200723/minimap2/H149_P1_200723.bam && samtools depth -aa H149_P1_200723/minimap2/H149_P1_200723.bam > H149_P1_200723/minimap2/H149_P1_200723.bed + + (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!) + +Removing output files of failed job minimap2_and_genomecov since they might be corrupted: +H149_P1_200723/minimap2/H149_P1_200723.bam +Shutting down, this might take some time. +Exiting because a job execution failed. Look above for error message +slurmstepd: error: Detected 1 oom_kill event in StepId=54467146.batch. Some of the step tasks have been OOM Killed. diff --git a/tests/snaketools/data/slurm_error_no_master/logs/minimap2_and_genomecov/minimap2_and_genomecov-sample=H149_P1_200723-slurm-54474584.out b/tests/snaketools/data/slurm_error_no_master/logs/minimap2_and_genomecov/minimap2_and_genomecov-sample=H149_P1_200723-slurm-54474584.out new file mode 100755 index 0000000..0f2f58c --- /dev/null +++ b/tests/snaketools/data/slurm_error_no_master/logs/minimap2_and_genomecov/minimap2_and_genomecov-sample=H149_P1_200723-slurm-54474584.out @@ -0,0 +1,43 @@ +Building DAG of jobs... +Using shell: /usr/bin/bash +Provided cores: 4 +Rules claiming more threads will be scaled down. +Provided resources: mem_mb=16000, mem_mib=6612, disk_mb=6933, disk_mib=6612 +Conda environments: ignored +Select jobs to execute... + +[Sat Jan 6 08:12:45 2024] +rule minimap2_and_genomecov: + input: /Data/2023/DryLab/B17420/20230720_1448_MN18671_FAV94301_57a76bad/nanomerge/B17420/H149_P1_200723.fastq.gz, H149_P1_200723/sorted_contigs/H149_P1_200723.fasta + output: H149_P1_200723/minimap2/H149_P1_200723.bam, H149_P1_200723/minimap2/H149_P1_200723.bed + jobid: 0 + reason: Forced execution + wildcards: sample=H149_P1_200723 + threads: 4 + resources: mem_mb=16000, mem_mib=6612, disk_mb=6933, disk_mib=6612, tmpdir=/local/scratch/tmp, partition=common, qos=normal, mem=16G, gres= + + + minimap2 -t 4 -ax map-ont H149_P1_200723/sorted_contigs/H149_P1_200723.fasta /Data/2023/DryLab/B17420/20230720_1448_MN18671_FAV94301_57a76bad/nanomerge/B17420/H149_P1_200723.fastq.gz | samtools sort -@ $((4 - 1)) -o H149_P1_200723/minimap2/H149_P1_200723.bam && samtools index H149_P1_200723/minimap2/H149_P1_200723.bam && samtools depth -aa H149_P1_200723/minimap2/H149_P1_200723.bam > H149_P1_200723/minimap2/H149_P1_200723.bed + +Activating singularity image //resources/condas/sequana_0.16/apptainers/c3e4a8244ce7b65fa873ebda134fea7f.simg +[M::mm_idx_gen::1.466*0.80] collected minimizers +[M::mm_idx_gen::1.713*1.17] sorted minimizers +[M::main::1.713*1.17] loaded/built the index for 103 target sequence(s) +[M::mm_mapopt_update::1.784*1.16] mid_occ = 36 +[M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 103 +[M::mm_idx_stat::1.833*1.16] distinct minimizers: 5111506 (89.06% are singletons); average occurrences: 1.235; average spacing: 5.336; total length: 33694841 +[M::worker_pipeline::116.471*3.72] mapped 65230 sequences +[M::worker_pipeline::220.260*3.82] mapped 66497 sequences +[M::worker_pipeline::370.054*3.72] mapped 65819 sequences +[M::worker_pipeline::437.724*3.75] mapped 66385 sequences +[M::worker_pipeline::541.548*3.79] mapped 65589 sequences +[M::worker_pipeline::693.729*3.74] mapped 66649 sequences +[M::worker_pipeline::759.447*3.76] mapped 66000 sequences +[M::worker_pipeline::830.013*3.77] mapped 54006 sequences +[M::main] Version: 2.24-r1122 +[M::main] CMD: minimap2 -t 4 -ax map-ont H149_P1_200723/sorted_contigs/H149_P1_200723.fasta /Data/2023/DryLab/B17420/20230720_1448_MN18671_FAV94301_57a76bad/nanomerge/B17420/H149_P1_200723.fastq.gz +[M::main] Real time: 830.103 sec; CPU: 3128.051 sec; Peak RSS: 4.470 GB +[bam_sort_core] merging from 2 files and 3 in-memory blocks... +[Sat Jan 6 08:30:29 2024] +Finished job 0. +1 of 1 steps (100%) done diff --git a/tests/snaketools/data/slurm_error_no_master/logs/minimap2_and_genomecov/minimap2_and_genomecov-sample=H149_P1_200723_1000-slurm-54467581.out b/tests/snaketools/data/slurm_error_no_master/logs/minimap2_and_genomecov/minimap2_and_genomecov-sample=H149_P1_200723_1000-slurm-54467581.out new file mode 100755 index 0000000..dce6210 --- /dev/null +++ b/tests/snaketools/data/slurm_error_no_master/logs/minimap2_and_genomecov/minimap2_and_genomecov-sample=H149_P1_200723_1000-slurm-54467581.out @@ -0,0 +1,48 @@ +Building DAG of jobs... +Using shell: /usr/bin/bash +Provided cores: 4 +Rules claiming more threads will be scaled down. +Provided resources: mem_mb=4000, mem_mib=6507, disk_mb=6823, disk_mib=6507 +Conda environments: ignored +Select jobs to execute... + +[Sat Jan 6 02:48:15 2024] +rule minimap2_and_genomecov: + input: //Drylab/InProgress/B17420/data/filters/H149_P1_200723_1000.fastq.gz, H149_P1_200723_1000/sorted_contigs/H149_P1_200723_1000.fasta + output: H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bam, H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bed + jobid: 0 + reason: Forced execution + wildcards: sample=H149_P1_200723_1000 + threads: 4 + resources: mem_mb=4000, mem_mib=3815, disk_mb=6823, disk_mib=6507, tmpdir=/local/scratch/tmp, partition=common, qos=normal, mem=4G, gres= + + + minimap2 -t 4 -ax map-ont H149_P1_200723_1000/sorted_contigs/H149_P1_200723_1000.fasta //Drylab/InProgress/B17420/data/filters/H149_P1_200723_1000.fastq.gz | samtools sort -@ $((4 - 1)) -o H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bam && samtools index H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bam && samtools depth -aa H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bam > H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bed + +Activating singularity image //resources/condas/sequana_0.16/apptainers/c3e4a8244ce7b65fa873ebda134fea7f.simg +[M::mm_idx_gen::1.158*0.99] collected minimizers +[M::mm_idx_gen::1.469*1.48] sorted minimizers +[M::main::1.469*1.48] loaded/built the index for 98 target sequence(s) +[M::mm_mapopt_update::1.541*1.46] mid_occ = 37 +[M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 98 +[M::mm_idx_stat::1.590*1.45] distinct minimizers: 5111798 (89.08% are singletons); average occurrences: 1.236; average spacing: 5.337; total length: 33703652 +[M::worker_pipeline::123.711*3.74] mapped 54811 sequences +[bam_sort_core] merging from 0 files and 3 in-memory blocks... +/bin/bash: line 2: 896648 Killed minimap2 -t 4 -ax map-ont H149_P1_200723_1000/sorted_contigs/H149_P1_200723_1000.fasta //Drylab/InProgress/B17420/data/filters/H149_P1_200723_1000.fastq.gz + 896649 Done | samtools sort -@ $((4 - 1)) -o H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bam +[Sat Jan 6 02:50:51 2024] +Error in rule minimap2_and_genomecov: + jobid: 0 + input: //Drylab/InProgress/B17420/data/filters/H149_P1_200723_1000.fastq.gz, H149_P1_200723_1000/sorted_contigs/H149_P1_200723_1000.fasta + output: H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bam, H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bed + shell: + + minimap2 -t 4 -ax map-ont H149_P1_200723_1000/sorted_contigs/H149_P1_200723_1000.fasta //Drylab/InProgress/B17420/data/filters/H149_P1_200723_1000.fastq.gz | samtools sort -@ $((4 - 1)) -o H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bam && samtools index H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bam && samtools depth -aa H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bam > H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bed + + (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!) + +Removing output files of failed job minimap2_and_genomecov since they might be corrupted: +H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bam +Shutting down, this might take some time. +Exiting because a job execution failed. Look above for error message +slurmstepd: error: Detected 1 oom_kill event in StepId=54467581.batch. Some of the step tasks have been OOM Killed. diff --git a/tests/snaketools/data/slurm_error_no_master/logs/minimap2_and_genomecov/minimap2_and_genomecov-sample=H149_P1_200723_1000-slurm-54474583.out b/tests/snaketools/data/slurm_error_no_master/logs/minimap2_and_genomecov/minimap2_and_genomecov-sample=H149_P1_200723_1000-slurm-54474583.out new file mode 100755 index 0000000..2579b27 --- /dev/null +++ b/tests/snaketools/data/slurm_error_no_master/logs/minimap2_and_genomecov/minimap2_and_genomecov-sample=H149_P1_200723_1000-slurm-54474583.out @@ -0,0 +1,43 @@ +Building DAG of jobs... +Using shell: /usr/bin/bash +Provided cores: 4 +Rules claiming more threads will be scaled down. +Provided resources: mem_mb=16000, mem_mib=6507, disk_mb=6823, disk_mib=6507 +Conda environments: ignored +Select jobs to execute... + +[Sat Jan 6 08:12:32 2024] +rule minimap2_and_genomecov: + input: //Drylab/InProgress/B17420/data/filters/H149_P1_200723_1000.fastq.gz, H149_P1_200723_1000/sorted_contigs/H149_P1_200723_1000.fasta + output: H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bam, H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bed + jobid: 0 + reason: Forced execution + wildcards: sample=H149_P1_200723_1000 + threads: 4 + resources: mem_mb=16000, mem_mib=6507, disk_mb=6823, disk_mib=6507, tmpdir=/local/scratch/tmp, partition=common, qos=normal, mem=16G, gres= + + + minimap2 -t 4 -ax map-ont H149_P1_200723_1000/sorted_contigs/H149_P1_200723_1000.fasta //Drylab/InProgress/B17420/data/filters/H149_P1_200723_1000.fastq.gz | samtools sort -@ $((4 - 1)) -o H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bam && samtools index H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bam && samtools depth -aa H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bam > H149_P1_200723_1000/minimap2/H149_P1_200723_1000.bed + +Activating singularity image //resources/condas/sequana_0.16/apptainers/c3e4a8244ce7b65fa873ebda134fea7f.simg +[M::mm_idx_gen::1.537*0.72] collected minimizers +[M::mm_idx_gen::1.792*1.09] sorted minimizers +[M::main::1.792*1.09] loaded/built the index for 98 target sequence(s) +[M::mm_mapopt_update::1.863*1.09] mid_occ = 37 +[M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 98 +[M::mm_idx_stat::1.913*1.08] distinct minimizers: 5111798 (89.08% are singletons); average occurrences: 1.236; average spacing: 5.337; total length: 33703652 +[M::worker_pipeline::117.688*3.73] mapped 54811 sequences +[M::worker_pipeline::221.966*3.83] mapped 55404 sequences +[M::worker_pipeline::371.283*3.73] mapped 55047 sequences +[M::worker_pipeline::442.477*3.76] mapped 55414 sequences +[M::worker_pipeline::546.451*3.80] mapped 55411 sequences +[M::worker_pipeline::692.858*3.75] mapped 54689 sequences +[M::worker_pipeline::766.288*3.77] mapped 54700 sequences +[M::worker_pipeline::826.454*3.78] mapped 33228 sequences +[M::main] Version: 2.24-r1122 +[M::main] CMD: minimap2 -t 4 -ax map-ont H149_P1_200723_1000/sorted_contigs/H149_P1_200723_1000.fasta //Drylab/InProgress/B17420/data/filters/H149_P1_200723_1000.fastq.gz +[M::main] Real time: 826.537 sec; CPU: 3122.557 sec; Peak RSS: 4.578 GB +[bam_sort_core] merging from 2 files and 3 in-memory blocks... +[Sat Jan 6 08:30:12 2024] +Finished job 0. +1 of 1 steps (100%) done diff --git a/tests/snaketools/data/slurm_error_no_master/logs/minimap2_and_genomecov/minimap2_and_genomecov-sample=H149_P1_200723_500-slurm-54466792.out b/tests/snaketools/data/slurm_error_no_master/logs/minimap2_and_genomecov/minimap2_and_genomecov-sample=H149_P1_200723_500-slurm-54466792.out new file mode 100755 index 0000000..0c71a69 --- /dev/null +++ b/tests/snaketools/data/slurm_error_no_master/logs/minimap2_and_genomecov/minimap2_and_genomecov-sample=H149_P1_200723_500-slurm-54466792.out @@ -0,0 +1,46 @@ +Building DAG of jobs... +Using shell: /usr/bin/bash +Provided cores: 4 +Rules claiming more threads will be scaled down. +Provided resources: mem_mb=4000, mem_mib=6584, disk_mb=6903, disk_mib=6584 +Conda environments: ignored +Select jobs to execute... + +[Sat Jan 6 02:42:27 2024] +rule minimap2_and_genomecov: + input: //Drylab/InProgress/B17420/data/filters/H149_P1_200723_500.fastq.gz, H149_P1_200723_500/sorted_contigs/H149_P1_200723_500.fasta + output: H149_P1_200723_500/minimap2/H149_P1_200723_500.bam, H149_P1_200723_500/minimap2/H149_P1_200723_500.bed + jobid: 0 + reason: Forced execution + wildcards: sample=H149_P1_200723_500 + threads: 4 + resources: mem_mb=4000, mem_mib=3815, disk_mb=6903, disk_mib=6584, tmpdir=/local/scratch/tmp, partition=common, qos=normal, mem=4G, gres= + + + minimap2 -t 4 -ax map-ont H149_P1_200723_500/sorted_contigs/H149_P1_200723_500.fasta //Drylab/InProgress/B17420/data/filters/H149_P1_200723_500.fastq.gz | samtools sort -@ $((4 - 1)) -o H149_P1_200723_500/minimap2/H149_P1_200723_500.bam && samtools index H149_P1_200723_500/minimap2/H149_P1_200723_500.bam && samtools depth -aa H149_P1_200723_500/minimap2/H149_P1_200723_500.bam > H149_P1_200723_500/minimap2/H149_P1_200723_500.bed + +Activating singularity image //resources/condas/sequana_0.16/apptainers/c3e4a8244ce7b65fa873ebda134fea7f.simg +[M::mm_idx_gen::1.084*0.99] collected minimizers +[M::mm_idx_gen::1.324*1.43] sorted minimizers +[M::main::1.324*1.43] loaded/built the index for 101 target sequence(s) +[M::mm_mapopt_update::1.397*1.41] mid_occ = 37 +[M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 101 +[M::mm_idx_stat::1.445*1.39] distinct minimizers: 5111260 (89.08% are singletons); average occurrences: 1.236; average spacing: 5.337; total length: 33723594 +/bin/bash: line 2: 2229689 Killed minimap2 -t 4 -ax map-ont H149_P1_200723_500/sorted_contigs/H149_P1_200723_500.fasta //Drylab/InProgress/B17420/data/filters/H149_P1_200723_500.fastq.gz + 2229690 Done | samtools sort -@ $((4 - 1)) -o H149_P1_200723_500/minimap2/H149_P1_200723_500.bam +[Sat Jan 6 02:43:32 2024] +Error in rule minimap2_and_genomecov: + jobid: 0 + input: //Drylab/InProgress/B17420/data/filters/H149_P1_200723_500.fastq.gz, H149_P1_200723_500/sorted_contigs/H149_P1_200723_500.fasta + output: H149_P1_200723_500/minimap2/H149_P1_200723_500.bam, H149_P1_200723_500/minimap2/H149_P1_200723_500.bed + shell: + + minimap2 -t 4 -ax map-ont H149_P1_200723_500/sorted_contigs/H149_P1_200723_500.fasta //Drylab/InProgress/B17420/data/filters/H149_P1_200723_500.fastq.gz | samtools sort -@ $((4 - 1)) -o H149_P1_200723_500/minimap2/H149_P1_200723_500.bam && samtools index H149_P1_200723_500/minimap2/H149_P1_200723_500.bam && samtools depth -aa H149_P1_200723_500/minimap2/H149_P1_200723_500.bam > H149_P1_200723_500/minimap2/H149_P1_200723_500.bed + + (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!) + +Removing output files of failed job minimap2_and_genomecov since they might be corrupted: +H149_P1_200723_500/minimap2/H149_P1_200723_500.bam +Shutting down, this might take some time. +Exiting because a job execution failed. Look above for error message +slurmstepd: error: Detected 1 oom_kill event in StepId=54466792.batch. Some of the step tasks have been OOM Killed. diff --git a/tests/snaketools/data/slurm_error_no_master/logs/minimap2_and_genomecov/minimap2_and_genomecov-sample=H149_P1_200723_500-slurm-54474582.out b/tests/snaketools/data/slurm_error_no_master/logs/minimap2_and_genomecov/minimap2_and_genomecov-sample=H149_P1_200723_500-slurm-54474582.out new file mode 100755 index 0000000..4a23726 --- /dev/null +++ b/tests/snaketools/data/slurm_error_no_master/logs/minimap2_and_genomecov/minimap2_and_genomecov-sample=H149_P1_200723_500-slurm-54474582.out @@ -0,0 +1,43 @@ +Building DAG of jobs... +Using shell: /usr/bin/bash +Provided cores: 4 +Rules claiming more threads will be scaled down. +Provided resources: mem_mb=16000, mem_mib=6584, disk_mb=6903, disk_mib=6584 +Conda environments: ignored +Select jobs to execute... + +[Sat Jan 6 08:12:33 2024] +rule minimap2_and_genomecov: + input: //Drylab/InProgress/B17420/data/filters/H149_P1_200723_500.fastq.gz, H149_P1_200723_500/sorted_contigs/H149_P1_200723_500.fasta + output: H149_P1_200723_500/minimap2/H149_P1_200723_500.bam, H149_P1_200723_500/minimap2/H149_P1_200723_500.bed + jobid: 0 + reason: Forced execution + wildcards: sample=H149_P1_200723_500 + threads: 4 + resources: mem_mb=16000, mem_mib=6584, disk_mb=6903, disk_mib=6584, tmpdir=/local/scratch/tmp, partition=common, qos=normal, mem=16G, gres= + + + minimap2 -t 4 -ax map-ont H149_P1_200723_500/sorted_contigs/H149_P1_200723_500.fasta //Drylab/InProgress/B17420/data/filters/H149_P1_200723_500.fastq.gz | samtools sort -@ $((4 - 1)) -o H149_P1_200723_500/minimap2/H149_P1_200723_500.bam && samtools index H149_P1_200723_500/minimap2/H149_P1_200723_500.bam && samtools depth -aa H149_P1_200723_500/minimap2/H149_P1_200723_500.bam > H149_P1_200723_500/minimap2/H149_P1_200723_500.bed + +Activating singularity image //resources/condas/sequana_0.16/apptainers/c3e4a8244ce7b65fa873ebda134fea7f.simg +[M::mm_idx_gen::1.476*0.74] collected minimizers +[M::mm_idx_gen::1.721*1.11] sorted minimizers +[M::main::1.721*1.11] loaded/built the index for 101 target sequence(s) +[M::mm_mapopt_update::1.793*1.11] mid_occ = 37 +[M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 101 +[M::mm_idx_stat::1.842*1.10] distinct minimizers: 5111260 (89.08% are singletons); average occurrences: 1.236; average spacing: 5.337; total length: 33723594 +[M::worker_pipeline::118.919*3.74] mapped 61338 sequences +[M::worker_pipeline::225.491*3.85] mapped 62458 sequences +[M::worker_pipeline::377.367*3.74] mapped 61688 sequences +[M::worker_pipeline::449.779*3.78] mapped 62331 sequences +[M::worker_pipeline::555.436*3.81] mapped 61765 sequences +[M::worker_pipeline::704.893*3.76] mapped 62013 sequences +[M::worker_pipeline::778.489*3.78] mapped 61261 sequences +[M::worker_pipeline::848.696*3.79] mapped 44951 sequences +[M::main] Version: 2.24-r1122 +[M::main] CMD: minimap2 -t 4 -ax map-ont H149_P1_200723_500/sorted_contigs/H149_P1_200723_500.fasta //Drylab/InProgress/B17420/data/filters/H149_P1_200723_500.fastq.gz +[M::main] Real time: 848.775 sec; CPU: 3215.568 sec; Peak RSS: 4.587 GB +[bam_sort_core] merging from 2 files and 3 in-memory blocks... +[Sat Jan 6 08:30:41 2024] +Finished job 0. +1 of 1 steps (100%) done diff --git a/tests/snaketools/data/slurm_error_no_master/logs/quast/quast-sample=H149_P1_200723-slurm-54467145.out b/tests/snaketools/data/slurm_error_no_master/logs/quast/quast-sample=H149_P1_200723-slurm-54467145.out new file mode 100755 index 0000000..f255084 --- /dev/null +++ b/tests/snaketools/data/slurm_error_no_master/logs/quast/quast-sample=H149_P1_200723-slurm-54467145.out @@ -0,0 +1,106 @@ +Building DAG of jobs... +Using shell: /usr/bin/bash +Provided cores: 4 +Rules claiming more threads will be scaled down. +Provided resources: mem_mb=4000, mem_mib=6612, disk_mb=6933, disk_mib=6612 +Conda environments: ignored +Select jobs to execute... + +[Sat Jan 6 02:46:31 2024] +rule quast: + input: /Data/2023/DryLab/B17420/20230720_1448_MN18671_FAV94301_57a76bad/nanomerge/B17420/H149_P1_200723.fastq.gz, H149_P1_200723/sorted_contigs/H149_P1_200723.fasta + output: H149_P1_200723/quast/quast.done + jobid: 0 + reason: Forced execution + wildcards: sample=H149_P1_200723 + threads: 4 + resources: mem_mb=4000, mem_mib=3815, disk_mb=6933, disk_mib=6612, tmpdir=/local/scratch/tmp, partition=common, qos=normal, mem=4G, gres= + + + quast.py -e --rna-finding -t 4 H149_P1_200723/sorted_contigs/H149_P1_200723.fasta --nanopore /Data/2023/DryLab/B17420/20230720_1448_MN18671_FAV94301_57a76bad/nanomerge/B17420/H149_P1_200723.fastq.gz -o H149_P1_200723/quast && touch H149_P1_200723/quast/quast.done + +Activating singularity image //resources/condas/sequana_0.16/apptainers/847f147b6a61a6684f5106b4bc25a0c7.simg +WARNING: Python locale settings can't be changed +/usr/local/bin/quast.py -e --rna-finding -t 4 H149_P1_200723/sorted_contigs/H149_P1_200723.fasta --nanopore /Data/2023/DryLab/B17420/20230720_1448_MN18671_FAV94301_57a76bad/nanomerge/B17420/H149_P1_200723.fastq.gz -o H149_P1_200723/quast + +Version: 5.2.0 + +System information: + OS: Linux-4.18.0-477.27.1.el8_8.x86_64-x86_64-with-glibc2.31 (linux_64) + Python version: 3.10.11 + CPUs number: 96 + +Started: 2024-01-06 02:46:32 + +Logging to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723/quast/quast.log + +CWD: //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate +Main parameters: + MODE: default, threads: 4, eukaryotic: true, min contig length: 500, min alignment length: 65, \ + min alignment IDY: 95.0, ambiguity: one, min local misassembly length: 200, min extensive misassembly length: 1000 + +Contigs: + Pre-processing... + H149_P1_200723/sorted_contigs/H149_P1_200723.fasta ==> H149_P1_200723 + +2024-01-06 02:46:34 +Running Reads analyzer... + Logging to files //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723/quast/reads_stats/reads_stats.log and //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723/quast/reads_stats/reads_stats.err... + Pre-processing reads... + Running BWA... + Done. + Sorting SAM-file... + Analysis is finished. + Creating total report... + saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723/quast/reads_stats/reads_report.txt, reads_report.tsv, and reads_report.tex +Done. + +2024-01-06 03:29:17 +Running Basic statistics processor... + Contig files: + H149_P1_200723 + Calculating N50 and L50... + H149_P1_200723, N50 = 787511, L50 = 13, auN = 1015747.0, Total length = 33694841, GC % = 59.64, # N's per 100 kbp = 0.00 + Drawing Nx plot... + saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723/quast/basic_stats/Nx_plot.pdf + Drawing cumulative plot... + saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723/quast/basic_stats/cumulative_plot.pdf + Drawing GC content plot... + saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723/quast/basic_stats/GC_content_plot.pdf + Drawing H149_P1_200723 GC content plot... + saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723/quast/basic_stats/H149_P1_200723_GC_content_plot.pdf +Done. + +NOTICE: Genes are not predicted by default. Use --gene-finding or --glimmer option to enable it. + +2024-01-06 03:29:20 +Running Barrnap... +Logging to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723/quast/predicted_genes/barrnap.log... + Ribosomal RNA genes = 15 + Predicted genes (GFF): //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723/quast/predicted_genes/H149_P1_200723.rna.gff +Done. + +2024-01-06 03:29:26 +Creating large visual summaries... +This may take a while: press Ctrl-C to skip this step.. + 1 of 2: Creating PDF with all tables and plots... + 2 of 2: Creating Icarus viewers... +Done + +2024-01-06 03:29:27 +RESULTS: + Text versions of total report are saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723/quast/report.txt, report.tsv, and report.tex + Text versions of transposed total report are saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723/quast/transposed_report.txt, transposed_report.tsv, and transposed_report.tex + HTML version (interactive tables and plots) is saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723/quast/report.html + PDF version (tables and plots) is saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723/quast/report.pdf + Icarus (contig browser) is saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723/quast/icarus.html + eog is saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723/quast/quast.log + +Finished: 2024-01-06 03:29:27 +Elapsed time: 0:42:54.840139 +NOTICEs: 1; WARNINGs: 1; non-fatal ERRORs: 0 + +Thank you for using QUAST! +[Sat Jan 6 03:29:29 2024] +Finished job 0. +1 of 1 steps (100%) done diff --git a/tests/snaketools/data/slurm_error_no_master/logs/quast/quast-sample=H149_P1_200723_1000-slurm-54467578.out b/tests/snaketools/data/slurm_error_no_master/logs/quast/quast-sample=H149_P1_200723_1000-slurm-54467578.out new file mode 100755 index 0000000..884dc32 --- /dev/null +++ b/tests/snaketools/data/slurm_error_no_master/logs/quast/quast-sample=H149_P1_200723_1000-slurm-54467578.out @@ -0,0 +1,106 @@ +Building DAG of jobs... +Using shell: /usr/bin/bash +Provided cores: 4 +Rules claiming more threads will be scaled down. +Provided resources: mem_mb=4000, mem_mib=6507, disk_mb=6823, disk_mib=6507 +Conda environments: ignored +Select jobs to execute... + +[Sat Jan 6 02:48:07 2024] +rule quast: + input: //Drylab/InProgress/B17420/data/filters/H149_P1_200723_1000.fastq.gz, H149_P1_200723_1000/sorted_contigs/H149_P1_200723_1000.fasta + output: H149_P1_200723_1000/quast/quast.done + jobid: 0 + reason: Forced execution + wildcards: sample=H149_P1_200723_1000 + threads: 4 + resources: mem_mb=4000, mem_mib=3815, disk_mb=6823, disk_mib=6507, tmpdir=/local/scratch/tmp, partition=common, qos=normal, mem=4G, gres= + + + quast.py -e --rna-finding -t 4 H149_P1_200723_1000/sorted_contigs/H149_P1_200723_1000.fasta --nanopore //Drylab/InProgress/B17420/data/filters/H149_P1_200723_1000.fastq.gz -o H149_P1_200723_1000/quast && touch H149_P1_200723_1000/quast/quast.done + +Activating singularity image //resources/condas/sequana_0.16/apptainers/847f147b6a61a6684f5106b4bc25a0c7.simg +WARNING: Python locale settings can't be changed +/usr/local/bin/quast.py -e --rna-finding -t 4 H149_P1_200723_1000/sorted_contigs/H149_P1_200723_1000.fasta --nanopore //Drylab/InProgress/B17420/data/filters/H149_P1_200723_1000.fastq.gz -o H149_P1_200723_1000/quast + +Version: 5.2.0 + +System information: + OS: Linux-4.18.0-477.27.1.el8_8.x86_64-x86_64-with-glibc2.31 (linux_64) + Python version: 3.10.11 + CPUs number: 96 + +Started: 2024-01-06 02:48:08 + +Logging to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_1000/quast/quast.log + +CWD: //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate +Main parameters: + MODE: default, threads: 4, eukaryotic: true, min contig length: 500, min alignment length: 65, \ + min alignment IDY: 95.0, ambiguity: one, min local misassembly length: 200, min extensive misassembly length: 1000 + +Contigs: + Pre-processing... + H149_P1_200723_1000/sorted_contigs/H149_P1_200723_1000.fasta ==> H149_P1_200723_1000 + +2024-01-06 02:48:10 +Running Reads analyzer... + Logging to files //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_1000/quast/reads_stats/reads_stats.log and //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_1000/quast/reads_stats/reads_stats.err... + Pre-processing reads... + Running BWA... + Done. + Sorting SAM-file... + Analysis is finished. + Creating total report... + saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_1000/quast/reads_stats/reads_report.txt, reads_report.tsv, and reads_report.tex +Done. + +2024-01-06 03:29:26 +Running Basic statistics processor... + Contig files: + H149_P1_200723_1000 + Calculating N50 and L50... + H149_P1_200723_1000, N50 = 787408, L50 = 12, auN = 1066532.2, Total length = 33703652, GC % = 59.64, # N's per 100 kbp = 0.00 + Drawing Nx plot... + saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_1000/quast/basic_stats/Nx_plot.pdf + Drawing cumulative plot... + saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_1000/quast/basic_stats/cumulative_plot.pdf + Drawing GC content plot... + saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_1000/quast/basic_stats/GC_content_plot.pdf + Drawing H149_P1_200723_1000 GC content plot... + saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_1000/quast/basic_stats/H149_P1_200723_1000_GC_content_plot.pdf +Done. + +NOTICE: Genes are not predicted by default. Use --gene-finding or --glimmer option to enable it. + +2024-01-06 03:29:28 +Running Barrnap... +Logging to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_1000/quast/predicted_genes/barrnap.log... + Ribosomal RNA genes = 15 + Predicted genes (GFF): //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_1000/quast/predicted_genes/H149_P1_200723_1000.rna.gff +Done. + +2024-01-06 03:29:34 +Creating large visual summaries... +This may take a while: press Ctrl-C to skip this step.. + 1 of 2: Creating PDF with all tables and plots... + 2 of 2: Creating Icarus viewers... +Done + +2024-01-06 03:29:35 +RESULTS: + Text versions of total report are saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_1000/quast/report.txt, report.tsv, and report.tex + Text versions of transposed total report are saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_1000/quast/transposed_report.txt, transposed_report.tsv, and transposed_report.tex + HTML version (interactive tables and plots) is saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_1000/quast/report.html + PDF version (tables and plots) is saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_1000/quast/report.pdf + Icarus (contig browser) is saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_1000/quast/icarus.html + Log is saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_1000/quast/quast.log + +Finished: 2024-01-06 03:29:35 +Elapsed time: 0:41:27.307706 +NOTICEs: 1; WARNINGs: 1; non-fatal ERRORs: 0 + +Thank you for using QUAST! +[Sat Jan 6 03:29:36 2024] +Finished job 0. +1 of 1 steps (100%) done diff --git a/tests/snaketools/data/slurm_error_no_master/logs/quast/quast-sample=H149_P1_200723_500-slurm-54466795.out b/tests/snaketools/data/slurm_error_no_master/logs/quast/quast-sample=H149_P1_200723_500-slurm-54466795.out new file mode 100755 index 0000000..4163d91 --- /dev/null +++ b/tests/snaketools/data/slurm_error_no_master/logs/quast/quast-sample=H149_P1_200723_500-slurm-54466795.out @@ -0,0 +1,106 @@ +Building DAG of jobs... +Using shell: /usr/bin/bash +Provided cores: 4 +Rules claiming more threads will be scaled down. +Provided resources: mem_mb=4000, mem_mib=6584, disk_mb=6903, disk_mib=6584 +Conda environments: ignored +Select jobs to execute... + +[Sat Jan 6 02:42:28 2024] +rule quast: + input: //Drylab/InProgress/B17420/data/filters/H149_P1_200723_500.fastq.gz, H149_P1_200723_500/sorted_contigs/H149_P1_200723_500.fasta + output: H149_P1_200723_500/quast/quast.done + jobid: 0 + reason: Forced execution + wildcards: sample=H149_P1_200723_500 + threads: 4 + resources: mem_mb=4000, mem_mib=3815, disk_mb=6903, disk_mib=6584, tmpdir=/local/scratch/tmp, partition=common, qos=normal, mem=4G, gres= + + + quast.py -e --rna-finding -t 4 H149_P1_200723_500/sorted_contigs/H149_P1_200723_500.fasta --nanopore //Drylab/InProgress/B17420/data/filters/H149_P1_200723_500.fastq.gz -o H149_P1_200723_500/quast && touch H149_P1_200723_500/quast/quast.done + +Activating singularity image //resources/condas/sequana_0.16/apptainers/847f147b6a61a6684f5106b4bc25a0c7.simg +WARNING: Python locale settings can't be changed +/usr/local/bin/quast.py -e --rna-finding -t 4 H149_P1_200723_500/sorted_contigs/H149_P1_200723_500.fasta --nanopore //Drylab/InProgress/B17420/data/filters/H149_P1_200723_500.fastq.gz -o H149_P1_200723_500/quast + +Version: 5.2.0 + +System information: + OS: Linux-4.18.0-477.27.1.el8_8.x86_64-x86_64-with-glibc2.31 (linux_64) + Python version: 3.10.11 + CPUs number: 96 + +Started: 2024-01-06 02:42:30 + +Logging to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_500/quast/quast.log + +CWD: //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate +Main parameters: + MODE: default, threads: 4, eukaryotic: true, min contig length: 500, min alignment length: 65, \ + min alignment IDY: 95.0, ambiguity: one, min local misassembly length: 200, min extensive misassembly length: 1000 + +Contigs: + Pre-processing... + H149_P1_200723_500/sorted_contigs/H149_P1_200723_500.fasta ==> H149_P1_200723_500 + +2024-01-06 02:42:32 +Running Reads analyzer... + Logging to files //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_500/quast/reads_stats/reads_stats.log and //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_500/quast/reads_stats/reads_stats.err... + Pre-processing reads... + Running BWA... + Done. + Sorting SAM-file... + Analysis is finished. + Creating total report... + saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_500/quast/reads_stats/reads_report.txt, reads_report.tsv, and reads_report.tex +Done. + +2024-01-06 03:25:10 +Running Basic statistics processor... + Contig files: + H149_P1_200723_500 + Calculating N50 and L50... + H149_P1_200723_500, N50 = 787466, L50 = 12, auN = 1062323.0, Total length = 33723594, GC % = 59.65, # N's per 100 kbp = 0.00 + Drawing Nx plot... + saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_500/quast/basic_stats/Nx_plot.pdf + Drawing cumulative plot... + saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_500/quast/basic_stats/cumulative_plot.pdf + Drawing GC content plot... + saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_500/quast/basic_stats/GC_content_plot.pdf + Drawing H149_P1_200723_500 GC content plot... + saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_500/quast/basic_stats/H149_P1_200723_500_GC_content_plot.pdf +Done. + +NOTICE: Genes are not predicted by default. Use --gene-finding or --glimmer option to enable it. + +2024-01-06 03:25:13 +Running Barrnap... +Logging to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_500/quast/predicted_genes/barrnap.log... + Ribosomal RNA genes = 15 + Predicted genes (GFF): //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_500/quast/predicted_genes/H149_P1_200723_500.rna.gff +Done. + +2024-01-06 03:25:19 +Creating large visual summaries... +This may take a while: press Ctrl-C to skip this step.. + 1 of 2: Creating PDF with all tables and plots... + 2 of 2: Creating Icarus viewers... +Done + +2024-01-06 03:25:20 +RESULTS: + Text versions of total report are saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_500/quast/report.txt, report.tsv, and report.tex + Text versions of transposed total report are saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_500/quast/transposed_report.txt, transposed_report.tsv, and transposed_report.tex + HTML version (interactive tables and plots) is saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_500/quast/report.html + PDF version (tables and plots) is saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_500/quast/report.pdf + Icarus (contig browser) is saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_500/quast/icarus.html + Log is saved to //Drylab/InProgress/B17420/analysis/lora_flye_filters_with_coverage_flyupdate/H149_P1_200723_500/quast/quast.log + +Finished: 2024-01-06 03:25:20 +Elapsed time: 0:42:50.432404 +NOTICEs: 1; WARNINGs: 1; non-fatal ERRORs: 0 + +Thank you for using QUAST! +[Sat Jan 6 03:25:21 2024] +Finished job 0. +1 of 1 steps (100%) done diff --git a/tests/snaketools/test_slurm.py b/tests/snaketools/test_slurm.py index ba616c1..8f62ad5 100644 --- a/tests/snaketools/test_slurm.py +++ b/tests/snaketools/test_slurm.py @@ -14,3 +14,8 @@ def test_slurm(): print(dj) dj dj._report() + + dj = SlurmParsing(sharedir / "slurm_error_no_master") + print(dj) + dj + dj._report()