QIIME2_DIVERSITY: QIIME2_TREE Not working for me #516
Comments
|
Hi there,
The relevant message is You can skip producing a tree by either not supplying a metadata file (but that will remove all downstream analysis) or by using Let me know how it goes! |
|
It seems that I don't have any "/tmp/qiime2" folder. Is it possible that qiime2 uses a different temp directory than the default one? Regarding this post, I suppose it doesn't use a different one. |
|
Sorry for the late reply. |
|
It appears the same errors as I sent before, nothing different. I was wondering if I need to have "mafft" software installed in my local computer for it to work or will it not make the difference? Thank you |
no that wont help. Did you try out It seems that the tmp dir behavior of QIIME2 was changed in 2022.8 (which is used by ampliseq 2.4.1) compared to 2021.8 (which is used by ampliseq 2.4.0), see https://forum.qiime2.org/t/qiime-2-2022-11-is-now-available/25074#artifact-cachehttpsdevqiime2orglatestapi-referencecache-3. That might cause that trouble, i.e. it would be a pipeline bug. Would you like to try with |
|
I am able to run the workflow completely if I use --skip_alpha_rarefaction --skip_diversity_indices -resume but still I dont get any taxa-bar plot (which I aimed to). I think I can get it using qiime2 as a separate tool taking the output of some of the processes but I still can make the workflow finish either with v 2.4.0 or 2.4.1. It always says that the "debug info" is in a file inside /tmp/ which it doesn't exist. |
|
Thanks for testing!
that is very weird, neither of those skipped steps are required for generating barplots. The barplots should be definitely produced with that settings. Could you share your
just to clarify, you mean you can not make it finish, correct? because if it does finish it should be fine... |
I can not make it finish. Sorry for misleading you... I attach here the .nextflow.log. Regarding the results, all the files I get relating to the barplot are the following: from level 1 to level 9 taxonomic classification (.jsonp and csv files), one folder named "q2templateassets" with some folder related to css, js, img and font files and one folder called "dist" with files relating to license. There isn't any qzv file that I can use to visualize the barplot with qiime tool. Thanks for your help, |
this sounds fine, yes, indeed, there is no qzv file and isnt supposed to be there, please check for the
this is a shame, unfortunately it is very hard to troubleshoot without having a reproducible problem. And all my tests (local, github, AWS) were fine. |
|
Thanks to your help I made the index.html file to work so that's great. Regarding the QIIME2_DIVERSITY:QIIME2_TREE error, I will try to solve it by myself since it is such a specific error that it is quite hard to reproduce. Thank you very much for your help again. |
If you find any solution, please let me know (writing here, in a new issue, nf-core slack, ...), maybe that could improve the pipeline. |
|
@d4straub I'm having a similar issue. I think it may be related to having /data as the temp directory instead of /tmp. I tried mapping /data:/data and /data:/tmp using $SINGULARITY_BIND, but maybe that's not the correct approach. mafft seems to be trying to write to /data in the sing image and it's finding that it's read only. I can send more info in my own ticket when I'm at my PC. I'm just wondering, it's it possible for you to set up a test environment where $TMPDIR=/data And /tmp and /data are both writable and present. Maybe your test environment is already set up like this and my hypothesis is off target. Let me know. What you think. For what it's worth, I ran other nxf pipeline test runs just fine, this is the only one that failed. Thanks! |
|
Thanks for the report, yes, it seems more and more apparent that there is some sort of tmp bug in that process. That requires solving. More info is welcome, but QIIME2 allows setting tmp variables now afaik so that would be a start testing that. |
|
I re-open the issue because it seems important that it wont be forgotten |
|
This actually may be due to the cache dirs they implemented in 2022.8. Some bugs regarding nfs were resolved in 2022.11 e.g. https://forum.qiime2.org/t/multiple-processes-and-qiime-2-2022-8/24401/30 Is it possible to upgrade this workflow to 2022.11? The odd thing is that my One thing I can do is test this mafft alignment outside of singularity, but within a qiime2 environment, to remove that as a factor. I'll have to try that tomorrow morning. I'm still assuming this is some intersection between qiime2 TMPDIR settings, mafft, and singularity, so that will help narrow it down a little. Here is the last error log I have: mafft Read-only file system error``` $ cat /tmp/qiime2-q2cli-err-wshk98r6.log mktemp: failed to create directory via template ‘/data/mafft.XXXXXXXXXX’: Read-only file system mktemp seems to be obsolete. Re-trying without -t mkdir: cannot create directory ‘/data/tmp’: Read-only file system mktemp: failed to create directory via template ‘/data/tmp/mafft.XXXXXXXXXX’: No such file or directory /opt/conda/envs/qiime2-2022.8/bin/mafft: line 1111: /infile: Read-only file system [100%] 2 of 2 ✔ /opt/conda/envs/qiime2-2022.8/bin/mafft: line 1112: /infile: Read-only file system pleID_1a) /opt/conda/envs/qiime2-2022.8/bin/mafft: line 1113: /_addfile: Read-only file system MPLISEQ:AMPLISEQ:DADA2_PREP /opt/conda/envs/qiime2-2022.8/bin/mafft: line 1121: /infile: Read-only file system [100%] 2 of 2 ✔ /opt/conda/envs/qiime2-2022.8/bin/mafft: line 1123: /_aamtx: Read-only file system /opt/conda/envs/qiime2-2022.8/bin/mafft: line 1124: /_subalignmentstable: Read-only file system /opt/conda/envs/qiime2-2022.8/bin/mafft: line 1125: /_guidetree: Read-only file system [100%] 1 of 1 ✔ /opt/conda/envs/qiime2-2022.8/bin/mafft: line 1126: /_codonpos: Read-only file system /opt/conda/envs/qiime2-2022.8/bin/mafft: line 1127: /_codonscore: Read-only file systemMPLISEQ:AMPLISEQ:DADA2_STAT /opt/conda/envs/qiime2-2022.8/bin/mafft: line 1128: /_seedtablefile: Read-only file system /opt/conda/envs/qiime2-2022.8/bin/mafft: line 1129: /_lara.params: Read-only file system /opt/conda/envs/qiime2-2022.8/bin/mafft: line 1130: /pdblist: Read-only file system MPLISEQ:AMPLISEQ:MERGE_STAT /opt/conda/envs/qiime2-2022.8/bin/mafft: line 1131: /ownlist: Read-only file system [100%] 1 of 1 ✔ /opt/conda/envs/qiime2-2022.8/bin/mafft: line 1132: /_externalanchors: Read-only file system grep: /infile: No such file or directory /opt/conda/envs/qiime2-2022.8/bin/mafft: line 1807: [: -gt: unary operator expected [100%] 1 of 1 ✔ grep: /infile: No such file or directory /opt/conda/envs/qiime2-2022.8/bin/mafft: line 1816: [: -eq: unary operator expected [100%] 1 of 1 ✔ /opt/conda/envs/qiime2-2022.8/bin/mafft: line 1823: [: too many arguments mv: cannot stat 'infile': No such file or directory Running external command line application. This may print messages to stdout and/or stderr. /51fcf0] process > NFCORE_AMPLISEQ:AMPLISEQ:DADA2_TAXO The command being run is below. This command cannot be manually re-run as it will depend on temporary files that no longer exist. IES (ASV_tax.rds,addSpecies.fna) - e I Chain I Command: mafft --preservecase --inputorder --thread 2 /tmp/qiime2/davised/data/72176087-15f3-43f1-9192-86fdbb369418/data/dna-sequences.fasta sampleID_2) /48dd5d] process > NFCORE_AMPLISEQ:AMPLISEQ:QIIME2_PRE [ 75%] 3 of 4 e I Chain J Traceback (most recent call last): File "/opt/conda/envs/qiime2-2022.8/lib/python3.8/site-packages/q2cli/commands.py", line 339, in __call__ results = action(**arguments) File "", line 2, in mafft File "/opt/conda/envs/qiime2-2022.8/lib/python3.8/site-packages/qiime2/sdk/action.py", line 234, in bound_callable outputs = self._callable_executor_(scope, callable_args, .qza) [100%] 1 of 1 ✔ File "/opt/conda/envs/qiime2-2022.8/lib/python3.8/site-packages/qiime2/sdk/action.py", line 381, in _callable_executor_ output_views = self._callable(**view_args) File "/opt/conda/envs/qiime2-2022.8/lib/python3.8/site-packages/q2_alignment/_mafft.py", line 128, in mafft return _mafft(sequences_fp, None, n_threads, parttree, False) File "/opt/conda/envs/qiime2-2022.8/lib/python3.8/site-packages/q2_alignment/_mafft.py", line 100, in _mafft run_command(cmd, result_fp) File "/opt/conda/envs/qiime2-2022.8/lib/python3.8/site-packages/q2_alignment/_mafft.py", line 26, in run_command subprocess.run(cmd, stdout=output_f, check=True) File "/opt/conda/envs/qiime2-2022.8/lib/python3.8/subprocess.py", line 516, in run MPLISEQ:AMPLISEQ:MERGE_STAT raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['mafft', '--preservecase', '--inputorder', '--thread', '2', '/tmp/qiime2/davised/data/72176087-15f3-43f1-9192-86fdbb369418/data/dna-s equences.fasta']' returned non-zer o exit status 1. ``` |
|
Excellent find!
Yes, see https://nf-co.re/ampliseq/2.4.1/usage#updating-containers, with a config all QIIME2 processes could recieve an updated container. please let me know whether that works, if yes, I could just update to 2022.11 and make a new release ;) |
|
Ok, did the update and have a similar issue. I downloaded the img from quay.io and set my config like this (example above is missing Still getting the issue. I'm going to build a conda env with the qiime2 version 2022.11 to see if that still has the same issue with a test qiime2 that runs this command qiime alignment mafft \
--i-sequences filtered-sequences.qza \
--o-alignment aligned-rep-seqs.qza \
--p-n-threads 2qiime2 2022.11 mafft error$ cat /tmp/qiime2-q2cli-err-9klusvt5.log
mktemp: failed to create directory via template ‘/data/mafft.XXXXXXXXXX’: Read-only file system
mktemp seems to be obsolete. Re-trying without -t
mkdir: cannot create directory ‘/data/tmp’: Read-only file system
mktemp: failed to create directory via template ‘/data/tmp/mafft.XXXXXXXXXX’: No such file or directory
/opt/conda/envs/qiime2-2022.11/bin/mafft: line 1112: /infile: Read-only file system
/opt/conda/envs/qiime2-2022.11/bin/mafft: line 1113: /infile: Read-only file system
/opt/conda/envs/qiime2-2022.11/bin/mafft: line 1114: /_addfile: Read-only file system
/opt/conda/envs/qiime2-2022.11/bin/mafft: line 1122: /infile: Read-only file system
/opt/conda/envs/qiime2-2022.11/bin/mafft: line 1124: /_aamtx: Read-only file system
/opt/conda/envs/qiime2-2022.11/bin/mafft: line 1125: /_subalignmentstable: Read-only file system
/opt/conda/envs/qiime2-2022.11/bin/mafft: line 1126: /_guidetree: Read-only file system
/opt/conda/envs/qiime2-2022.11/bin/mafft: line 1127: /_codonpos: Read-only file system
/opt/conda/envs/qiime2-2022.11/bin/mafft: line 1128: /_codonscore: Read-only file system
/opt/conda/envs/qiime2-2022.11/bin/mafft: line 1129: /_seedtablefile: Read-only file system
/opt/conda/envs/qiime2-2022.11/bin/mafft: line 1130: /_lara.params: Read-only file system
/opt/conda/envs/qiime2-2022.11/bin/mafft: line 1131: /pdblist: Read-only file system
/opt/conda/envs/qiime2-2022.11/bin/mafft: line 1132: /ownlist: Read-only file system
/opt/conda/envs/qiime2-2022.11/bin/mafft: line 1133: /_externalanchors: Read-only file system
grep: /infile: No such file or directory
/opt/conda/envs/qiime2-2022.11/bin/mafft: line 1808: [: -gt: unary operator expected
grep: /infile: No such file or directory
/opt/conda/envs/qiime2-2022.11/bin/mafft: line 1817: [: -eq: unary operator expected
/opt/conda/envs/qiime2-2022.11/bin/mafft: line 1824: [: too many arguments
mv: cannot stat 'infile': No such file or directory
Running external command line application. This may print messages to stdout and/or stderr.
The command being run is below. This command cannot be manually re-run as it will depend on temporary files that no longer exist.
Command: mafft --preservecase --inputorder --thread 2 /tmp/qiime2/davised/data/2e85b3fb-5e55-480f-a8f4-d8e887b54104/data/dna-sequences.fasta
Traceback (most recent call last):
File "/opt/conda/envs/qiime2-2022.11/lib/python3.8/site-packages/q2cli/commands.py", line 352, in __call__
results = action(**arguments)
File "<decorator-gen-44>", line 2, in mafft
File "/opt/conda/envs/qiime2-2022.11/lib/python3.8/site-packages/qiime2/sdk/action.py", line 234, in bound_callable
outputs = self._callable_executor_(scope, callable_args,
File "/opt/conda/envs/qiime2-2022.11/lib/python3.8/site-packages/qiime2/sdk/action.py", line 381, in _callable_executor_
output_views = self._callable(**view_args)
File "/opt/conda/envs/qiime2-2022.11/lib/python3.8/site-packages/q2_alignment/_mafft.py", line 128, in mafft
return _mafft(sequences_fp, None, n_threads, parttree, False)
File "/opt/conda/envs/qiime2-2022.11/lib/python3.8/site-packages/q2_alignment/_mafft.py", line 100, in _mafft
run_command(cmd, result_fp)
File "/opt/conda/envs/qiime2-2022.11/lib/python3.8/site-packages/q2_alignment/_mafft.py", line 26, in run_command
subprocess.run(cmd, stdout=output_f, check=True)
File "/opt/conda/envs/qiime2-2022.11/lib/python3.8/subprocess.py", line 516, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['mafft', '--preservecase', '--inputorder', '--thread', '2', '/tmp/qiime2/davised/data/2e85b3fb-5e55-480f-a8f4-d8e887b54104/data/dna-sequences.fasta']' returned non-zero exit status 1. |
|
Ok, as a first positive test, I ran mafft from the singularity image and was able to get it to run: singularity exec -B "$PWD" /local/cluster/singularity-images/nxf/quay.io-qiime2-core-2022.11.img /opt/conda/envs/qiime2-2022.11/bin/mafft --preservecase --inputorder --thread 2 ex_in.fa > ex_in.alnWorking through the conda install of qiime2 and I'll report back. |
|
Ok, while I was waiting for the conda env to install, I realized something. [Linux@olympus1 nextflow]$ /bin/singularity --version
singularity-ce version 3.10.5-1.el7
[Linux@olympus1 nextflow]$ singularity --version
singularity-ce version 3.8.3I have two singularities installed. I ran the above command with the singularity 3.8.3 that was successful, and it fails with the 3.10.5-1.el7 from the epel. I'm thinking this might be a singularity configuration issue, then. Looking into this more. |
|
Ok very strange. I have two terminals open on the same machine, one is failing with both versions now, and one is successful with both versions. So maybe it isn't a config issue? This is odd. Edit - Alright, sorry for the barrage. WITHOUT Version difference was not the cause. |
|
I installed a conda env with the 2022.11 version and downloaded the rep-seqs-dada2.qza from the moving pictures tutorial and ran So it does seem to be an interaction between the q2 install inside the singularity container, mafft, and tmpdir settings. |
|
Ok here is the minimal test set to show what's happening outside of nextflow. Seems like maybe there's an issue with the [Linux@olympus1 nextflow]$ SINGULARITY_BIND=/data:/tmp,/data:/data /bin/singularity exec -B "$PWD" /local/cluster/singularity-images/nxf/quay.io-qiime2-core-2022.11.img qiime alignment mafft --i-sequences rep-seqs.qza --o-alignment aligned-rep-seqs.qza --p-n-threads 2
Matplotlib created a temporary config/cache directory at /data/matplotlib-h5ojlfgl because the default path (/home/qiime2/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.
Saved FeatureData[AlignedSequence] to: aligned-rep-seqs.qza
[Linux@olympus1 nextflow]$ /bin/singularity exec -B "$PWD" /local/cluster/singularity-images/nxf/quay.io-qiime2-core-2022.11.img qiime alignment mafft --i-
sequences rep-seqs.qza --o-alignment aligned-rep-seqs.qza --p-n-threads 2
Matplotlib created a temporary config/cache directory at /tmp/matplotlib-1fx3gk03 because the default path (/home/qiime2/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.
Plugin error from alignment:
Command '['mafft', '--preservecase', '--inputorder', '--thread', '2', '/tmp/qiime2/davised/data/bc72c816-a0fe-4885-90b7-ee91107bfeee/data/dna-sequences.fasta']' returned non-zero exit status 1.
Debug info has been saved to /tmp/qiime2-q2cli-err-_273ajap.log |
|
Yes! -[nf-core/ampliseq] Pipeline completed successfully-
Completed at: 31-Jan-2023 11:46:44
Duration : 3m 12s
CPU hours : 0.9 (46.1% cached)
Succeeded : 33
Cached : 74Working config file: singularity.autoMounts = true
singularity.runOptions = '-B /data:/tmp,/data:/data'
process {
withName: QIIME2_ALPHARAREFACTION { container = "quay.io/qiime2/core:2022.11" }
withName: QIIME2_ANCOM_ASV { container = "quay.io/qiime2/core:2022.11" }
withName: QIIME2_ANCOM_TAX { container = "quay.io/qiime2/core:2022.11" }
withName: QIIME2_BARPLOT { container = "quay.io/qiime2/core:2022.11" }
withName: QIIME2_CLASSIFY { container = "quay.io/qiime2/core:2022.11" }
withName: QIIME2_DIVERSITY_ADONIS { container = "quay.io/qiime2/core:2022.11" }
withName: QIIME2_DIVERSITY_ALPHA { container = "quay.io/qiime2/core:2022.11" }
withName: QIIME2_DIVERSITY_BETA { container = "quay.io/qiime2/core:2022.11" }
withName: QIIME2_DIVERSITY_BETAORD { container = "quay.io/qiime2/core:2022.11" }
withName: QIIME2_DIVERSITY_CORE { container = "quay.io/qiime2/core:2022.11" }
withName: QIIME2_EXPORT_ABSOLUTE { container = "quay.io/qiime2/core:2022.11" }
withName: QIIME2_EXPORT_RELASV { container = "quay.io/qiime2/core:2022.11" }
withName: QIIME2_EXPORT_RELTAX { container = "quay.io/qiime2/core:2022.11" }
withName: QIIME2_EXTRACT { container = "quay.io/qiime2/core:2022.11" }
withName: QIIME2_FEATURETABLE_GROUP { container = "quay.io/qiime2/core:2022.11" }
withName: QIIME2_FILTERASV { container = "quay.io/qiime2/core:2022.11" }
withName: QIIME2_FILTERTAXA { container = "quay.io/qiime2/core:2022.11" }
withName: QIIME2_INASV { container = "quay.io/qiime2/core:2022.11" }
withName: QIIME2_INSEQ { container = "quay.io/qiime2/core:2022.11" }
withName: QIIME2_INTAX { container = "quay.io/qiime2/core:2022.11" }
withName: QIIME2_TRAIN { container = "quay.io/qiime2/core:2022.11" }
withName: QIIME2_TREE { container = "quay.io/qiime2/core:2022.11" }
}@sgaleraalq could you add |
|
I managed to finish the pipeline with the I think it has to be related to some permissions, since I'm running on a cluster and I don't posses admin privileges. However, if I run the complete pipeline without the option, and when it fails with QIIME2_TREE I re-run it using Thank you very much for your help guys! |
|
Glad it worked, and thanks for sharing the solution to the problem! Lines 172 to 174 in 83877d2 |
|
That fastqc issue is interesting. On our systems, the /tmp dir is relatively small, while we have /data as the standard TMPDIR. The '-B /data:/data,/data:/tmp' part maps /data as /data in the singularity image, in addition to mapping /data to /tmp in the image. I did this so I would get my qiime2 progress to go into /data and not fill up the small /tmp partition. Maybe a more general solution would be just '-B /data' to bind /data in the image. /tmp is bound automatically. I think since this is an infrastructure-dependent issue, it works be good to have a FAQ or something for the mafft issue. I can change my TMPDIR env var to /tmp to see if it resolves the issue without the additional mount bind option. If so, you can have a note for folks who have TMPDIR set to something outside of /tmp and tell them the proper option to fix. @sgaleraalq could you try another run with just '-B /data' to see if you can get it to run to completion from scratch? |
|
Yes! It finally worked completely from scratch! But now it seems to work just fine! Thank you a lot for all the information @davised |
|
Did you get the missing output file again on the re-run? It seems like that may have been caused by mapping /data:/tmp. |
|
Yes, I did. I got all the outputs! I think |
|
Ok, in the future, we can instruct users to mount the non-standard TMPDIR in their config file. This may be e.g. /data or /scratch or /ssd, for example, so including a directory in the base config won't solve the issue for everyone. I'm going to test getting my sysadmin to mount it automatically in the singularity .conf file for the system. I'm not quite sure why my original env var approach did not work, but I'm glad we found a more general solution for folks. Cheers! |
|
Hey @davised . I was wondering if using that |
|
Your work dir will be specified using the -w flag, which is where most of the temp files go. I'm not quite sure where else they may be. You can probably change your TMPDIR environment variable to somewhere outside of /data and then you won't have to mount it. At least, that's my understanding. |
That might be something to keep in mind. I'll close that issue now because its resolved for you. |
Description of the bug
I can't make qiime2_tree pipeline to work due to "QIIME is caching your current deployment for improved performance. This may take a few moments and should only happen once per deployment." I searched for it within QIIME2 forum but they said it has to do with some "Songbird" package which I'm not familiar with. Is there a way to skip this step and continue with the rest of the analysis? Thanks in advance.
Command used and terminal output
Relevant files
No response
System information
No response
The text was updated successfully, but these errors were encountered: