ETE3_db_status_check trying to write to default database_directory with database_directory specified

[shuzvvn]

Hi,

I was trying to run mob_recon on the lab server which I have limited access. My account doesn't have the permission to write to the default database_directory. I thought I could solve this problem by specifying the database_directory while running mob_recon. Here is the command that I used:

mob_recon -u --infile myseq.fasta --outdir myoutput/ --database_directory ~/.local/lib/python3.8/site-packages/mob_suite/databases --debug

However, I got the Error Messages as follows. It seems like it is still trying to create the Lock file under the defaut database_directory?

2021-03-11 00:49:27,904 root INFO: Creating Lock file /opt/anaconda3/lib/python3.8/site-packages/mob_suite/databases/ETE3_DB.lock [in /opt/anaconda3/lib/python3.8/site-packages/mob_suite/utils.py:438]
Traceback (most recent call last):
  File "/opt/anaconda3/bin/mob_recon", line 8, in <module>
    sys.exit(main())
  File "/opt/anaconda3/lib/python3.8/site-packages/mob_suite/mob_recon.py", line 1005, in main
    dbstatus = ETE3_db_status_check(1, ETE3_LOCK_FILE, ETE3DBTAXAFILE, logging)
  File "/opt/anaconda3/lib/python3.8/site-packages/mob_suite/utils.py", line 439, in ETE3_db_status_check
    open(file=lockfilepath, mode="w").close()
PermissionError: [Errno 13] Permission denied: '/opt/anaconda3/lib/python3.8/site-packages/mob_suite/databases/ETE3_DB.lock'

[kbessonov1984]

@shuzvvn Thank you for brining you user case. This is a know issue that we plan to address soon in next release. Currently mob-suite is not read-only file system friendly. I have the same issue on public Galaxy EU read-only file system. The database lock was put to avoid race condition (multiple instances launched on a server simultaneously) upon eventual ete3 database update (that could sporadically occur). We will need to make lock file being written to your local copy of the database directory (if such parameter is specified) and also implement recovery on permission denied error during lock file write. For now you can quickly fix this issue by putting a try and except directive command around the line 439. I will create issue so that we can address this use case scenario. We would ideally turn off ete3 library taxonomy updates to avoid the need of having this preventive lock file logic.

[fmaguire]

Worth noting ETE writing to ~/.etetoolkit during mob_init also causes similar issues when trying to deploy mob-suite in a containerised workflow.

Any idea if there is a very rough horizon for when an update to more read-only friendly behavior might happen? While totally acknowledging we are all neck-deep in other public health work right now!

[kbessonov1984]

Indeed we are currently busy with multiple tasks linked both to COVID-19 work and food pathogen surveillance (Salmonella and E. coli). Yet we've started working on this issue today. The code will be posted at the v3.1.0 branch and I will provide updates here. Will be working slowly towards release v3.1.0 that will address this and other issues in parallel with other commitments. I agree that this custom database_directory is a very desirable feature to have especially in light of read-only Singularity containers and public Galaxy read-only file system configurations. In addition, ete3 taxonomy database initialization will be performed as a post-install task avoid a racing condition. You will be able to pull intermediate code from this branch or build and install temporary Python package. Thank you for your patience.

[kbessonov1984]

Issue confirmed with the placement of the lock file after running a Singularity container singularity run https://depot.galaxyproject.org/singularity/mob_suite:3.0.1--py_0 mob_recon -i test_file.fasta -o test. This prevents to run MOB-Suite in Singularity containers which have read-only file system layer. Will need to make .lock file placement optional for such case-scenarios

2021-04-26 13:39:32,868 root INFO: Creating Lock file /usr/local/lib/python3.8/site-packages/mob_suite/databases/ETE3_DB.lock [in /usr/local/lib/python3.8/site-packages/mob_suite/utils.py:438]
Traceback (most recent call last):
  File "/usr/local/bin/mob_recon", line 10, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/site-packages/mob_suite/mob_recon.py", line 1095, in main
    dbstatus = ETE3_db_status_check(1, ETE3_LOCK_FILE, ETE3DBTAXAFILE, logging)
  File "/usr/local/lib/python3.8/site-packages/mob_suite/utils.py", line 439, in ETE3_db_status_check
    open(file=lockfilepath, mode="w").close()
OSError: [Errno 30] Read-only file system: '/usr/local/lib/python3.8/site-packages/mob_suite/databases/ETE3_DB.lock'

[cmfield]

I am having this issue with the program installed as a module - my users don't have write permission for this lock file. Is there any word on when this will be fixed?

[kbessonov1984]

@cmfield and @fmaguire Sorry for the delay with the read-only file systems compatibility implementation. We see this is an important issue to fix and I will start working on it starting this Friday and hope to have a fix by the end of the next week. This is also relevant for Singularity containers compatibility.

[kbessonov1984]

Had completed preliminary testing of the 3.0.2 version inside fresh ubuntu 21.04 container.

Please pull mob-3.0.2 branch and install from the source. Also updated README and added info in install from the source code (in case you are testing/building container).

git checkout mob-3.0.2
python setup.py install

Will continue testing tomorrow and then release it as a new version 3.0.2 which will automatically trigger Singularity and Conda resources.