Issue with pplacer #256

ab654 · 2020-05-27T16:43:37Z

Hi there,

I have used this program 5 times before and I never had an issue, however, I am now getting the following error:

**(gtdbtk) abreen:[gtdbtk]$ gtdbtk classify_wf --genome_dir all_genomes --extension fasta --out_dir gtdbtk_output

[2020-05-27 15:41:27] INFO: GTDB-Tk v1.1.1

[2020-05-27 15:41:27] INFO: gtdbtk classify_wf --genome_dir all_genomes --extension fasta --out_dir gtdbtk_output

[2020-05-27 15:41:27] INFO: Using GTDB-Tk reference data version r89: /home/.conda/envs/gtdbtk/share/gtdbtk-1.1.1/db/

[2020-05-27 15:41:27] INFO: Identifying markers in 24 genomes with 1 threads.

[2020-05-27 15:41:27] INFO: Running Prodigal V2.6.3 to identify genes.

==> Finished processing 24 of 24 (100.0%) genomes.

[2020-05-27 15:48:47] INFO: Identifying TIGRFAM protein families.

==> Finished processing 24 of 24 (100.0%) genomes.

[2020-05-27 15:51:45] INFO: Identifying Pfam protein families.

==> Finished processing 24 of 24 (100.0%) genomes.

[2020-05-27 15:52:05] INFO: Annotations done using HMMER 3.1b2 (February 2015)

[2020-05-27 15:52:06] INFO: Done.

[2020-05-27 15:52:07] INFO: Aligning markers in 24 genomes with 1 threads.

[2020-05-27 15:52:07] INFO: Processing 24 genomes identified as bacterial.

[2020-05-27 15:52:29] INFO: Read concatenated alignment for 23458 GTDB genomes.

==> Finished aligning 24 of 24 (100.0%) genomes.

[2020-05-27 15:58:23] INFO: Masking columns of multiple sequence alignment using canonical mask.

[2020-05-27 15:59:03] INFO: Masked alignment from 41155 to 5040 AAs.

[2020-05-27 15:59:03] INFO: 0 user genomes have amino acids in <10.0% of columns in filtered MSA.

[2020-05-27 15:59:03] INFO: Creating concatenated alignment for 23482 GTDB and user genomes.

[2020-05-27 15:59:03] INFO: Creating concatenated alignment for 24 user genomes.

[2020-05-27 15:59:03] INFO: Done.

[2020-05-27 15:59:03] INFO: Placing 24 bacterial genomes into reference tree with pplacer using 30 cpus (be patient).

Initialising pplacer [Caching likelihood information on reference tree..]

[2020-05-27 16:10:50] ERROR: Controlled exit resulting from an unrecoverable error or warning.

================================================================================

EXCEPTION: PplacerException

MESSAGE: An error was encountered while running pplacer, check the log file: gtdbtk_output/classify/intermediate_results/pplacer/pplacer.bac120.out

Traceback (most recent call last):

File "/home/.conda/envs/gtdbtk/bin/gtdbtk", line 492, in

gt_parser.parse_options(args)

File "/home/.conda/envs/gtdbtk/lib/python3.8/site-packages/gtdbtk/main.py", line 666, in parse_options

self.classify(options)

File "/home/.conda/envs/gtdbtk/lib/python3.8/site-packages/gtdbtk/main.py", line 426, in classify

classify.run(genomes,

File "/home/.conda/envs/gtdbtk/lib/python3.8/site-packages/gtdbtk/classify.py", line 351, in run

classify_tree = self.place_genomes(user_msa_file,

File "/home/.conda/envs/gtdbtk/lib/python3.8/site-packages/gtdbtk/classify.py", line 168, in place_genomes

pplacer.run(self.pplacer_cpus, 'WAG', pplacer_ref_pkg, pplacer_json_out,

File "/home.conda/envs/gtdbtk/lib/python3.8/site-packages/gtdbtk/external/pplacer.py", line 154, in run

raise PplacerException('An error was encountered while '

gtdbtk.exceptions.PplacerException: An error was encountered while running pplacer, check the log file: gtdbtk_output/classify/intermediate_results/pplacer/pplacer.bac120.out**

Would you be able to help me please? I would really appreciate it.

Thank you.

Kind Regards,
A Breen

The text was updated successfully, but these errors were encountered:

donovan-h-parks · 2020-05-27T16:52:46Z

Hi. Can you send us the information in gtdbtk_output/classify/intermediate_results/pplacer/pplacer.bac120.out? The most likely issue is that your machine is running out of memory.

ab654 · 2020-05-27T16:55:01Z

Hi there,

Running pplacer v1.1.alpha19-0-g807f6f3 analysis on gtdbtk_output/align/gtdbtk.bac120.user_msa.fasta...
Didn't find any reference sequences in given alignment file. Using supplied reference alignment.
Pre-masking sequences... sequence length cut from 5040 to 5040.
Determining figs... figs disabled.
Allocating memory for internal nodes... done.
Caching likelihood information on reference tree...

I have ~128Gb, is this sufficient? It worked for me before using this.

Thanks,
Jamie

donovan-h-parks · 2020-05-27T21:08:27Z

It should be enough memory. From the output it looks like you might have run it using 30 CPUs. Can you try running it with a single CPU?

ab654 · 2020-05-27T21:29:21Z

Hi there, I tried that already and got the same error. I also re downloaded it to see if it would make any difference but it didn’t work either.

aaronmussig · 2020-05-29T09:19:02Z

Hello, typically errors that are encountered during the 'caching likelihood' step of pplacer indicates that there's not enough memory. This is the step where pplacer starts consuming the 100GB+ of memory.

If possible, keep an eye on what is using the memory on the system while the classify step is running. I would hazard a guess that there could be something else using the memory. As an alternative, pplacer has an mmap flag (maps the memory to disk to try and reduce the memory burden), this can be accessed using the scratch_dir command.

btemperton · 2020-08-25T20:31:42Z

I'm currently running into an issue where pplacer is running out of memory despite using --pplacer_cpus 1 and running as an exclusive user on a node with 3 Tb of RAM when placing 191 MAGs into the bacterial tree.

The command I'm using is :

echo "The amount of available memory is:"
free -h

gtdbtk classify_wf \
--out_dir surface/gtdbtk \
--cpus 32 \
--batchfile batchfile.txt \
--prefix AE1916_HiC \
--pplacer_cpus 1

The full output log is attached, but the error that's killing it is:

[2020-08-25 21:15:56] INFO: Placing 191 bacterial genomes into reference tree with pplacer using 1 cpus (be patient).
==> Step 4 of 9 Allocating memory for internal nodes (9.65 GB).Uncaught exception: Out of memory                 
Fatal error: exception Out_of_memory
==> Step 5 of 9: Caching likelihood information on reference tree.Process Process-205:
Traceback (most recent call last):
  File "/gpfs/ts0/home/bt273/miniconda3/envs/gtdbtk/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/gpfs/ts0/home/bt273/miniconda3/envs/gtdbtk/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/gpfs/ts0/home/bt273/miniconda3/envs/gtdbtk/lib/python3.6/site-packages/gtdbtk/external/pplacer.py", line 126, in _worker
    'file: {}'.format(pplacer_out))
gtdbtk.exceptions.PplacerException: An error was encountered while running pplacer, check the log file: surface/gtdbtk/classify/intermediate_results/pplacer/pplacer.bac120.out
==> Step 5 of 9: Caching likelihood information on reference tree.

[2020-08-25 21:17:55] ERROR: Controlled exit resulting from an unrecoverable error or warning.

================================================================================
EXCEPTION: PplacerException
  MESSAGE: An error was encountered while running pplacer.
________________________________________________________________________________

Traceback (most recent call last):
  File "/gpfs/ts0/home/bt273/miniconda3/envs/gtdbtk/lib/python3.6/site-packages/gtdbtk/__main__.py", line 509, in main
    gt_parser.parse_options(args)
  File "/gpfs/ts0/home/bt273/miniconda3/envs/gtdbtk/lib/python3.6/site-packages/gtdbtk/main.py", line 754, in parse_options
    self.classify(options)
  File "/gpfs/ts0/home/bt273/miniconda3/envs/gtdbtk/lib/python3.6/site-packages/gtdbtk/main.py", line 489, in classify
    options.split_tree)
  File "/gpfs/ts0/home/bt273/miniconda3/envs/gtdbtk/lib/python3.6/site-packages/gtdbtk/classify.py", line 454, in run
    scratch_dir)
  File "/gpfs/ts0/home/bt273/miniconda3/envs/gtdbtk/lib/python3.6/site-packages/gtdbtk/classify.py", line 194, in place_genomes
    user_msa_file, pplacer_out, pplacer_mmap_file)
  File "/gpfs/ts0/home/bt273/miniconda3/envs/gtdbtk/lib/python3.6/site-packages/gtdbtk/external/pplacer.py", line 93, in run
    'An error was encountered while running pplacer.')
gtdbtk.exceptions.PplacerException: An error was encountered while running pplacer.
================================================================================

GTDBTK_3664.log

Update:
An out of memory error occurs even when using the --scratch_dir parameter (see log file GTDBTK_3666.log)
GTDBTK_3666.log

aaronmussig · 2020-10-29T02:01:22Z

@btemperton It sounds like you've done everything right to get pplacer working but why it's out of memory is not obvious. There are certainly some quirks when using pplacer, e.g. >64 threads will cause it to hang.

The issue ultimately is with pplacer, unfortunately. However, I do have an idea to try and get it working on your system:

Can you try run GTDB-Tk using the Docker container (or using singularity)?
- If it fails, can you try cap the maximum amount of RAM for the container to < 500 GB?
This is pretty crude, but can you try request 200 GB of RAM and see if it's possible? (warning that it could potentially crash other programs if the server is out of memory)

Save the following script: aaronmussig/memtest.c
Compile it with gcc -o memtest memtest.c
Run it with ./memtest 200 and wait for it to fail (or not).

e.g. this is my output:

> gcc -o memtest memtest.c && ./memtest 40
Requesting 1 GB chunks up to 40 GB.
 INFO: Memory requested (0/40) GB... requesting memory... initialising... done!
 INFO: Memory requested (1/40) GB... requesting memory... initialising... done!
 INFO: Memory requested (2/40) GB... requesting memory... initialising... done!
 INFO: Memory requested (3/40) GB... requesting memory... initialising... done!
 INFO: Memory requested (4/40) GB... requesting memory... initialising... done!
 INFO: Memory requested (5/40) GB... requesting memory... initialising... done!
 INFO: Memory requested (6/40) GB... requesting memory... initialising... done!
 INFO: Memory requested (7/40) GB... requesting memory... initialising... done!
memtest(8550,0x108360dc0) malloc: can't allocate region
:*** mach_vm_map(size=18446744065529683968, flags: 100) failed (error code=3)
memtest(8550,0x108360dc0) malloc: *** set a breakpoint in malloc_error_break to debug
 INFO: Memory requested (8/40) GB... requesting memory...
ERROR: Unable to request more memory (failed at 8 GB).

Thanks.

aaronmussig · 2021-02-02T03:27:54Z

Closing due to inactivity.

susheelbhanu · 2021-09-09T17:40:24Z

Hi all,
I have the exact issue when trying to run GTDB-tk on a single MAG (don't ask - it's for downstream purposes so we run gtdbtk on single MAGs) ;)

I use a SLURM system and here are the job stats:

seff 2480916
Job ID: 2480916
Cluster: iris
User/Group: sbusi/clusterusers
State: OUT_OF_MEMORY (exit code 0)
Nodes: 1
Cores per node: 8
CPU Utilized: 00:44:04
CPU Efficiency: 12.56% of 05:50:56 core-walltime
Job Wall-clock time: 00:43:52
Memory Utilized: 203.69 GB
Memory Efficiency: 96.56% of 210.94 GB

It's failing at the same pplacer step when trying to place the MAG. Any insights will be greatly appreciated.

Thank you!

aaronmussig added the stale Awaiting submitter feedback. label Nov 29, 2020

aaronmussig closed this as completed Feb 2, 2021

B-1991-ing mentioned this issue Apr 21, 2022

gtdbtk.exceptions.PplacerException #386

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Issue with pplacer #256

Issue with pplacer #256

ab654 commented May 27, 2020

donovan-h-parks commented May 27, 2020

ab654 commented May 27, 2020

donovan-h-parks commented May 27, 2020

ab654 commented May 27, 2020

aaronmussig commented May 29, 2020

btemperton commented Aug 25, 2020 •

edited

Loading

aaronmussig commented Oct 29, 2020 •

edited

Loading

aaronmussig commented Feb 2, 2021

susheelbhanu commented Sep 9, 2021

Issue with pplacer #256

Issue with pplacer #256

Comments

ab654 commented May 27, 2020

donovan-h-parks commented May 27, 2020

ab654 commented May 27, 2020

donovan-h-parks commented May 27, 2020

ab654 commented May 27, 2020

aaronmussig commented May 29, 2020

btemperton commented Aug 25, 2020 • edited Loading

aaronmussig commented Oct 29, 2020 • edited Loading

aaronmussig commented Feb 2, 2021

susheelbhanu commented Sep 9, 2021

btemperton commented Aug 25, 2020 •

edited

Loading

aaronmussig commented Oct 29, 2020 •

edited

Loading