Flye install fails with Python 3.12

[pushkarnk]

With Python 3.12 installed on the Ubuntu dev release, when I tried to make, install and run the install test, the test fails with :

python3 flye/tests/test_toy.py
Running toy test:

Traceback (most recent call last):
  File "/usr/local/bin/flye", line 33, in <module>
    sys.exit(load_entry_point('flye==2.9.3', 'console_scripts', 'flye')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/bin/flye", line 25, in importlib_load_entry_point
    return next(matches).load()
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/importlib/metadata/__init__.py", line 205, in load
    module = import_module(match.group('module'))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 994, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/usr/local/lib/python3.12/dist-packages/flye/main.py", line 19, in <module>
    import flye.polishing.alignment as aln
  File "/usr/local/lib/python3.12/dist-packages/flye/polishing/alignment.py", line 18, in <module>
    import flye.utils.fasta_parser as fp
  File "/usr/local/lib/python3.12/dist-packages/flye/utils/fasta_parser.py", line 26, in <module>
    from flye.six.moves import range
ModuleNotFoundError: No module named 'flye.six.moves'
Traceback (most recent call last):
  File "/root/Flye/flye/tests/test_toy.py", line 41, in <module>
    sys.exit(main())
             ^^^^^^
  File "/root/Flye/flye/tests/test_toy.py", line 36, in main
    test_toy()
  File "/root/Flye/flye/tests/test_toy.py", line 29, in test_toy
    subprocess.check_call(["flye", "--pacbio-corr", reads_file, "-g", "500k",
  File "/usr/lib/python3.12/subprocess.py", line 413, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['flye', '--pacbio-corr', '/root/Flye/flye/tests/data/ecoli_500kb_reads_hifi.fastq.gz', '-g', '500k', '-o', 'flye_toy_test', '-t', '8', '-m', '1000']' returned non-zero exit status 1.

In short, Python 3.12 fails to find the "moves" module which is dynamically loaded using flye/six.py. This is likely to be because Python3 has gradually moved away from find_module() to find_spec() for path-based finders searching the meta path:

Changed in version 3.4: The [find_spec()](https://docs.python.org/3/library/importlib.html#importlib.abc.MetaPathFinder.find_spec) method of meta path finders replaced find_module(), which is now deprecated. While it will continue to work without change, the import machinery will try it only if the finder does not implement [find_spec()](https://docs.python.org/3/library/importlib.html#importlib.abc.MetaPathFinder.find_spec).

Changed in version 3.10: Use of find_module() by the import system now raises [ImportWarning](https://docs.python.org/3/library/exceptions.html#ImportWarning).

Changed in version 3.12: find_module() has been removed. Use [find_spec()](https://docs.python.org/3/library/importlib.html#importlib.abc.MetaPathFinder.find_spec) instead.

flye/six.py hasn't yet moved from find_module() to find_spec() and this is likely to be the reason for the above error.

[pushkarnk]

Fix proposed via #670

[mikolmogorov]

Thanks, will work on that in the next release.

[a-detiste]

six should be removed too

[mikolmogorov]

Version 2.9.5 now supports Python 3.12. six is not removed yet, but will be in the next release.

[tnn111]

Thanks. I went and installed it. I followed the instructions to install from source. All looked good (apart from a large number of warnings from g++), but when I went to test it, I got the attached error. I’m totally baffled; it’s the same error message I have been receiving. I ran the same dataset thought 2.9.2 on another system and everything works fine. Any suggestions? [2024-08-27 13:12:22] ERROR: Error running minimap2, terminating. See the alignment error log for details: /Spawn/Isolates/LT001001/results_flye_correct/10-consensus/minimap.stderr [2024-08-27 13:12:22] ERROR: Cmd: flye-minimap2 '/Spawn/Isolates/LT001001/results_flye_correct/00-assembly/draft_assembly.fasta' '/Spawn/Isolates/LT001001/LT001001_correct_nano.fasta.gz' -x map-ont -t 32 -k 17 -a -p 0.5 -N 10 --sam-hit-only -L -K 1G -z 1000 -Q --secondary-seq -I 64G | flye-samtools view -T '/Spawn/Isolates/LT001001/results_flye_correct/00-assembly/draft_assembly.fasta' -u - | flye-samtools sort -T '/Spawn/Isolates/LT001001/results_flye_correct/10-consensus/sort_240827_131218' -O bam -@ 4 -l 1 -m 1G -o '/Spawn/Isolates/LT001001/results_flye_correct/10-consensus/minimap.bam' [2024-08-27 13:12:22] ERROR: Command '['/bin/bash', '-c', "set -eo pipefail; flye-minimap2 '/Spawn/Isolates/LT001001/results_flye_correct/00-assembly/draft_assembly.fasta' '/Spawn/Isolates/LT001001/LT001001_correct_nano.fasta.gz' -x map-ont -t 32 -k 17 -a -p 0.5 -N 10 --sam-hit-only -L -K 1G -z 1000 -Q --secondary-seq -I 64G | flye-samtools view -T '/Spawn/Isolates/LT001001/results_flye_correct/00-assembly/draft_assembly.fasta' -u - | flye-samtools sort -T '/Spawn/Isolates/LT001001/results_flye_correct/10-consensus/sort_240827_131218' -O bam -@ 4 -l 1 -m 1G -o '/Spawn/Isolates/LT001001/results_flye_correct/10-consensus/minimap.bam'"]' returned non-zero exit status 134. [2024-08-27 13:12:22] ERROR: Pipeline aborted Command exited with non-zero status 1 572.74user 3.78system 1:53.98elapsed 505%CPU (0avgtext+0avgdata 1805432maxresident)k 992inputs+8432outputs (3major+774067minor)pagefaults 0swaps (base) ***@***.***:/Spawn/Isolates/LT001001$ less /Spawn/Isolates/LT001001/results_flye_correct/10-consensus/minimap.stderr [samfaipath] build FASTA index... flye-samtools: bgzf.c:305: bgzf_open: Assertion `compressBound(BGZF_BLOCK_SIZE) < BGZF_MAX_BLOCK_SIZE' failed. [M::mm_idx_gen::0.061*1.00] collected minimizers [M::mm_idx_gen::0.076*5.62] sorted minimizers [M::main::0.076*5.61] loaded/built the index for 1 target sequence(s) flye-samtools: bgzf.c:351: bgzf_hopen: Assertion `compressBound(BGZF_BLOCK_SIZE) < BGZF_MAX_BLOCK_SIZE' failed. [M::mm_mapopt_update::0.083*5.22] mid_occ = 10 [M::mm_idx_stat] kmer size: 17; skip: 10; is_hpc: 0; #seq: 1 [M::mm_idx_stat::0.088*5.01] distinct minimizers: 666242 (95.39% are singletons); average occurrences: 1.053; average spacing: 5.496; total length: 3855235 /bin/bash: line 1: 525222 Broken pipe flye-minimap2 '/Spawn/Isolates/LT001001/results_flye_correct/00-assembly/draft_assembly.fasta' '/Spawn/Isolates/LT001001/LT001001_correct_nano.fasta.gz' -x map-ont -t 32 -k 17 -a -p 0.5 -N 10 --sam-hit-only -L -K 1G -z 1000 -Q --secondary-seq -I 64G 525223 Aborted (core dumped) | flye-samtools view -T '/Spawn/Isolates/LT001001/results_flye_correct/00-assembly/draft_assembly.fasta' -u - 525224 Aborted (core dumped) | flye-samtools sort -T '/Spawn/Isolates/LT001001/results_flye_correct/10-consensus/sort_240827_131218' -O bam -@ 4 -l 1 -m 1G -o '/Spawn/Isolates/LT001001/results_flye_correct/10-consensus/minimap.bam'

…

On Aug 27, 2024, at 08:56, Mikhail Kolmogorov ***@***.***> wrote: Version 2.9.5 now supports Python 3.12. six is not removed yet, but will be in the next release. — Reply to this email directly, view it on GitHub <#669 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABMXPRSHAWMQMDHKWIB7OQ3ZTSORTAVCNFSM6AAAAABCYFZ2IWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMJSHE2DMNJSHA>. You are receiving this because you are subscribed to this thread.

[shaze]

@mikolmogorov Thanks -- much appreciated -- fixed for me Python 3.12 under Rocky 9.