Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Activity Hangs on Import #193

Open
bigsteve76 opened this issue Oct 23, 2024 · 14 comments
Open

Activity Hangs on Import #193

bigsteve76 opened this issue Oct 23, 2024 · 14 comments
Labels
bug Something isn't working

Comments

@bigsteve76
Copy link

In the activity log, the downloaded file hangs on import (100% Complete) until I restart the Docker instance on Unraid. Does not happen with every file, but does randomly. Will continue to next files after the restart. Downloading with Mega, GetComics, etc. No torrents. I've noticed it with GetComics, but not sure on the others.

@bigsteve76 bigsteve76 added the bug Something isn't working label Oct 23, 2024
@Casvt
Copy link
Owner

Casvt commented Oct 24, 2024

What version are you running? Do the logs say anything?

@bigsteve76
Copy link
Author

bigsteve76 commented Oct 24, 2024 via email

@Casvt
Copy link
Owner

Casvt commented Oct 24, 2024

The logs are cut off. It seems smarter to just share the log file that you can download at Settings -> General -> Logging -> Download Logs.

@bigsteve76
Copy link
Author

bigsteve76 commented Oct 25, 2024 via email

@Casvt
Copy link
Owner

Casvt commented Oct 25, 2024

The logs are still cut off. Go to the bottom of your own message, don't you see that the logs stop in the middle of the line? As I requested, please just share the downloaded file. Don't paste the logs in your message, share the file.

@hwwilliams
Copy link

hwwilliams commented Nov 10, 2024

I am having the same issue but I don't see anything in the logs about imports.
Kapowarr_log_2024_11_10_15_41.txt

OS: Docker on Debian
Version: 1.0.0

Edit: I've noticed it happening on GetComics and Mega downloads.

@Casvt
Copy link
Owner

Casvt commented Nov 11, 2024

Thanks for responding. The logs indeed don't give any error whatsoever, which is weird. Normally when the queue stalls it's because there was some error.

Can you set the logging level to DEBUG at Settings -> General -> Logging, then wait until the queue stalls again and then share the log file? Hope that will give more information.

@hwwilliams
Copy link

hwwilliams commented Nov 11, 2024

If you notice shutdowns/startups that's me restarting it from the GUI because I've noticed it's hung on import, it was hunt on Batman Gotham Nights (2020) Volume 01 Issue 021 and I restarted it and now it's moved to being hung on Batman Gotham Nights (2020) Volume 01 Issue 020.

I do see an errors in the logs but I'm not sure they're related since they're not near the events related to the files I mentioned as examples and don't re-appear whenever it hangs.

Kapowarr_log_2024_11_11_15_29.txt

Edit: The error about a conversion failure due to the file not being a zip is due to it being corrupted even though it is a zip so that may have been because I restarted it during a move operation and it was a large zip file of several GBs.

@Casvt
Copy link
Owner

Casvt commented Nov 11, 2024

Sorry, if there are any errors they don't appear in the log file you're downloading, but they do appear in the console (using docker CLI, docker logs kapowarr). Check there.

@bigsteve76
Copy link
Author

bigsteve76 commented Nov 11, 2024 via email

@hwwilliams
Copy link

Sorry, if there are any errors they don't appear in the log file you're downloading, but they do appear in the console (using docker CLI, docker logs kapowarr). Check there.

Not sure why you're not seeing them in that file, this is a screenshot of that file.

image

@Casvt Casvt moved this to Handled Soon in Kapowarr plans Nov 19, 2024
@Casvt
Copy link
Owner

Casvt commented Nov 21, 2024

@hwwilliams I checked the log file and I'm pretty sure there is an error happening that is not logged in the log file. That is what I meant with:

Sorry, if there are any errors they don't appear in the log file you're downloading, but they do appear in the console (using docker CLI, docker logs kapowarr). Check there.

To clarify, pretty much always the queue gets stuck because an error occurred. These errors are not logged to the file, but are sent to the console. So check the logs at docker logs kapowarr. There should be an error there (that does not have to do with the bad ZIP file).

Based on the logs you've given, the error will not be the same one as the one bigsteve76 has.


@bigsteve76 thank you for supplying an error.

Looking at the error, you've downloaded an archive covering multiple issues (correct, zip file for 31-45), you've set up that it gets extracted (the setting "Extract Archives Covering Multiple Issues"), Kapowarr succeeds in doing this, but when trying to rename the extracted files, it fails because files already existed with the planned new names.

I'll update the handling of Kapowarr so that it handles this better, but the source of this problem is that Kapowarr downloaded for issues that were already downloaded. The file /data/Fantastic Four/Volume 03 (1998)/Fantastic Four (1998) Volume 03 Issue 031 - 045.zip already existed (a.k.a. the issues were already downloaded), and yet it was downloaded again. Do you maybe have an idea how that could've happened? Is the file matched to the volume?

@hwwilliams
Copy link

@hwwilliams I checked the log file and I'm pretty sure there is an error happening that is not logged in the log file. That is what I meant with:

Sorry, if there are any errors they don't appear in the log file you're downloading, but they do appear in the console (using docker CLI, docker logs kapowarr). Check there.

To clarify, pretty much always the queue gets stuck because an error occurred. These errors are not logged to the file, but are sent to the console. So check the logs at docker logs kapowarr. There should be an error there (that does not have to do with the bad ZIP file).

Based on the logs you've given, the error will not be the same one as the one bigsteve76 has.

@bigsteve76 thank you for supplying an error.

Looking at the error, you've downloaded an archive covering multiple issues (correct, zip file for 31-45), you've set up that it gets extracted (the setting "Extract Archives Covering Multiple Issues"), Kapowarr succeeds in doing this, but when trying to rename the extracted files, it fails because files already existed with the planned new names.

I'll update the handling of Kapowarr so that it handles this better, but the source of this problem is that Kapowarr downloaded for issues that were already downloaded. The file /data/Fantastic Four/Volume 03 (1998)/Fantastic Four (1998) Volume 03 Issue 031 - 045.zip already existed (a.k.a. the issues were already downloaded), and yet it was downloaded again. Do you maybe have an idea how that could've happened? Is the file matched to the volume?

image

@HereForCodeThatsIt
Copy link

HereForCodeThatsIt commented Dec 12, 2024

Experiencing this too, following up from reddit post.
I let the downloads get stuck on import with log level on debug

[OP pasted logs here. The removed logs did not contain useful information. Removed from comment by Casvt to maintain readability of thread.]

[15:32:50][INFO] Starting download: 3
[15:32:50][DEBUG] Closing connection <DBConnection; Download Importer; 137455125973184>
[15:32:50][DEBUG] Starting new HTTPS connection (1): g.api.mega.co.nz:443
[15:32:52][DEBUG] https://g.api.mega.co.nz:443 "POST /cs?id=868275723&sid=t5MxlXY9gEj7IMly0_RjJG93VXdHMFhfN19Nervaw2hugtWKxmdpMc7zRg HTTP/1.1" 200 542
[15:32:52][DEBUG] Starting new HTTP connection (1): gfs214n185.userstorage.mega.co.nz:80
[15:32:52][DEBUG] http://gfs214n185.userstorage.mega.co.nz:80 "GET /dl/uKIXGbs5jtMgJgfb1YVutWFyf274DfxkPOa77A-QW418JI0BZCVMe32wW06shxrE94NHHhtgHytUhH7B1eI1gvV61VpaEdZBY3aqrWQOPtWc8NR1qcPiy-unTY4Ed75K4X5JOTP1S7TiSkfBN_19TOQ9h6eYp4aF_vDwM4H2IvAF7KkjYe32TeyZzizHUpKLZ27-57lZKWhqzF1n7o-_lFafyJafs3DNoE_yV3zzZ8kRDW_TOD6G9xUftA HTTP/1.1" 200 66591670
[15:33:01][INFO] Postprocessing of successful download: 3
[15:33:01][DEBUG] Creating connection <DBConnection; Download Handler; 137455125973728>
[15:33:01][DEBUG] Moving download to final destination: <MegaDownload, https://mega.nz/file/8TFH2RKT#aOlzwQrkCbeAivbGOqE-AqvIIYUjWMEB4FWNgZCA_wo, /app/temp_downloads/Star Wars (2020) Vol. 03 Issue 049.cbz>, Dest: /Comics/Star Wars/Star Wars Volume 03 (2020)/Star Wars (2020) Vol. 03 Issue 049.cbz
[15:33:01][DEBUG] Renaming file /app/temp_downloads/Star Wars (2020) Vol. 03 Issue 049.cbz to /Comics/Star Wars/Star Wars Volume 03 (2020)/Star Wars (2020) Vol. 03 Issue 049.cbz
[15:33:01][DEBUG] Scanning for files for 3
[15:33:01][DEBUG] Extracting filename data: /Comics/Star Wars/Star Wars Volume 03 (2020)/Star Wars (2020) Vol. 03 Issue 049.cbz
[15:33:01][DEBUG] Extracting filename data: {'series': 'Star Wars', 'year': 2020, 'volume_number': 3, 'special_version': None, 'issue_number': 49.0, 'annual': False}
[15:33:01][DEBUG] Adding file to the database: /Comics/Star Wars/Star Wars Volume 03 (2020)/Star Wars (2020) Vol. 03 Issue 049.cbz
[15:33:07][DEBUG] Deleting folders from /Comics/Star Wars/Star Wars Volume 03 (2020)/.archive_extract until /Comics/Star Wars/Star Wars Volume 03 (2020)
[15:33:07][DEBUG] Deleting folder and children: /Comics/Star Wars/Star Wars Volume 03 (2020)/.archive_extract
[15:33:07][DEBUG] Renaming file /Comics/Star Wars/Star Wars Volume 03 (2020)/Star Wars (2020) Vol. 03 Issue 049.rar to /Comics/Star Wars/Star Wars Volume 03 (2020)/Star Wars (2020) Vol. 03 Issue 049.cbr
[15:33:07][DEBUG] Closing connection <DBConnection; Download Handler; 137455125973728>
Exception in thread Download Handler:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/shutil.py", line 791, in move
    os.rename(src, real_dst)
FileNotFoundError: [Errno 2] No such file or directory: '/Comics/Star Wars/Star Wars Volume 03 (2020)/Star Wars (2020) Vol. 03 Issue 049.rar' -> '/Comics/Star Wars/Star Wars Volume 03 (2020)/Star Wars (2020) Vol. 03 Issue 049.cbr'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/app/backend/download_queue.py", line 124, in __run_download
    PostProcesser.success(download)
  File "/app/backend/post_processing.py", line 236, in success
    cls._run_actions(cls.actions_success, download)
  File "/app/backend/post_processing.py", line 230, in _run_actions
    action(download)
  File "/app/backend/post_processing.py", line 117, in convert_file
    mass_convert(
  File "/app/backend/conversion.py", line 264, in mass_convert
    convert_file(
  File "/app/backend/conversion.py", line 97, in convert_file
    return conversion_class.convert(file)
  File "/app/backend/converters.py", line 300, in convert
    cbr_file = RARtoCBR.convert(rar_file)
  File "/app/backend/converters.py", line 324, in convert
    rename_file(
  File "/app/backend/files.py", line 261, in rename_file
    move(before, after)
  File "/usr/local/lib/python3.8/shutil.py", line 811, in move
    copy_function(src, real_dst)
  File "/usr/local/lib/python3.8/shutil.py", line 435, in copy2
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/usr/local/lib/python3.8/shutil.py", line 264, in copyfile
    with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst:
FileNotFoundError: [Errno 2] No such file or directory: '/Comics/Star Wars/Star Wars Volume 03 (2020)/Star Wars (2020) Vol. 03 Issue 049.rar'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Status: Handled Soon
Development

No branches or pull requests

4 participants