Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DeviantArt Stash download can lead to infinite loop (rare) #489

Closed
a-washing-machine opened this issue Nov 26, 2019 · 1 comment
Closed

DeviantArt Stash download can lead to infinite loop (rare) #489

a-washing-machine opened this issue Nov 26, 2019 · 1 comment
Labels
bug

Comments

@a-washing-machine
Copy link

a-washing-machine commented Nov 26, 2019 •
edited
Loading

...Not the best time, with the deviantArt downloader throwing 404s since late last night, but a few days ago I noticed a problem when downloading with "extra=true":

Sometimes a stash image's description contains links to another stash-item, which links back to the first, or a link to the stash folder containing the item. This creates an infinite loop.

(There are also infinite loop cases where a stash item's description just flat out links to itself, presumably the artist copy-pasted the description from the original deviation linking the stash item.)

(Not sure, but this might also be able to happen with regular deviations linking back to each other.)

The loop also isn't broken if you set the "A" parameter to, say, "A=20". Eventually, python crashes.

Example:
https://www.deviantart.com/lopoddity/art/Comm-Saving-Owlowiscious-649082718
https://sta.sh/0cmi51hr5pv

(The loop is still reproducible right now, it just won't be able to download the image at the moment.)

I found it's an extremely rare case (maybe 10 in 3000 galleries), but it does happen. Wouldn't give it high priority, just check if it happens to be an easy fix.

My config:

{
    "extractor":
    {
          "deviantart":
        {
			"refresh-token": "..................................",
			"client-id": "..................................",
            "client-secret": "..................................",			
			"extra": true,
			"metadata": true,
            "postprocessors": [{
                "name": "metadata",
                "mode": "custom",
                "format": "{description}\n"
            }]
        }
    }

}

Fatal Python error: Cannot recover from stack overflow.

Current thread 0x000638d4 (most recent call first):
  File "E:\venv-win32\lib\abc.py", line 143 in __subclasscheck__
  File "E:\venv-win32\lib\abc.py", line 139 in __instancecheck__
  File "site-packages\cryptography\x509\extensions.py", line 1204 in <genexpr>
  File "site-packages\cryptography\x509\extensions.py", line 1204 in __init__
  File "site-packages\cryptography\x509\extensions.py", line 1243 in __init__
  File "site-packages\cryptography\hazmat\backends\openssl\decode_asn1.py", line 438 in _decode_subject_alt_name
  File "site-packages\cryptography\hazmat\backends\openssl\decode_asn1.py", line 252 in parse
  File "site-packages\cryptography\hazmat\backends\openssl\x509.py", line 127 in extensions
  File "site-packages\cryptography\utils.py", line 170 in inner
  File "site-packages\urllib3\contrib\pyopenssl.py", line 223 in get_subj_alt_name
  File "site-packages\urllib3\contrib\pyopenssl.py", line 381 in getpeercert
  File "site-packages\urllib3\connection.py", line 409 in connect
  File "site-packages\urllib3\connectionpool.py", line 994 in _validate_conn
  File "site-packages\urllib3\connectionpool.py", line 376 in _make_request
  File "site-packages\urllib3\connectionpool.py", line 672 in urlopen
  File "site-packages\requests\adapters.py", line 449 in send
  File "site-packages\requests\sessions.py", line 646 in send
  File "site-packages\requests\sessions.py", line 247 in resolve_redirects
  File "site-packages\requests\sessions.py", line 668 in <listcomp>
  File "site-packages\requests\sessions.py", line 668 in send
  File "site-packages\requests\sessions.py", line 533 in request
  File "gallery_dl\downloader\http.py", line 91 in _download_impl
  File "gallery_dl\downloader\http.py", line 50 in download
  File "gallery_dl\job.py", line 283 in download
  File "gallery_dl\job.py", line 204 in handle_url
  File "gallery_dl\job.py", line 83 in dispatch
  File "gallery_dl\job.py", line 49 in run
  File "gallery_dl\job.py", line 256 in handle_queue
  File "gallery_dl\job.py", line 92 in dispatch
  File "gallery_dl\job.py", line 49 in run
  File "gallery_dl\job.py", line 256 in handle_queue
  File "gallery_dl\job.py", line 92 in dispatch
  File "gallery_dl\job.py", line 49 in run
  File "gallery_dl\job.py", line 256 in handle_queue
  File "gallery_dl\job.py", line 92 in dispatch
  File "gallery_dl\job.py", line 49 in run
  File "gallery_dl\job.py", line 256 in handle_queue
  File "gallery_dl\job.py", line 92 in dispatch
  File "gallery_dl\job.py", line 49 in run
  File "gallery_dl\job.py", line 256 in handle_queue
  File "gallery_dl\job.py", line 92 in dispatch
  File "gallery_dl\job.py", line 49 in run
  File "gallery_dl\job.py", line 256 in handle_queue
  File "gallery_dl\job.py", line 92 in dispatch
  File "gallery_dl\job.py", line 49 in run
  File "gallery_dl\job.py", line 256 in handle_queue
  File "gallery_dl\job.py", line 92 in dispatch
  File "gallery_dl\job.py", line 49 in run
  File "gallery_dl\job.py", line 256 in handle_queue
  File "gallery_dl\job.py", line 92 in dispatch
  File "gallery_dl\job.py", line 49 in run
  File "gallery_dl\job.py", line 256 in handle_queue
  File "gallery_dl\job.py", line 92 in dispatch
  File "gallery_dl\job.py", line 49 in run
  File "gallery_dl\job.py", line 256 in handle_queue
  File "gallery_dl\job.py", line 92 in dispatch
  File "gallery_dl\job.py", line 49 in run
  File "gallery_dl\job.py", line 256 in handle_queue
  File "gallery_dl\job.py", line 92 in dispatch
  File "gallery_dl\job.py", line 49 in run
  File "gallery_dl\job.py", line 256 in handle_queue
  File "gallery_dl\job.py", line 92 in dispatch
  File "gallery_dl\job.py", line 49 in run
  File "gallery_dl\job.py", line 256 in handle_queue
  File "gallery_dl\job.py", line 92 in dispatch
  File "gallery_dl\job.py", line 49 in run
  File "gallery_dl\job.py", line 256 in handle_queue
  File "gallery_dl\job.py", line 92 in dispatch
  File "gallery_dl\job.py", line 49 in run
  File "gallery_dl\job.py", line 256 in handle_queue
  File "gallery_dl\job.py", line 92 in dispatch
  File "gallery_dl\job.py", line 49 in run
  File "gallery_dl\job.py", line 256 in handle_queue
  File "gallery_dl\job.py", line 92 in dispatch
  File "gallery_dl\job.py", line 49 in run
  File "gallery_dl\job.py", line 256 in handle_queue
  File "gallery_dl\job.py", line 92 in dispatch
  File "gallery_dl\job.py", line 49 in run
  File "gallery_dl\job.py", line 256 in handle_queue
  File "gallery_dl\job.py", line 92 in dispatch
  File "gallery_dl\job.py", line 49 in run
  File "gallery_dl\job.py", line 256 in handle_queue
  File "gallery_dl\job.py", line 92 in dispatch
  File "gallery_dl\job.py", line 49 in run
  File "gallery_dl\job.py", line 256 in handle_queue
  File "gallery_dl\job.py", line 92 in dispatch
  File "gallery_dl\job.py", line 49 in run
  File "gallery_dl\job.py", line 256 in handle_queue
  File "gallery_dl\job.py", line 92 in dispatch
  File "gallery_dl\job.py", line 49 in run
  File "gallery_dl\job.py", line 256 in handle_queue
  File "gallery_dl\job.py", line 92 in dispatch
  File "gallery_dl\job.py", line 49 in run
  File "gallery_dl\job.py", line 256 in handle_queue
  File "gallery_dl\job.py", line 92 in dispatch
  File "gallery_dl\job.py", line 49 in run
  File "gallery_dl\job.py", line 256 in handle_queue
  File "gallery_dl\job.py", line 92 in dispatch
  File "gallery_dl\job.py", line 49 in run
  File "gallery_dl\job.py", line 256 in handle_queue
  ...
@mikf mikf added the bug label Nov 26, 2019
@mikf
Copy link
Owner

mikf commented Nov 30, 2019

A proper fix for this might not be as easy as I hoped, but, if you are OK with not following sta.sh links from other sta.sh pages, you can prevent those infinte loops by disabling extra for sta.sh-ed content only:

{
    "extractor":{
        "deviantart": {
            "extra": true,
            "stash": {
                "extra": false
            }
        }
    }
}

@mikf mikf closed this as completed in 2e2fc7f Dec 28, 2019
@calboi91 calboi91 mentioned this issue Mar 28, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mikf @a-washing-machine