[deviantart] download automatically, if sta.sh is available in the description.

[ghost]

I don't know how useful this is going to be, but I hope this possible

The post for testing

[Hrxn]

You mean to additionally download those 5 sta.sh links in the description box?

[ghost]

yes, is that possible?

[Hrxn]

Not yet 😄

It's up to @mikf to decide how much change that would be.

[mikf]

2fb8517 adds a new option to search for URLs in descriptions and extract images from them, but it needs metadata to be enabled to actually have description texts available.

There is the possibility that if a description contains a link to another deviantart account, and this one links back to the first one, you'd get stuck in an infinite cycle ... so it's probably necessary to put something in place to prevent that, but at least the basic functionality is now there.

[Hrxn]

There is the possibility that if a description contains a link to another deviantart account, and this one links back to the first one, you'd get stuck in an infinite cycle ... so it's probably necessary to put something in place to prevent that, but at least the basic functionality is now there.

Store the full URL too (if processed successfully) in the archive file if external is enabled?

[ghost]

There is the possibility that if a description contains a link to another deviantart account, and this one links back to the first one, you'd get stuck in an infinite cycle ... so it's probably necessary to put something in place to prevent that

This is why I only ask to support sta.sh, most of sta.sh link only goes to Uploader's extra content, not else.

So, the configuration ended something like this?

{
    "extractor":
    {
        "deviantart":
        {
            "filename": "{date:%Y_%m_%d_%H_%M_%S}_{title}.{extension}",
            "directory": ["{category}"],
            "metadata": true,
            "external": true,
            "pattern": "sta.sh",
            "skip": "abort:3"
        },
    }
}

[mikf]

The external option has been renamed to stash and now only looks for sta.sh links in description texts. I've taken a quick look at a few descriptions and found all sort of links (Twitter timelines, Patreon accounts, other DeviantArt pages, etc). Taking all external links into consideration would have created a mess ...

Your configiration should now look something like this:

{
    "extractor":
    {
        "deviantart":
        {
            "filename": "{date:%Y_%m_%d_%H_%M_%S}_{title}.{extension}",
            "directory": ["{category}"],
            "metadata": true,
            "stash": true,
            "skip": "abort:3"
        },
    }
}

(Replaced "external" with "stash" and removed "pattern"; you can also remove "metadata" since it is now implicitly enabled when "stash" is set to true)

[ghost]

C:\Users\ucok66>gallery-dl https://www.deviantart.com/uotapo/gallery/?catpath=/
# .\gallery-dl\deviantart\2019_06_06_16_10_47_Playful Pillow Fight.jpg
[deviantart][error] An unexpected error occurred: TypeError - argument of type 'bool' is not iterable. Please run gallery-dl again with the --verbose flag, copy its output and report this issue on https://github.com/mikf/gallery-dl/issues .

C:\Users\ucok66>gallery-dl --verbose https://www.deviantart.com/uotapo/gallery/?catpath=/
[gallery-dl][debug] Version 1.8.6-dev
[gallery-dl][debug] Python 3.7.3 - Windows-10-10.0.17763-SP0
[gallery-dl][debug] requests 2.21.0 - urllib3 1.24
[gallery-dl][debug] Starting DownloadJob for 'https://www.deviantart.com/uotapo/gallery/?catpath=/'
[gallery-dl][debug] updating default urllib3 ciphers
[deviantart][debug] Using DeviantartGalleryExtractor for 'https://www.deviantart.com/uotapo/gallery/?catpath=/'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): www.deviantart.com:443
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/user/profile/uotapo HTTP/1.1" 200 1728
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/gallery/all?username=uotapo&offset=0&limit=24&mature_content=true HTTP/1.1" 200 None
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/deviation/metadata?deviationids[0]=017B485C-BC9E-9633-C3C1-86D23947AD08&deviationids[1]=3D89C6D1-4CE9-3B0E-9F32-156FA0E86A39&deviationids[2]=4998BA1C-433C-9084-78B2-EEF10191F29B&deviationids[3]=0F36F647-D17C-4700-29B9-3CA338B5E8D9&deviationids[4]=1ADC1FFD-DE96-1CA6-E79B-C66929624D81&deviationids[5]=8DC82772-7DED-A39E-F41F-53D8818829DA&deviationids[6]=1DDAA0E0-0E8E-1AED-E456-5CE698961E4F&deviationids[7]=88EE2095-6702-0B3A-4289-78E4926E33FC&deviationids[8]=9EEA78FE-F554-370A-12DD-EBE81B69EA30&deviationids[9]=5ABB6DBD-0E28-FF3C-274E-2C5C66C0DA13&deviationids[10]=EF1CB8BB-208C-A34A-F56A-B79F5EA7407B&deviationids[11]=43F26821-D5B7-9885-DB4A-1E5E247A6C13&deviationids[12]=6E473B31-9973-D4B4-26E7-DBFEFA299E12&deviationids[13]=5A91164C-BCEB-9BBF-DF97-268FC8994C5A&deviationids[14]=1707127D-D4FF-01C5-910C-575D4298EB07&deviationids[15]=B153269E-113A-F9B2-2C31-F9DA5E32EF35&deviationids[16]=9EA1493C-EB39-8752-1B35-651BF83B77DF&deviationids[17]=9F06C57D-C831-B8B6-F947-19B23D7265C7&deviationids[18]=9186C273-507A-EA76-E92C-61181184018D&deviationids[19]=A64AC8EE-0AD1-1851-FB32-7B2AF2FE23B2&deviationids[20]=5ED7E449-926A-E2F2-7DD7-E98D064D0F48&deviationids[21]=FAEB2E4E-435D-511C-1FDD-1D96CE50AC79&deviationids[22]=76966203-BDE6-8161-39BF-873B1B6AF382&deviationids[23]=CA474620-EFC0-8893-5FE1-0F638FE5FA29&mature_content=true HTTP/1.1" 200 4654
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/deviation/download/017B485C-BC9E-9633-C3C1-86D23947AD08?mature_content=true HTTP/1.1" 200 1236
# .\gallery-dl\deviantart\2019_06_06_16_10_47_Playful Pillow Fight.jpg
[deviantart][error] An unexpected error occurred: TypeError - argument of type 'bool' is not iterable. Please run gallery-dl again with the --verbose flag, copy its output and report this issue on https://github.com/mikf/gallery-dl/issues .
[deviantart][debug]
Traceback (most recent call last):
  File "c:\users\ucok66\appdata\local\programs\python\python37\lib\site-packages\gallery_dl\job.py", line 54, in run
    self.dispatch(msg)
  File "c:\users\ucok66\appdata\local\programs\python\python37\lib\site-packages\gallery_dl\job.py", line 107, in dispatch
    self.handle_queue(url, kwds)
  File "c:\users\ucok66\appdata\local\programs\python\python37\lib\site-packages\gallery_dl\job.py", line 253, in handle_queue
    extr = keywords["_extractor"].from_url(url)
  File "c:\users\ucok66\appdata\local\programs\python\python37\lib\site-packages\gallery_dl\extractor\common.py", line 51, in from_url
    return cls(match) if match else None
  File "c:\users\ucok66\appdata\local\programs\python\python37\lib\site-packages\gallery_dl\extractor\deviantart.py", line 427, in __init__
    DeviantartExtractor.__init__(self, match)
  File "c:\users\ucok66\appdata\local\programs\python\python37\lib\site-packages\gallery_dl\extractor\deviantart.py", line 37, in __init__
    Extractor.__init__(self, match)
  File "c:\users\ucok66\appdata\local\programs\python\python37\lib\site-packages\gallery_dl\extractor\common.py", line 39, in __init__
    self._init_headers()
  File "c:\users\ucok66\appdata\local\programs\python\python37\lib\site-packages\gallery_dl\extractor\common.py", line 133, in _init_headers
    "user-agent", ("Mozilla/5.0 (X11; Linux x86_64; rv:62.0) "
  File "c:\users\ucok66\appdata\local\programs\python\python37\lib\site-packages\gallery_dl\extractor\common.py", line 64, in config
    ("extractor", self.category, self.subcategory, key), default)
  File "c:\users\ucok66\appdata\local\programs\python\python37\lib\site-packages\gallery_dl\config.py", line 95, in interpolate
    if lkey in conf:
TypeError: argument of type 'bool' is not iterable

My config is same

{
    "extractor":
    {
        "deviantart":
        {
            "filename": "{date:%Y_%m_%d_%H_%M_%S}_{title}.{extension}",
            "directory": ["{category}"],
            "metadata": true,
            "stash": true,
            "skip": "abort:3"
        },
    }
}

What is happening?

[mikf]

I'm sorry. I made a stupid mistake and forgot the name stash is already used by the StashExtractor. Everything worked fine when I tested this with -o stash=1, but setting it in a config file as anything other than a dictionary causes problems during config lookups when initializing a StashExtractor object, as you noticed.

I've now renamed this option (again) to extra to avoid any further name collisions.

[ghost]

yup, this is working well now. Thanks for the fix.