Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DeviantArt 404s on every picture #436

Closed
kargaroc opened this issue Oct 3, 2019 · 18 comments
Closed

DeviantArt 404s on every picture #436

kargaroc opened this issue Oct 3, 2019 · 18 comments

Comments

@kargaroc
Copy link

kargaroc commented Oct 3, 2019

See:

[gallery-dl][debug] Version 1.10.5-dev
[gallery-dl][debug] Python 3.7.4 - Linux-5.3.0-arch1-1-ARCH-x86_64-with-arch
[gallery-dl][debug] requests 2.22.0 - urllib3 1.25.3
[gallery-dl][debug] Starting DownloadJob for 'https://www.deviantart.com/storybirdartist'
[gallery-dl][debug] Updating urllib3 ciphers
[deviantart][debug] Using DeviantartGalleryExtractor for 'https://www.deviantart.com/storybirdartist'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): www.deviantart.com:443
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/user/profile/storybirdartist HTTP/1.1" 200 4318
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/gallery/all?username=storybirdartist&offset=0&limit=24&mature_content=true HTTP/1.1" 200 None
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/deviation/metadata?deviationids%5B0%5D=B3840D14-6F54-B80F-19A3-661670DAD6EC&deviationids%5B1%5D=22C51359-8A09-315B-25AA-18537E93671E&deviationids%5B2%5D=6099535B-A0B1-AAB9-43C8-D31A2E9C379E&deviationids%5B3%5D=177DD0AD-E138-6FB6-485E-633721AB0066&deviationids%5B4%5D=65CE3EE6-980B-B18A-087A-AD5DC6E7FD1B&deviationids%5B5%5D=E8BC535D-5604-7B00-E55D-5B6D4A46A94E&deviationids%5B6%5D=BFC43F6F-FF4C-4233-5299-3EF0ED6A5E4D&deviationids%5B7%5D=D21FEF05-2367-E32F-CABA-BDDF16FE9496&deviationids%5B8%5D=97133E5D-A867-9FBA-8322-3C660EAFEAFE&deviationids%5B9%5D=488ECA68-D0BD-5695-7ECE-CFD41054D411&deviationids%5B10%5D=EC8EA346-2081-8841-9AE6-73120E967EAE&deviationids%5B11%5D=2C17EE56-9476-6EFE-7D01-45776491031C&deviationids%5B12%5D=E891F332-76C1-49B4-96B9-4B1CEAB5553F&deviationids%5B13%5D=6EFDCAE9-6C07-E057-1CA4-61E198A739B9&deviationids%5B14%5D=46DB5D0E-854B-B215-8C01-7A87E7D2C461&deviationids%5B15%5D=414A7004-F930-3465-6909-76E8222AE08F&deviationids%5B16%5D=6F3F7601-86CC-7765-7C56-C8E476A34E45&deviationids%5B17%5D=13179795-C07C-F527-4C6C-8BA245038BEC&deviationids%5B18%5D=878472AF-9A7A-8D7C-19E3-DFB078C45360&deviationids%5B19%5D=0C4186ED-54E2-A5DB-4F82-3C2BF87BED99&deviationids%5B20%5D=C4259037-9774-496A-BA0A-3F998942F078&deviationids%5B21%5D=69430E75-5A96-1822-B411-A90FB1A7BFAF&deviationids%5B22%5D=BF81F754-7194-D80D-6660-B38E50AAC00B&deviationids%5B23%5D=E672882D-3FD0-DD3D-3EF8-4F1E13CD10D3&mature_content=true HTTP/1.1" 200 None
[deviantart][debug] Active postprocessor modules: [MetadataPP]
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/deviation/download/B3840D14-6F54-B80F-19A3-661670DAD6EC?mature_content=true HTTP/1.1" 200 209
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): fc00.deviantart.net:443
[urllib3.connectionpool][debug] https://fc00.deviantart.net:443 "GET /wixmp/f/31c208e7-448b-455e-a443-772347d7fe46/ddhgi8f-4cda2f3d-0c96-41d8-955f-205f193dcaad.png?st=6kj44BSaXbKMTnIsi4yCPA&e=1570068963 HTTP/1.1" 301 178
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): orig06.deviantart.net:443
[urllib3.connectionpool][debug] https://orig06.deviantart.net:443 "GET /7c80/f/31c208e7-448b-455e-a443-772347d7fe46/ddhgi8f-4cda2f3d-0c96-41d8-955f-205f193dcaad.png HTTP/1.1" 404 7057
[downloader.http][warning] 404: Not Found for url: https://fc00.deviantart.net/wixmp/f/31c208e7-448b-455e-a443-772347d7fe46/ddhgi8f-4cda2f3d-0c96-41d8-955f-205f193dcaad.png?st=6kj44BSaXbKMTnIsi4yCPA&e=1570068963
[download][error] Failed to download deviantart_815383887_A coo'l commission.png
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/deviation/download/22C51359-8A09-315B-25AA-18537E93671E?mature_content=true HTTP/1.1" 200 209
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): fc06.deviantart.net:443
[urllib3.connectionpool][debug] https://fc06.deviantart.net:443 "GET /wixmp/f/31c208e7-448b-455e-a443-772347d7fe46/ddhggtl-fa9e2e46-60c8-4954-a3f6-1a6b31e2f027.png?st=Y51RV8kpn2GUlEstYVAJfw&e=1570068964 HTTP/1.1" 301 178
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): orig09.deviantart.net:443
[urllib3.connectionpool][debug] https://orig09.deviantart.net:443 "GET /76b7/f/31c208e7-448b-455e-a443-772347d7fe46/ddhggtl-fa9e2e46-60c8-4954-a3f6-1a6b31e2f027.png HTTP/1.1" 404 7057
[downloader.http][warning] 404: Not Found for url: https://fc06.deviantart.net/wixmp/f/31c208e7-448b-455e-a443-772347d7fe46/ddhggtl-fa9e2e46-60c8-4954-a3f6-1a6b31e2f027.png?st=Y51RV8kpn2GUlEstYVAJfw&e=1570068964
[download][error] Failed to download deviantart_815382057_Pokesquad.png
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/deviation/download/6099535B-A0B1-AAB9-43C8-D31A2E9C379E?mature_content=true HTTP/1.1" 200 207
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): fc04.deviantart.net:443
[urllib3.connectionpool][debug] https://fc04.deviantart.net:443 "GET /wixmp/f/31c208e7-448b-455e-a443-772347d7fe46/ddhggmy-1556a3ba-7ef0-4e9f-9084-3fbf4af21b70.png?st=Tb0Ynos5bRME-BSd58c0aA&e=1570068965 HTTP/1.1" 301 178
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): orig10.deviantart.net:443

@mechalincoln
Copy link

I am seeing this as well.

@Twi-Hard
Copy link

Twi-Hard commented Oct 3, 2019

It started a day or 2 ago.

It does get an image successfully rarely.

@kattjevfel
Copy link
Contributor

It seems to only affect when downloading an entire album, not individual posts.

@Twi-Hard
Copy link

Twi-Hard commented Oct 3, 2019

I'm having this problem while downloading full accounts.

@mikf
Copy link
Owner

mikf commented Oct 3, 2019

The URLs for original file downloads got changed on DeviantArt's end and are currently all pointing to a 404 page. I would expect that this gets fixed on their side in the next couple of days, but who knows. As a not very useful "workaround", turning off the original file downloads makes things work again, but only being able to download preview images doesn't really help, I suppose.

In the meantime, I've been working on a rewrite of what is basically the whole deviantart module to use the new site-internal API of their React web interface instead of the several years old OAuth API: https://github.com/mikf/gallery-dl/tree/deviantart-rewrite
It's not entirely finished or thoroughly tested at this point, but it doesn't have this 404 issue and most of the old functionality should work, i.e. downloading user galleries and favourites. Some options are either not implemented or always enabled, and it requires a traditional login with username and password to see adult (or private?) content, but it would be really neat if some of you could test this version since it might have to replace the current one. (Use secondary accounts and not your main ones. You might get banned for "suspicious" behavior and such)

@kargaroc
Copy link
Author

kargaroc commented Oct 3, 2019

rewrite of what is basically the whole deviantart module

Most important things for me is downloading the description and downloading full resolution images, for older images where that is available.
Kinda off topic though - though I guess I'll just wait for things to hopefully fix themselves.

@Twi-Hard
Copy link

Twi-Hard commented Oct 3, 2019

Everything in --write-metadata is super important for me.. espicially the tags.

@biznizz
Copy link

biznizz commented Oct 5, 2019

Can confirm that this issue is still present trying to rip whole galleries, but inputting individual pics into the command line results in a successful rip.

@sledgehammer93
Copy link

sledgehammer93 commented Oct 6, 2019

It's not entirely finished or thoroughly tested at this point, but it doesn't have this 404 issue and most of the old functionality should work

Tried out the deviantart-rewrite to test it out, @mikf. I am still occasionally getting 404's in some galleries, and a few sta.sh links as well. It's nowhere near as bad as before, but it happens once in a blue moon.

@cloudywings2
Copy link

Just wanted to confirm that I'm having the same problem

mikf added a commit that referenced this issue Oct 9, 2019
... except for sta.sh content.

Instead of using the old '/api/v1/oauth2/deviation/download' endpoint,
which started delivering URLs to 404 pages a while ago,
it is also possible to get a download URL from the relatively new
'/_napi/da-browse/shared_api/deviation/extended_fetch' endpoint
used by DeviantArt's Eclipse interface.

The current strategy is therefore:
- Iterate over deviations using the OAuth2 API
- Fetch original download URLs with the new NAPI/Shared API
@mikf
Copy link
Owner

mikf commented Oct 9, 2019

I've found a way to get things to work again. Everything should work as it did before, except sta.sh content. And fetching a download URL might be a bit slower than before. See 27b5b24 for details.

@sledgehammer93 thank you for testing, but I think using the new interface might after all be more trouble than it's worth. A lot of "non-mainstream" things like sta.sh and groups are currently not supported (and might never be), and logging in also doesn't work anymore (captcha), so I'd rather leave everything as is for as long as possible.

@mikf mikf added the bug label Oct 9, 2019
@Twi-Hard
Copy link

Twi-Hard commented Oct 9, 2019

A lot of the content I download if from sta.sh. Is there a way to tell what I missed so I can download it myself or something? That would be missing out on thousands of files for me considering I'm downloading ~21,000 accounts with an average of 150 images per account. I see sta.sh stuff in the logs all the time.

@Twi-Hard
Copy link

Twi-Hard commented Oct 9, 2019

I haven't even gotten through the letter "a" (for account names) yet and there's already 18,146 sta.sh links in the log.

@mikf
Copy link
Owner

mikf commented Oct 10, 2019

Downloading from sta.sh should now also work again: 94eb7c6

@mechalincoln
Copy link

Downloading from sta.sh should now also work again: 94eb7c6

How can I add this to the Windows exec version of gallery-dl? Should I wait for a new version?

I appreciate your help on this.

@Twi-Hard
Copy link

Thanks for fixing that :D

@Hrxn
Copy link
Contributor

Hrxn commented Oct 11, 2019

How can I add this to the Windows exec version of gallery-dl? Should I wait for a new version?

I appreciate your help on this.

You can't. So either wait for a new release version, or, if you have Python installed, you can use pip to install gallery-dl directly from the repo, like this:

pip install --upgrade "https://github.com/mikf/gallery-dl/archive/master.zip"

@mikf
Copy link
Owner

mikf commented Oct 11, 2019

Well, here is the new version including a Windows executable:
https://github.com/mikf/gallery-dl/releases/tag/v1.10.6

Let's hope the DeviantArt devs don't spend more of their time "fixing" old APIs and instead finish their new Eclipse design.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

9 participants