[deviantart] Add support for Groups galleries #26
Comments
|
I've investigated this a bit and came to the conclusion that groups behave in exactly the same manner as regular users do, with one major exception: You can't get all Deviation-objects of a group in the same way as you can for a user, i.e. gallery/all returns nothing. Everything else works the same (gallery folders, favorites, journals). There also seems to be no way to differentiate between groups and users based on their URL or otherwise. I only see two possible "solutions" for this, but I don't really like any of them. Maybe you have a better idea?
Another idea that comes to mind as I am writing this:
It's just that checking for an empty result (in this case an iterator) is quite messy. |
The code for this and the available metadata is probably going to change again. This extractor is very similar to the favorite- extractor, so they might be "combined" or something like that.
Good point. The URL scheme is exactly the same, and except for profile-related information, i.e. personal data and stats etc. they are basically the same. Minus this difference in I noticed something else, but it's the same underlying problem, as far as I understand. Using this profile as example again: http://arsenixc.deviantart.com/ And its Favourites and Gallery and what gallery-dl returns
Little side note: So when it comes to the API, the difference between user and group is pretty irrelevant, at least for the purposes of gallery-dl, but DeviantArt makes a distinction between Gallery and Collection and uses different API endpoints, /gallery/folders, /collections/folders, and /gallery/all, which only exists for Gallery and not Collection. Well, at first sight, not relying on By the way, I'm pretty sure that I've seen it on a few profiles: A gallery folder that also contains folders. Another thing from the API documentation: GET /user/profile/{username} I'm not sure, but couldn't this be used as a check to differentiate between users and groups? |
|
Another thing, if I read it right Okay, I think I have a small suggestion here: It would be nice if some others could chime in here, to get some different opinions, maybe. Wouldn't do any harm at least. Well, I'll start with the first point: §1 This means that, in case our hypothetical user wants both, gallery-dl has to be fed twice. With different URLs. |
They previously weren't supported for galleries and journals. This also increases the 'limit' parameter for API calls to its respective maximum.
|
Missing support for
I tried this for a couple of users and just using the contents of their gallery-folders didn't work all the time. Without the "Featured" folder you can get less then their actual Deviation count; with the "Featured" folder you get more than that (duplicates, probably).
I've found an example of that: http://rachychan.deviantart.com/gallery/46449774/Pokemon
Doesn't work for groups: There are actually undocumented API endpoints for groups that don't really work: wix-incubator/DeviantArt-API#122
That, and consistent behavior, are quite important, I think. For that reason I tried to compile a list of all possible URLs for a Deviantart user:
My suggestion would be that (2), (7) and (10) provide the contents of all the available (sub)folders (just "Featured" for journals) and put them in appropriately named directories: (3), (8) and (12) would just produce a flat list: Not quite sure what to do, but the way things currently work is not consistent and defies expectations quite a bit. |
Not too surprising, probably. But as a check to differentiate between users and groups?
But the most important part is (6) (All deviations in this gallery folder), as long as this also works on group folders, because that's probably all that is relevant here. And using multiple URLs for the distinct group folders isn't asking too much, I'd say.
Again, not too surprising, this is what I saw with some profiles on the web site. DeviantArt allows its users to create any folder structure and then to add any deviation-objects at will to each of them. Now using gallery-dl to download all folders there then naturally results in duplicates. But the I agree on your suggestion in regard to points (2), (7) and (10) vs. (3), (8) and (12). With regard to (4) and (5): What initially made me think about requiring a consensus for the defaults in the first place is the inconsistency you mention in (7), my first thought was that maybe using A small remark to (13) and (14): Nothing wrong with that suggestion, but alternatively a separate category directory could be used here instead, directly below the top level, i.e.
|
|
I tried to adjust the default directories to a more sane default and make them work the same for both users and groups (af9bd17). It's not exactly the same as I suggested, but I think I like this version more.
So Everything, except getting all images of a gallery (e.g.
That is correct. Adding folder-structure information for this might be tricky and DeviantArt's end since images can be in multiple folders at once.
There is only a folder entry for "Featured" (and all the other user-created folders), but not for "All", "Scraps" or any special Category Path. Someone on the DeviantArt-API issue tracker mentioned that the "Scraps" entries where previously included in the output of
Sorry for not explaining this good enough, but that is not exactly what I meant. I also forgot that this isn't an issue on Windows, so you probably haven't noticed the problem with this. |
For groups the 'GalleryExtractor' collects all gallery-folder URLs and defers its work to the 'FolderExtractor'.
Very good news. So the default directory structure would be like this?
But without any deviation-items in the
Ah, okay. I assumed that
* : Depends on your setting of Damn, Github's columns not wide enough. I think I should split the second half of the table into a separate table below. Let me know if anything is not correct, I'll update it accordingly. |
|
Everything in your table seems to be correct, as is your assumption about the directory structure (a flat structure for group-images doesn't exist and doesn't make too much sense either). I've also added the Regarding the initial topic of this issue: everything that works for users now also works for groups, even group.deviantart.com/gallery/. |
|
Good to hear, will do some tests in the next days.. So the Basically the same approach as with all endpoints in the table above that use a |
|
Yes, pretty much that. |
|
With the two-step method, one sub-extractor delegating the tasked URL to another sub-extractor, which format settings get used, if specified (Directory, Filename)? Always the last one (i.e. 2nd sub-extractor)? |
|
Yes, always the last one. You can also manually reproduce the "delegation" by using the # write gallery-folder URLs to file
$ gallery-dl -g http://adoptik.deviantart.com/ > urls
# download their images
$ gallery-dl -i urlsgallery-dl basically does the same thing internally. |
|
Uh, almost forgot. I made a test run few days ago with the initial example ( But I'll count it as an success anyway 😄 One little thing, but I guess this again is the API response difference seen before: Another question: What is the keyword difference for Some accounts also have names in other languages (Example: Not sure how the API handles this.. |
This happens every time an extractor offloads its work to other extractors and doesn't provide any metadata itself. Some sort of info-message would be appropriate here, I think.
That is exactly it: the letter case.
The API says that this user's username is just {
"user": {
"userid": "E9A92126-5927-365C-80D9-3D0E1783790F",
"username": "arsenixc",
"usericon": "http://a.deviantart.net/avatars/a/r/arsenixc.jpg?5",
"type": "regular"
},
"profile_url": "http://arsenixc.deviantart.com",
"real_name": "アルス",
"tagline": "~~~",
} |
|
Yup, I thought for a moment that this could eventually be useful as a keyword, because file systems should not be an issue anymore with lack of Unicode support, but I later realized that many accounts on DeviantArt don't use this real name at all, so this would be pretty pointless anyway. |
|
Thinking about directory structures, I remembered this bit here:
So, for a group the result is an error with |
|
That is what I have been using ever since thinking about this again because of |
|
Understood. If my rudimentary knowledge of Python doesn't fail me again, the check is called from here in the gallery-dl/gallery_dl/extractor/deviantart.py Lines 176 to 181 in dcf42c5 Line 177, to be exact. And gallery-dl always ends up in And in What I'm trying to get at, basically, is right now the output options for groups cannot be changed, they use the setting for PS. |
No, this is just part of the extractor-"infrastructure" used by gallery-dl and independent of the language used. The important line for URL-delegation is line 40: ( gallery-dl/gallery_dl/extractor/deviantart.py Lines 39 to 41 in 9aa95fb ... which then ends up here ... Lines 136 to 140 in 9aa95fb ... and starts a new download job using another sub-extractor.
That would actually be possible, but it would use the settings and formats of the "parent"-extractor, which might or might not be useful. To address your actual issue, i.e. the possibility to configure different paths for users and groups: Directory- and filename-formats as well as (sub)category values are bound to the extractor class being used, but I could create group-specific classes and "delegate" (yes, again) from the main classes to these new ones (GalleryExtractor to GroupGalleryExtractor). This would allow for different configuration options to be a possibility. |
This is done by prepending "group-" to an extractor's subcategory
if the URL belongs to a group ("folder" becomes "group-folder" and
so on). This changes the configuration-path being used and is also
reflected in the output of '--list-keywords'.
|
Ok, I went with a different approach which yields the same results: subcategory values for groups now get prepended with "group-", which allows you to set different output options for group-related URLs. edit: |
|
Good idea, thanks for changing this. Changed my config to this now: "deviantart":
{
"gallery":
{
"directory": ["DeviantArt", "Galleries", "{author[urlname]}"],
"filename": "{index}_{title}.{extension}"
},
"favorite":
{
"directory": ["DeviantArt", "Favorites", "{username}"],
"filename": "{index}_{title}_by_{author[username]}.{extension}"
},
"deviation":
{
"directory": ["DeviantArt", "Deviations", "{author[username]}"],
"filename": "{index}_{title}_by_{author[username]}.{extension}"
},
"folder":
{
"directory": ["DeviantArt", "Galleries", "{folder[owner]}", "Folders", "{folder[title]}"],
"filename": "{index}_{title}.{extension}"
},
"group-folder":
{
"directory": ["DeviantArt", "Groups", "{folder[owner]}", "Folders", "{folder[title]}"],
"filename": "{index}_{title}.{extension}"
},
"collection":
{
"directory": ["DeviantArt", "Favorites", "{collection[owner]}", "Collections", "{collection[title]}"],
"filename": "{index}_{title}_by_{author[username]}.{extension}"
},
(Clipped at bit at the end) Just to be clear: this only affects two possible categories, that can now be prepended with |
|
It also affects This change causes the username part of the initial URL ( |
|
Okay, changed it again. The full "deviantart":
{
"gallery":
{
"directory": ["DeviantArt", "Galleries", "{author[urlname]}"],
"filename": "{index}_{title}.{extension}"
},
"folder":
{
"directory": ["DeviantArt", "Galleries", "{folder[owner]}", "Folders", "{folder[title]}"],
"filename": "{index}_{title}.{extension}"
},
"favorite":
{
"directory": ["DeviantArt", "Favorites", "{username}"],
"filename": "{index}_{title}_by_{author[username]}.{extension}"
},
"collection":
{
"directory": ["DeviantArt", "Favorites", "{collection[owner]}", "Collections", "{collection[title]}"],
"filename": "{index}_{title}_by_{author[username]}.{extension}"
},
"journal":
{
"directory": ["DeviantArt", "Journals", "{username}"],
"filename": "{index}_{title}.{extension}"
},
"group-gallery":
{
"directory": ["DeviantArt", "Groups", "{username}"],
"filename": "{index}_{title}_by_{author[username]}.{extension}"
},
"group-folder":
{
"directory": ["DeviantArt", "Groups", "{folder[owner]}", "Folders", "{folder[title]}"],
"filename": "{index}_{title}_by_{author[username]}.{extension}"
},
"group-favorite":
{
"directory": ["DeviantArt", "Groups", "{username}", "Favorites"],
"filename": "{index}_{title}_by_{author[username]}.{extension}"
},
"group-collection":
{
"directory": ["DeviantArt", "Groups", "{collection[owner]}", "Favorites", "{collection[title]}"],
"filename": "{index}_{title}_by_{author[username]}.{extension}"
},
"group-journal":
{
"directory": ["DeviantArt", "Groups", "{username}", "Journals"],
"filename": "{index}_{title}.{extension}"
},
"deviation":
{
"directory": ["DeviantArt", "Deviations", "{category_path}"],
"filename": "{index}_{title}_by_{author[username]}.{extension}"
},
"flat": true,
"mature": true
},I think this should do it.. 😄 |
gallery-dl --versionOS: Windows 10 CU x64
Python 3.6.1. x64
Found something on DeviantArt (again) 😄
Groups..
1: Home URL of a group
http://cgpinups.deviantart.com/2: Gallery URL of group
http://cgpinups.deviantart.com/gallery/Same result as above.
3: Gallery Folders in the Group (The actual galleries, so to speak) (One example of them)
http://cgpinups.deviantart.com/gallery/25871850/Fantasy-and-Sci-FiI think the question is how the API handles this stuff.
It doesn't make sense to me right now, but I guess this is related to the difference between Gallery and Favourites. If this is the same distinction by the API...
For example, a user's gallery has Gallery Folders (just as a group), while the user's favourites has collections.
Gallery folders also don't work.
Example:
No point in having two separate issues here, I think. Depends on the API results, I guess.
The text was updated successfully, but these errors were encountered: