support multi-page URL #79
Conversation
|
Thanks for your contribution. As a general rule, you should check your code with In this case you should just add, for example ("https://chan.sankakucomplex.com/?tags=marie_rose&page=98&next=3874906", None),to the list. |
gallery_dl/extractor/idolcomplex.py
Outdated
| r"/\?(?:[^&#]*&)*tags=([^&#]+)"] | ||
| r"/\?(?:[^&#]*&)*tags=([^&#]+)" | ||
| r"((?:[^&#]*&)*page=([^&#]+))*" | ||
| r"((?:[^&#]*&)*next=([^&#]+))*"] |
There was a problem hiding this comment.
Instead of capturing every query-parameter on its own, capture the whole query string ((?:\?([^#]*)?) and parse it with text.parse_query(...)
There was a problem hiding this comment.
... OK ignore this one. What I wrote wouldn't work. Just leave it as it is.
gallery_dl/extractor/sankaku.py
Outdated
| r"/\?(?:[^&#]*&)*tags=([^&#]+)"] | ||
| r"/\?(?:[^&#]*&)*tags=([^&#]+)" | ||
| r"((?:[^&#]*&)*page=([^&#]+))*" | ||
| r"((?:[^&#]*&)*next=([^&#]+))*"] |
gallery_dl/extractor/sankaku.py
Outdated
| @@ -150,6 +152,8 @@ class SankakuTagExtractor(SankakuExtractor): | |||
| def __init__(self, match): | |||
| SankakuExtractor.__init__(self) | |||
| self.tags = text.unquote(match.group(1).replace("+", " ")) | |||
| self.start_page = 1 if match.group(3) is None else int(text.unquote(match.group(3))) | |||
There was a problem hiding this comment.
Use util.safe_int(..., 1).
You don't need to check if this particular group is None, safe_int() already handles that.
Also don't unquote something that should be a number.
gallery_dl/extractor/sankaku.py
Outdated
| @@ -150,6 +152,8 @@ class SankakuTagExtractor(SankakuExtractor): | |||
| def __init__(self, match): | |||
| SankakuExtractor.__init__(self) | |||
| self.tags = text.unquote(match.group(1).replace("+", " ")) | |||
| self.start_page = 1 if match.group(3) is None else int(text.unquote(match.group(3))) | |||
| self.next = 0 if match.group(5) is None else int(text.unquote(match.group(5))) | |||
gallery_dl/extractor/sankaku.py
Outdated
| @@ -170,6 +174,7 @@ def get_metadata(self): | |||
|
|
|||
| def get_posts(self): | |||
| params = {"tags": self.tags, "page": self.start_page} | |||
| if self.next > 0 : params["next"] = self.next | |||
There was a problem hiding this comment.
No space between condition and :. flake8 should tell you that as well.
|
Nice, thank you. The only remaining issue is the regex, which now also matches the URLs of the other two extractors. |
|
@mikf Oh, I did not see your message, it seems we still need to commit it once.😂 |
Hello, I modified
sankaku.pyandidolcomplex.py. It now supports multi-page URL like https://chan.sankakucomplex.com/?tags=marie_rose&page=98&next=3874906 (NSFW).Please review my code.