yaplog.jp Errors

[schleifen]

Hello,
Firstly thank you for adding yaplog to the supported websites. It is closing on 31th January 2019 and this tool has been an immense help in trying to save blogs. But there are a few errors I noticed during the downloading:

• blogs with "-" in the blog name don't get recognized and give the error message "No suitable extractor found for link" (example: https://yaplog.jp/a-pierrot-o/)

• certain image links don't get recognized with the error message "URLs are not supported/enabled" or "No suitable extractor found for imagelink" or they appear downloaded with a link that has underlines where slashes should be (example structure: yaplog.jp/cv/blogname/img/image number in the blog?/imagename.jpg ; actual example: https://yaplog.jp/cv/msjane/img/246/img100105225224_t.jpg)

• after successfully downloading about 2000 images of a blog, the following error gets displayed every time: "An unexpected error occurred: IndexError - list assignment index out of range. Please run gallery-dl again with the --verbose flag, copy its output and report this issue on https://github.com/mikf/gallery-dl/issues"
  I did that with an example and got the following:
  [gallery-dl][debug] Version 1.10.5
  [gallery-dl][debug] Python 3.7.4 - Windows-10-10.0.17134-SP0
  [gallery-dl][debug] requests 2.22.0 - urllib3 1.25.3
  [gallery-dl][debug] Starting DownloadJob for 'http://yaplog.jp/f_l_a_s_c_o/'
  [gallery-dl][debug] Updating urllib3 ciphers
  [yaplog][debug] Using YaplogBlogExtractor for 'http://yaplog.jp/f_l_a_s_c_o/'
  [urllib3.connectionpool][debug] Starting new HTTPS connection (1): yaplog.jp:443
  It starts (successfully?) scanning many pictures with the message like the following "# .\gallery-dl\yaplog\f_l_a_s_c_o\1023_EXCENTRIQUE◆新入荷・コルセットスカート_8200.jpg
  [urllib3.connectionpool][debug] https://yaplog.jp:443 "GET /f_l_a_s_c_o/image/1023/8205 HTTP/1.1" 200 None" but then there appears this:
  [yaplog][error] An unexpected error occurred: IndexError - list assignment index out of range. Please run gallery-dl again with the --verbose flag, copy its output and report this issue on https://github.com/mikf/gallery-dl/issues .
  [yaplog][debug]
  Traceback (most recent call last):
  File "file path\python\python37-32\lib\site-packages\gallery_dl\job.py", line 47, in run
  for msg in self.extractor:
  File "file path\python\python37-32\lib\site-packages\gallery_dl\extractor\common.py", line 363, in iter
  raise msg
  File "file path\python\python37-32\lib\site-packages\gallery_dl\extractor\common.py", line 369, in async_items
  for msg in self.items():
  File "file path\python\python37-32\lib\site-packages\gallery_dl\extractor\yaplog.py", line 29, in items
  for post, urls in self.posts():
  File file path\python\python37-32\lib\site-packages\gallery_dl\extractor\yaplog.py", line 88, in posts
  url, images, data = self.parse_post(url)
  File "file path_\python\python37-32\lib\site-packages\gallery_dl\extractor\yaplog.py", line 55, in _parse_post
  urls[0] = page # cache HTML of first page
  IndexError: list assignment index out of range

Is there any solution?

[mikf]

Problems number 1 and 2 should be fixed (a782b00, 93aac8d). Number 3 might be because you are "overusing" the site and it sends an error message instead of the expected webpage. I'll look into that tomorrow, but maybe adding a little wait-time between downloads with --sleep might help.

[mikf]

As for your third point: gallery-dl didn't handle empty posts like https://yaplog.jp/f_l_a_s_c_o/image/872 properly and crashed, but that should now be fixed as well.

I hope that covers everything. Let me know if you find anything else.

[schleifen]

Amazing, thank you! Almost everything works now. Only pictures with a "_" in the image name still get skipped like in the second point where the link turns slashes into underlines, for example https://yaplog.jp/cv/pintuks/img/12/20080404t_p.jpg .

[mikf]

Should be fixed with d4ffd6c and the new release.

The issue wasn't a _ in filenames, but the fact that 20080404t for example couldn't be converted to a numeric ID and ended up as 0 instead.