Favorites Parser Occasionally Fails with Error InvalidChunkLength

[oldmenewme]

I am using the favorites parser with the options -F --page-all. Every now and then, I get the following error for some pages.
Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read)

The number of doujins detected at the beginning favorites_parser: You have X favorites in Y pages. will not match with the final results at the end print_doujinshi: Search Result || Found Z doujinshis, as I see that the code just logs the error, and then continues iterating through all the pages.

Also, even though I get the error above for some pages (say 2 pages), the number of found doujins and found pages will have a difference of more than 2*25, even though the error was logged only twice.

[oldmenewme]

I tried rerunning today, and I didn't see any errors captured in logger. All pages were visited by the script, however 100 doujins were somehow skipped. Not sure if some pages have different html or something to cause the script to not count them anymore. The last time it worked for me was on 2025-02-02

[RicterZ]

My testcase:

[16:26:33] check_cookie: Login successfully! Your username: ricter-z
[16:26:33] main: You do not specify --download option
[16:26:34] favorites_parser: You have 347 favorites in 14 pages.
[16:26:34] favorites_parser: Getting doujinshi ids of page 1
[16:26:34] favorites_parser: Getting doujinshi ids of page 2
[16:26:35] favorites_parser: Getting doujinshi ids of page 3
[16:26:36] favorites_parser: Getting doujinshi ids of page 4
[16:26:37] favorites_parser: Getting doujinshi ids of page 5
[16:26:37] favorites_parser: Getting doujinshi ids of page 6
[16:26:39] favorites_parser: Getting doujinshi ids of page 7
[16:26:40] favorites_parser: Getting doujinshi ids of page 8
[16:26:41] favorites_parser: Getting doujinshi ids of page 9
[16:26:42] favorites_parser: Getting doujinshi ids of page 10
[16:26:43] favorites_parser: Getting doujinshi ids of page 11
[16:26:44] favorites_parser: Getting doujinshi ids of page 12
[16:26:44] favorites_parser: Getting doujinshi ids of page 13
[16:26:46] favorites_parser: Getting doujinshi ids of page 14
[16:26:47] print_doujinshi: Search Result || Found 347 doujinshis

Maybe I need to figure out which doujinshi been skipped in print_doujinshi

[oldmenewme]

parse failure.txt

Here's the HTML I logged. What seems to happen is that even if a page is visited, Cloudflare Error 520 can sometimes popup, resulting in 0 doujins

[oldmenewme]

I let the above run for a few more pages and it was successful between pages 3 to 6, then same Error 520 popped up for page 7

[RicterZ]

Does error 520 persists at page 7?

[RicterZ]

added a global retry option (default 3 times) for fetching favorites at commit 023c896

[RicterZ]

Try version 0.5.24

[oldmenewme]

0.5.24 works with no issues! I didn't encounter InvalidChunkLength at all recently, so maybe it was tied to the error 520. Thanks again for the changes