e-hentai.org gives back " You do not have permission to access the resource at" and can't dl certain tags #356

Hentai4me · 2019-07-27T01:59:15Z

There are two issues that are present but I think they are connected in how they act:
First one is that I can't download any favorites from e-hentai.org/favorites.php.

Since exhentai for now is dead, using e-hentai is much more cumbersome since it has weird rules in place in that only some users can access certain content.
This makes it so that gallery-dl can't access them but a typical web browser can.
I have username + password in gallery-dl.conf

[gallery-dl][debug] Version 1.9.0
[gallery-dl][debug] Python 3.7.3 - Windows-10-10.0.17134
[gallery-dl][debug] requests 2.22.0 - urllib3 1.24.3
[gallery-dl][debug] Starting DownloadJob for 'https://e-hentai.org/?f_search=[redacted]'
[gallery-dl][debug] Updating urllib3 ciphers
[exhentai][debug] Using ExhentaiSearchExtractor for 'https://e-hentai.org/?f_search=[redacted]'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): e-hentai.org:443
[urllib3.connectionpool][debug] https://e-hentai.org:443 "GET /?f_search=[redacted]&page=0 HTTP/1.1" 200 2064

But download any typical tag that doesn't require user registration (at least) works fine.

The text was updated successfully, but these errors were encountered:

mikf · 2019-07-27T09:11:28Z

Logging in with username and password only caches the ipb_member_id and ipb_pass_hash cookies. The other cookies are then supposed to be filled in during the first request to e-hentai, but that might not be enough to grant access to "certain" tags.

You could try putting all of your e-hentai cookies in your config file so your "special" permissions are known to the e-hentai servers beforehand:

        "exhentai":
        {
            "cookies":
            {
                "ipb_member_id": "123",
                "ipb_pass_hash": "foo",
                "sk": "bar",
                ...
            }
        }

Hentai4me · 2019-07-27T18:48:25Z

Well the problem there then is that I get a TypeError:

[exhentai][error] An unexpected error occurred: TypeError - expected string or bytes-like object. Please run gallery-dl again with the --verbose flag, copy its output and report this issue on https://github.com/mikf/gallery-dl/issues .
[exhentai][debug]
Traceback (most recent call last):
  File "gallery_dl\job.py", line 47, in run
  File "gallery_dl\extractor\exhentai.py", line 355, in items
  File "gallery_dl\extractor\exhentai.py", line 50, in request
  File "gallery_dl\extractor\common.py", line 80, in request
  File "site-packages\requests\sessions.py", line 519, in request
  File "site-packages\requests\sessions.py", line 462, in prepare_request
  File "site-packages\requests\models.py", line 315, in prepare
  File "site-packages\requests\models.py", line 572, in prepare_cookies
  File "site-packages\requests\cookies.py", line 142, in get_cookie_header
  File "http\cookiejar.py", line 1358, in add_cookie_header
  File "http\cookiejar.py", line 1317, in _cookie_attrs
TypeError: expected string or bytes-like object

This is with ipb_member_id, ipb_pass_hash and sk

mikf · 2019-07-27T18:57:27Z

TypeError: expected string or bytes-like object

Are you sure those values are actual strings, enclosed in double quotes and everything?
Maybe post your anonymized exhentai-config here, so we can see what may be wrong there.

(I'm using the cookies-in-config method myself and it's working as it should)

Hrxn · 2019-07-27T20:30:17Z

Maybe the cookies have not been "exported" properly?

What would you recommend? Still that old and trusty cookies.txt browser extension, for example?

Hentai4me · 2019-07-27T20:57:10Z

Here's my config:

{
    "extractor":
    {
        "base-directory": "./gallery-dl/",
        "postprocessors": null,
        "archive": null,
        "cookies": 
        {
            "exhentai":
            {
                "cookies":
                {
                "ipb_member_id": "id",
                "ipb_pass_hash": "hash",
                "sk": "characters"
                }
            }
        },
        "proxy": null,
        "skip": true,
        "sleep": 0,
        "user-agent": "Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0",
        "restrict-filenames": "auto",

        "artstation":
        {
            "external": false
        },
        "danbooru":
        {
            "username": null,
            "password": null
        },
        "deviantart":
        {
            "refresh-token": null,
            "extra": false,
            "flat": true,
            "folders": false,
            "journals": "html",
            "mature": true,
            "metadata": false,
            "original": true,
            "wait-min": 0
        },
        "exhentai":
        {
            "username": null,
            "password": null,
            "original": true,
            "wait-min": 3.0,
            "wait-max": 6.0
        },
        "flickr":
        {
            "access-token": null,
            "access-token-secret": null,
            "videos": true,
            "size-max": null
        },
        "gelbooru":
        {
            "api": true
        },
        "gfycat":
        {
            "format": "mp4"
        },
        "idolcomplex":
        {
            "username": null,
            "password": null,
            "wait-min": 3.0,
            "wait-max": 6.0
        },
        "imgur":
        {
            "mp4": true
        },
        "kissmanga":
        {
            "captcha": "stop"
        },
        "nijie":
        {
            "username": null,
            "password": null
        },
        "oauth":
        {
            "browser": true
        },
        "pixiv":
        {
            "username": null,
            "password": null,
            "ugoira": true
        },
        "reactor":
        {
            "wait-min": 3.0,
            "wait-max": 6.0
        },
        "readcomiconline":
        {
            "captcha": "stop"
        },
        "recursive":
        {
            "blacklist": ["directlink", "oauth", "recursive", "test"]
        },
        "reddit":
        {
            "refresh-token": null,
            "comments": 500,
            "morecomments": false,
            "date-min": 0,
            "date-max": 253402210800,
            "date-format": "%Y-%m-%dT%H:%M:%S",
            "id-min": "0",
            "id-max": "zik0zj",
            "recursion": 0,
            "user-agent": "Python:gallery-dl:0.8.4 (by /u/mikf1)"
        },
        "sankaku":
        {
            "username": null,
            "password": null,
            "wait-min": 3.0,
            "wait-max": 6.0
        },
        "seiga":
        {
            "username": null,
            "password": null
        },
        "tumblr":
        {
            "avatar": false,
            "external": false,
            "inline": true,
            "posts": "all",
            "reblogs": true
        },
        "twitter":
        {
            "content": false,
            "retweets": true,
            "videos": false
        },
        "wallhaven":
        {
            "api-key": null
        },
        "booru":
        {
            "tags": false
        }
    },

    "downloader":
    {
        "part": true,
        "part-directory": null,

        "http":
        {
            "mtime": true,
            "rate": null,
            "retries": 4,
            "timeout": 30.0,
            "verify": true
        },

        "ytdl":
        {
            "format": null,
            "forward-cookies": true,
            "mtime": true,
            "rate": null,
            "retries": 4,
            "timeout": 30.0,
            "verify": true
        }
    },

    "output":
    {
        "mode": "auto",
        "progress": true,
        "shorten": true,
        "log": "[{name}][{levelname}] {message}",
        "logfile": null,
        "unsupportedfile": null
    },

    "netrc": false
}

mikf · 2019-07-27T21:01:28Z

Delete

        "cookies": 
        {
            "exhentai":
            {
                "cookies":
                {
                "ipb_member_id": "id",
                "ipb_pass_hash": "hash",
                "sk": "characters"
                }
            }
        },

And put the cookies part in the middle in the already existing exhentai section

        "exhentai":
        {
            "username": null,
            "password": null,
            "original": true,
            "wait-min": 3.0,
            "wait-max": 6.0,
            "cookies":
            {
                "ipb_member_id": "id",
                "ipb_pass_hash": "hash",
                "sk": "characters"
            }
        },

mikf · 2019-07-27T21:06:51Z

Also: if your browser has any more e-hentai cookies than those three, I would advise you to put them in there as well.

Another, maybe safer, way to achieve the same thing would be to export your browser cookies with a browser addon, as HRXN mentioned, and point the cookies setting to the resulting cookies.txt file

        "exhentai":
        {
            "username": null,
            "password": null,
            "original": true,
            "wait-min": 3.0,
            "wait-max": 6.0,
            "cookies": "C:\\path\\to\\cookies.txt"
        },

Hentai4me · 2019-07-28T08:12:25Z

After exporting my cookie to cookies.txt (via cookies.txt extension) and added it as a path instead like this:

{
    "extractor":
    {
        "base-directory": "./gallery-dl/",
        "postprocessors": null,
        "archive": null,
        "cookies": null,
        "proxy": null,
        "skip": true,
        "sleep": 0,
        "user-agent": "Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0",
        "restrict-filenames": "auto",

        "artstation":
        {
            "external": false
        },
        "danbooru":
        {
            "username": null,
            "password": null
        },
        "deviantart":
        {
            "refresh-token": null,
            "extra": false,
            "flat": true,
            "folders": false,
            "journals": "html",
            "mature": true,
            "metadata": false,
            "original": true,
            "wait-min": 0
        },
        "exhentai":
        {
            "username": "null",
            "password": "null",
            "original": true,
            "wait-min": 3.0,
            "wait-max": 6.0,
            "cookies": "C:\\Path\\cookies.txt"
        },
        "flickr":
        {
            "access-token": null,
            "access-token-secret": null,
            "videos": true,
            "size-max": null
        },
        "gelbooru":
        {
            "api": true
        },
        "gfycat":
        {
            "format": "mp4"
        },
        "idolcomplex":
        {
            "username": null,
            "password": null,
            "wait-min": 3.0,
            "wait-max": 6.0
        },
        "imgur":
        {
            "mp4": true
        },
        "kissmanga":
        {
            "captcha": "stop"
        },
        "nijie":
        {
            "username": null,
            "password": null
        },
        "oauth":
        {
            "browser": true
        },
        "pixiv":
        {
            "username": null,
            "password": null,
            "ugoira": true
        },
        "reactor":
        {
            "wait-min": 3.0,
            "wait-max": 6.0
        },
        "readcomiconline":
        {
            "captcha": "stop"
        },
        "recursive":
        {
            "blacklist": ["directlink", "oauth", "recursive", "test"]
        },
        "reddit":
        {
            "refresh-token": null,
            "comments": 500,
            "morecomments": false,
            "date-min": 0,
            "date-max": 253402210800,
            "date-format": "%Y-%m-%dT%H:%M:%S",
            "id-min": "0",
            "id-max": "zik0zj",
            "recursion": 0,
            "user-agent": "Python:gallery-dl:0.8.4 (by /u/mikf1)"
        },
        "sankaku":
        {
            "username": null,
            "password": null,
            "wait-min": 3.0,
            "wait-max": 6.0
        },
        "seiga":
        {
            "username": null,
            "password": null
        },
        "tumblr":
        {
            "avatar": false,
            "external": false,
            "inline": true,
            "posts": "all",
            "reblogs": true
        },
        "twitter":
        {
            "content": false,
            "retweets": true,
            "videos": false
        },
        "wallhaven":
        {
            "api-key": null
        },
        "booru":
        {
            "tags": false
        }
    },

    "downloader":
    {
        "part": true,
        "part-directory": null,

        "http":
        {
            "mtime": true,
            "rate": null,
            "retries": 4,
            "timeout": 30.0,
            "verify": true
        },

        "ytdl":
        {
            "format": null,
            "forward-cookies": true,
            "mtime": true,
            "rate": null,
            "retries": 4,
            "timeout": 30.0,
            "verify": true
        }
    },

    "output":
    {
        "mode": "auto",
        "progress": true,
        "shorten": true,
        "log": "[{name}][{levelname}] {message}",
        "logfile": null,
        "unsupportedfile": null
    },

    "netrc": false
}

I'm getting image page not found error:
[exhentai][error] The image page at 'https://e-hentai.org/g/[num]/[hash]' does not exist

mikf · 2019-07-28T08:37:40Z

This can most likely solved by setting another cookie. I've gotten the same (?) error myself yesterday, and adding "nw": "1" to my list of cookies "fixed" it.

Since you are using a cookies.txt file you need to open it, duplicate any e-hentai cookie line, and set its name and value to nw and 1.

edit: dc73d02 now automatically sets this cookie

Hentai4me · 2019-07-28T10:17:36Z

Hmm this is a bit confusing then.

So if I use the latest version I don't need to edit it manually?

If that is the case, I tried that but I get the exact same error (and I'm not banned or anything).

also I noticed that for .e-hentai.org I don't have any nw: 1 yet on e-hentai.org I got tons of nw: 1 but no iqd.

mikf · 2019-07-28T15:42:10Z

So if I use the latest version I don't need to edit it manually?

With the latest git snapshot, but not the latest version on PyPI. dc73d02 hasn't been "officially" released yet.

Hmm this is a bit confusing then.

All you have to do is open your cookies file in a text editor and go to the first line with .e-hentai.org as domain

.e-hentai.org	TRUE	/	FALSE	1595863995	ipb_member_id	123

Use copy and paste to duplicate it

.e-hentai.org	TRUE	/	FALSE	1595863995	ipb_member_id	123
.e-hentai.org	TRUE	/	FALSE	1595863995	ipb_member_id	123

And change name and value of the second line to nw and 1

.e-hentai.org	TRUE	/	FALSE	1595863995	ipb_member_id	123
.e-hentai.org	TRUE	/	FALSE	1595863995	nw	1

(All entries need to be separated by tab stops. Don't accidentally overwrite them)

Hentai4me · 2019-07-28T16:19:38Z

I see, so even if I used:
pip install --upgrade https://github.com/mikf/gallery-dl/archive/master.zip
I haven't gotten the latest commit?

Also I did as you said:

.e-hentai.org	TRUE	/	FALSE	1595671110	ipb_member_id	123
.e-hentai.org	TRUE	/	FALSE	1595671110	nw	1
.e-hentai.org	TRUE	/	FALSE	1595671110	ipb_pass_hash	hash123
.e-hentai.org	TRUE	/	FALSE	1595671113	sk	characters
.e-hentai.org	TRUE	/	FALSE	1595671113	star	text
.e-hentai.org	TRUE	/	FALSE	1595674160	hath_perks	text

But I'm still getting the same error.

mikf · 2019-07-28T16:42:09Z

No, that gets you the latest commit.

Hmm, ok, then I really don't know why you can't access that gallery, because any HTTP request by gallery-dl should be the exact same as from a webbrowser.

As a last resort, could you add a print(page) under line 243 in exhentai.py

gallery-dl/gallery_dl/extractor/exhentai.py

Line 243 in dc73d02

page = response.text

and send me the results (and maybe an uncensored verbose log when accessing this gallery) per email or as private message on Gitter

I don't have a star account myself and can there not reproduce your issue. Understanding what exactly is going wrong here would be nice.

Hentai4me · 2019-07-28T19:31:06Z

I've sent a message on gitter.

mikf · 2019-07-28T20:38:07Z

Thank you. I even managed to find out what is causing those problems: The multi page viewer and the website layout that comes with it. I'm not sure I can make it work with the MPV enabled since I don't have access to this feature myself, but disabling it on your end will definitely help.

Hentai4me · 2019-07-29T03:52:28Z

Ah I see, thanks I'll check it out shortly.
Edit:
As you said it was the MPV.

- warn on MPV-enabled galleries - catch parsing errors for gallery pages and image info - write page content to debug output

mikf · 2019-08-01T22:10:18Z

v1.10.0 is out and logging in with username & password now (hopefully) gets all cookies necessary to access restricted galleries. Exporting your cookies should no longer be necessary. (6ce22f6)
Gallery pages in MPV mode are now detected and warned about instead of causing a crash. (81b35ed)

mikf mentioned this issue Jul 27, 2019

[e-hentai] "The image page at [gallery url] does not exist" #358

Closed

mikf added a commit that referenced this issue Jul 29, 2019

[exhentai] catch more error states (#356, #360)

81b35ed

- warn on MPV-enabled galleries - catch parsing errors for gallery pages and image info - write page content to debug output

mikf closed this as completed Aug 1, 2019

mikf mentioned this issue Aug 3, 2019

[exhentai][error] Authentication failed... #370

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

e-hentai.org gives back " You do not have permission to access the resource at" and can't dl certain tags #356

e-hentai.org gives back " You do not have permission to access the resource at" and can't dl certain tags #356

Hentai4me commented Jul 27, 2019

mikf commented Jul 27, 2019

Hentai4me commented Jul 27, 2019

mikf commented Jul 27, 2019

Hrxn commented Jul 27, 2019

Hentai4me commented Jul 27, 2019

mikf commented Jul 27, 2019

mikf commented Jul 27, 2019

Hentai4me commented Jul 28, 2019 •

edited

Loading

mikf commented Jul 28, 2019 •

edited

Loading

Hentai4me commented Jul 28, 2019 •

edited

Loading

mikf commented Jul 28, 2019

Hentai4me commented Jul 28, 2019 •

edited

Loading

mikf commented Jul 28, 2019

Hentai4me commented Jul 28, 2019

mikf commented Jul 28, 2019 •

edited

Loading

Hentai4me commented Jul 29, 2019 •

edited

Loading

mikf commented Aug 1, 2019

e-hentai.org gives back " You do not have permission to access the resource at" and can't dl certain tags #356

e-hentai.org gives back " You do not have permission to access the resource at" and can't dl certain tags #356

Comments

Hentai4me commented Jul 27, 2019

mikf commented Jul 27, 2019

Hentai4me commented Jul 27, 2019

mikf commented Jul 27, 2019

Hrxn commented Jul 27, 2019

Hentai4me commented Jul 27, 2019

mikf commented Jul 27, 2019

mikf commented Jul 27, 2019

Hentai4me commented Jul 28, 2019 • edited Loading

mikf commented Jul 28, 2019 • edited Loading

Hentai4me commented Jul 28, 2019 • edited Loading

mikf commented Jul 28, 2019

Hentai4me commented Jul 28, 2019 • edited Loading

mikf commented Jul 28, 2019

Hentai4me commented Jul 28, 2019

mikf commented Jul 28, 2019 • edited Loading

Hentai4me commented Jul 29, 2019 • edited Loading

mikf commented Aug 1, 2019

Hentai4me commented Jul 28, 2019 •

edited

Loading

mikf commented Jul 28, 2019 •

edited

Loading

Hentai4me commented Jul 28, 2019 •

edited

Loading

Hentai4me commented Jul 28, 2019 •

edited

Loading

mikf commented Jul 28, 2019 •

edited

Loading

Hentai4me commented Jul 29, 2019 •

edited

Loading