Nodriver proxy with authentication

[strukiii]

Hi all, I'm putting this here since there is no nodriver discussions section. I'm trying to get the browser setup with an authenticated proxy but chrome does not support credentials via browser args. I found this, where the answer from Arkadiy Bolotov is interesting. There are the same cdp methods as in his post available in Network though they are deprecated for those in Fetch. Here is my current attempt I put together this afternoon, which does intercept the requests however no authRequired events are fired as the documentation suggests.

class Scraper:
    main_tab: tab

    def __init__(self):
        loop().run_until_complete(self.run())

    async def run(self):
        browser = await start(
            browser_args=[f"--proxy-server={PROXY}"],
        )
        self.main_tab = browser.main_tab
        await self.main_tab.send(cdp.fetch.enable(handle_auth_requests=True))
        self.main_tab.add_handler(cdp.fetch.RequestPaused, self.req_paused)
        self.main_tab.add_handler(cdp.fetch.AuthRequired, self.auth_challenge_handler)
        page = await browser.get('https://www.whatismyip.com/')
        await asyncio.sleep(6000)


    async def auth_challenge_handler(self, event: cdp.fetch.AuthRequired):
        # Split the credentials
        # Respond to the authentication challenge
        await self.main_tab.send(cdp.fetch.continue_with_auth(
            request_id=event.request_id,
            auth_challenge_response=cdp.fetch.AuthChallengeResponse(
                response="ProvideCredentials",
                username=USERNAME,
                password=PASSWORD
            )
        ))


    async def req_paused(self, event: cdp.fetch.RequestPaused):
        await self.main_tab.send(cdp.fetch.continue_request(request_id=event.request_id))


if __name__ == '__main__':
    Scraper()

[WindBora]

Hi

Here 2 problems arises:

1. You add handlers after enabling domain (internal implementation overrides enabling domain) - that's the reason why handlers do not receive an event
2. When you will finally receive an event, your response will block all processes

Solutions:

1. Add handlers before enabling domain. Only after adding them do enable domain
2. For such blocking requests there is a must to just use fire-and-forget method e.g. do not await for sending response on req_paused or auth_required. Just wrap the call into asyncio.create_task

Example of working code based of of yours:

class Scraper:
    main_tab: uc.Tab

    def __init__(self):
        uc.loop().run_until_complete(self.run())

    async def run(self):
        browser = await uc.start(
            browser_args=[f"--proxy-server={PROXY}"],
        )
        self.main_tab = await browser.get("draft:,")
        self.main_tab.add_handler(uc.cdp.fetch.RequestPaused, self.req_paused)
        self.main_tab.add_handler(
            uc.cdp.fetch.AuthRequired, self.auth_challenge_handler
        )
        await self.main_tab.send(uc.cdp.fetch.enable(handle_auth_requests=True))
        page = await browser.get("https://www.whatismyip.com/")
        await asyncio.sleep(6000)

    async def auth_challenge_handler(self, event: uc.cdp.fetch.AuthRequired):
        # Split the credentials
        # Respond to the authentication challenge
        asyncio.create_task(
            self.main_tab.send(
                uc.cdp.fetch.continue_with_auth(
                    request_id=event.request_id,
                    auth_challenge_response=uc.cdp.fetch.AuthChallengeResponse(
                        response="ProvideCredentials",
                        username=USERNAME,
                        password=PASSWORD,
                    ),
                )
            )
        )

    async def req_paused(self, event: uc.cdp.fetch.RequestPaused):
        asyncio.create_task(
            self.main_tab.send(
                uc.cdp.fetch.continue_request(request_id=event.request_id)
            )
        )


if __name__ == "__main__":
    Scraper()

[mushifali]

import asyncio
import nodriver as uc

These should be enough.

[tidal10]

Do you have a way to run multiple proxy sessions on a threaded environment with different proxies?

[epicwhale]

This works for me with non-ssl domains/pages, but fails when it's an https:// url I am trying to access. I get the following (screenshot below)...

Wondering is there some flag or extra args you needed to set somewhere to workaround this by any chance? @mushifali @strukiii @WindBora

[mushifali]

@epicwhale It is also working for https:// domains for me. Can you please share your code?

[epicwhale]

@mushifali looks like it was an issue with the proxy I was using, thanks for confirming! A better proxy seemed to do it for me too.

[SSujitX]

import nodriver as uc
from nodriver.cdp import fetch

PROXYHOST = "aaaaaaaaaaaaaa.ika.na.pyproxy.io"
PROXYPORT = "2000"
PROXYUSERNAME = "aaa"
PROXYPASSWORD = "bbbb"
PROXY = f"http://{PROXYHOST}:{PROXYPORT}"

async def setup_proxy(username, password, tab):
    async def auth_challenge_handler(event: fetch.AuthRequired):
        # Respond to the authentication challenge
        await tab.send(
            fetch.continue_with_auth(
                request_id=event.request_id,
                auth_challenge_response=fetch.AuthChallengeResponse(
                    response="ProvideCredentials",
                    username=username,
                    password=password,
                ),
            )
        )

    async def req_paused(event: fetch.RequestPaused):
        # Continue with the request
        await tab.send(fetch.continue_request(request_id=event.request_id))

    # Add handlers for fetch events
    tab.add_handler(
        fetch.RequestPaused, lambda event: asyncio.create_task(req_paused(event))
    )
    tab.add_handler(
        fetch.AuthRequired,
        lambda event: asyncio.create_task(auth_challenge_handler(event)),
    )

    # Enable fetch domain with auth requests handling
    await tab.send(fetch.enable(handle_auth_requests=True))


    # -------- PROXY -------- #
    browser = await uc.start(browser_args=[f"--proxy-server={PROXY}"])
    main_tab = await browser.get("draft:,")
    await setup_proxy(PROXYUSERNAME, PROXYPASSWORD, main_tab)
    ipPage = await browser.get("https://www.myexternalip.com/raw")
    await asyncio.sleep(2)
    ip = await ipPage.evaluate("document.body.textContent.trim()")
    print(f">= IP Address: {ip}")
    # -------- PROXY -------- #

[aoddspotato]

This is awesome