-
Hi I have 100 proxy servers (ipv6) at my disposal and for a long time I have been trying to write a program based on your class that does something like this. In a loop, it sets up a proxy server, connects to the selected page with the Cloudflate preloader (security check), when the page loads (there is immediately json on it), cookies are downloaded, saved in the database, the same is done for the next proxy, etc. target_url = 'https://www.xxxx.com/api/1.0.0/products/?file.json'
async with (webdriver.Chrome(options=options, max_ws_size=2** 60, debug=True) as driver):
driver.base_target.socket.on_closed.append(lambda code, reason: driver.quit())
user_agent = await driver.execute_async_script("return navigator.userAgent;")
await asyncio.sleep(2)
while True:
proxies = proxy_db.get_proxy_list_to_recive_cf_cookie()
# if there is no cookie to refresh, wait
if not proxies:
time.sleep(60 * 4)
continue
for proxy in proxies:
try:
ip = proxy["ip"]
port = proxy["port"]
step = step + 1
await driver.set_single_proxy(
'http://{username}:{password}@{ip}:{port}'.format(
ip=ip,
port=port,
username=proxy["username"],
password=proxy["password"].rstrip(),
)
)
await driver.get(target_url, wait_load=True, timeout=loading_timeout)
await asyncio.sleep(5)
wait_step = 1
found = False
while wait_step <= 5 and not found:
found = await driver.find_element(By.TAG_NAME, "pre")
if found:
cookies = await driver.get_cookies()
for cookies in cookies:
if cookie["name"] == "cf_clearance":
cookie_value = cookie["value"]
# save the downloaded copokie in the database
proxy_db.set_cookie_for_proxy(ip, port, cookie_value, user_agent)
break
await asyncio.sleep(0.5)
wait_step = wait_step + 1
await driver.delete_all_cookies()
await driver.clear_proxy()
except NoSuchElementException:
errors = errors + 1
continue
finally:
if timeouts_errors == 5:
driver.quit()
exit() One more question, in the case as described, where is the Cloudflate preloader with security check, what is the easiest way to wait for me to be redirected to the right page and load it? The only thing that worked for me was this: await driver.get(target_url, wait_load=True, timeout=loading_timeout)
await asyncio.sleep(5)
wait_step = 1
found = False
while wait_step <= 5 and not found:
found = await driver.find_element(By.TAG_NAME, "pre")
if found:
cookies = await driver.get_cookies()
for cookie in cookies:
if cookie["name"] == "cf_clearance":
cookie_value = cookie["value"]
# save cookiue to DB
proxy_db.set_cookie_for_proxy(ip, port, cookie_value, user_agent)
break
await asyncio.sleep(0.5)
wait_step = wait_step + 1 In plain selenium I used:
Regarding the errors I receive when the script is running, for example:
Thank you, for your help. |
Beta Was this translation helpful? Give feedback.
Replies: 4 comments 13 replies
-
@juhacz Does found = await driver.find_element(By.TAG_NAME, "pre", timeout=5) work for you? |
Beta Was this translation helpful? Give feedback.
-
Indeed. And |
Beta Was this translation helpful? Give feedback.
-
Selenium-Driverless/src/selenium_driverless/types/target.py Lines 569 to 577 in b90265b Selenium-Driverless/src/selenium_driverless/types/target.py Lines 569 to 576 in bdf9a7f Are you sure the dev-version has been installed correctly with those changes? If yes, then I'm pretty sure it either works or there's a different error. |
Beta Was this translation helpful? Give feedback.
-
@kaliiiiiiiiii https://webshare.io provides free plan 10 proxies for every user after registration. If something goes wrong, let me know and I'll try to help with my proxy server. |
Beta Was this translation helpful? Give feedback.
I checked version 1.9.1 and the problem does not occur there. I went back to using selenium-driverless. Thank you.