Use Established requests.Session in save_website

[james0048]

Is it possible to use an already created sessions object when calling save_website? In order to get past a login screen I have to get a value from the login page and pass it as a value when logging in. I can do that with requests but I don't see a good way to do it inside of pywebcopy.

Here's the code I have but the save_website is just getting the login screen:

with requests.Session() as session:
    response = session.get(request_URL)
    soup = BeautifulSoup(response.text, 'lxml')
    loc = (soup.findAll('input'))[0]['value']
    payload['sph_org_location'] = loc
    post = session.post(post_login, data=payload)
    pywebcopy.SESSION = session
    pywebcopy.save_website(url=request_URL, project_folder="C:\\temp", project_name="project", bypass_robots=True, debug=True, open_in_browser=False, delay=1, threaded=False)

[rajatomar788]

Hey @james0048

from pywebcopy.config import get_config
config = get_config(url, project_folder, project_name, bypass_robots, debug, delay, threaded)
webpage = config.create_page()

# here you have the ready webpage instance to do anything
response = webpage.session.get(request_URL)

soup = BeautifulSoup(response.text, 'lxml')

loc = (soup.findAll('input'))[0]['value']

payload['sph_org_location'] = loc

post = webpage.session.post(post_login, data=payload)

# Now you can get the desired page without seeing the login screen
# webpage.get(url)
    
# webpage.save_complete(pop=open_in_browser)

You can use the webpage.session object which is a requests library session that you are using. So it should not be a problem.

[james0048]

Awesome, thanks for the help. That did the trick for logging in to the site. Is there a way to use the save_website function with that established session?

[rajatomar788]

Yes use the crawler scheduler instead.
Go to the init.py file and copy the save website function code.

[james0048]

Thanks for the help and the great library!

Here's the code I used in case it helps someone else.

config = get_config(request_URL, project_folder="C:\\temp", project_name="proj", bypass_robots=True, debug=True, delay=1, threaded=False)
crawler = config.create_crawler()
response = crawler.session.get(request_URL)
soup = BeautifulSoup(response.text, 'lxml')
loc = (soup.find_all('input'))[0]['value']
payload['sph_org_location'] = loc
post = crawler.session.post(post_login, data=payload)
crawler.get(request_URL)
crawler.save_complete(True)