How to Run Selenium Headless in Python

Selenium is a browser automation framework that supports multiple languages, including Python, and is commonly used to run browsers without a UI for testing or scraping. Its headless mode is faster and lighter than full browsers, offering better performance, optimization, and programmatic control.

Installation

To use Selenium headless, install Python and an IDE like free PyCharm. Create a project and run:

pip install selenium 

Ensure you have a browser WebDriver (modern browsers usually include one).

Running Selenium in Headful Mode

First, test Selenium in headful mode to confirm setup. Import webdriver, define a function to open a URL, and quit after loading. The function creates a Chrome WebDriver, opens the given URL, then closes. If you see a missing driver error, download the correct ChromeDriver for your OS:

from selenium import webdriver

def open_browser(URL: str):
    browser = webdriver.Chrome()
    browser.get(URL)
    browser.quit()

open_browser('https://iproyal.com/')

Selenium Headless Mode

In the previous example, the browser opened with a full GUI, but for testing and scraping it’s more common to run Selenium in headless mode. To do this, we import Chrome options, update the browser object to run headless, and execute the code again. This time no window appears, but the page’s title is logged in the console, confirming everything works correctly.

from selenium import webdriver
from selenium.webdriver import ChromeOptions

def open_browser(URL: str):
    options = ChromeOptions()
    options.add_argument("--headless=new")
    browser = webdriver.Chrome(options=options)
    browser.get(URL)
    print(browser.title)  # log page title
    browser.quit()

open_browser('https://iproyal.com/')

Running Through a List of URLs

Usually, you’ll want Selenium to visit multiple URLs instead of just one. We can update the function to loop through a list.

from selenium import webdriver
from selenium.webdriver import ChromeOptions

def open_browser(URL):
    options = ChromeOptions()
    options.add_argument("--headless=new")
    browser = webdriver.Chrome(options=options)
    for i in URL:
        browser.get(i)
        print("Page title:", browser.title)  # log page title
    browser.quit()

list_of_urls = ['https://iproyal.com/', 'https://iproyal.com/pricing/residential-proxies/']
open_browser(list_of_urls)

The function now accepts lists (or other sequences), visits each URL in order, and prints the page titles to the console. For testing, you can comment out the headless option to watch the browser run with a UI.