tool

Selenium

Selenium is a browser automation framework originally designed for testing that is widely used for web scraping when JavaScript rendering and browser interaction are required.

Selenium WebDriver controls a real browser (Chrome, Firefox, Safari, Edge) through the W3C WebDriver protocol. It can click elements, fill forms, scroll, navigate, and read the rendered DOM, making it capable of scraping any site that a human can browse. Selenium was the dominant browser automation tool for scraping before Playwright and Puppeteer emerged.

Compared to Playwright, Selenium is slower (due to the WebDriver protocol's HTTP round-trip overhead), has less ergonomic async support, and lacks built-in network interception capabilities. However, its broad browser support and large existing user base mean it remains widely deployed.

Selenium Grid allows distributing browser sessions across multiple machines, enabling horizontal scaling of browser-based scraping. For anti-bot evasion, `undetected-chromedriver` is a popular Selenium wrapper that patches Chrome's WebDriver signatures to reduce detection likelihood.

Examples

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome()
driver.get("https://example.com")
el = WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.CSS_SELECTOR, ".product-price"))
)
print(el.text)
driver.quit()