Web Scraping Using Selenium and Python

Selenium is a set of different open-source projects used for browser automation. It supports bindings for all major programming languages, including our favorite language: Python.

The Selenium API uses the WebDriver protocol to control web browsers such as Chrome, Firefox, or Safari. Selenium can control both a locally installed browser instance and one running on a remote machine over a network.

Originally (and that's already about 20 years!), Selenium was intended for cross-browser end-to-end testing (acceptance tests). However, over time, it has mainly become used as a platform for general-purpose browser automation (e.g., for taking screenshots), which naturally includes web scraping and the like. Rarely anything can be better at "communicating" with a website than a real, proper browser, right?

Selenium provides a wide range of ways to interact with websites, such as:

clicking buttons
filling forms with data
scrolling the page
taking screenshots
executing custom, user-defined JavaScript code.

But the strongest argument in its favor is the ability to work with websites naturally, just like any browser does. This is especially evident when working with single-page sites that contain JavaScript. If you try to scrape such a site using the traditional combination of an HTTP client and an HTML parser, you will mostly get a lot of JavaScript files, but not much data to scrape.

Installation

Although Selenium supports several browser engines, for this example we will use Chrome, so make sure you have the following packages installed:

Chrome download page
ChromeDriver binary corresponding to your version of Chrome
Selenium Python Binding package.

To install the Selenium package, as usual, I recommend creating a virtual environment (e.g., with virtualenv), and then:

pip install selenium

Quick Start

After downloading Chrome and ChromeDriver and installing the Selenium package, you should be ready to launch the browser:

from selenium import webdriver

DRIVER_PATH = '/path/to/chromedriver'
driver = webdriver.Chrome(executable_path=DRIVER_PATH)
driver.get('https://google.com')

Since we did not configure headless mode, a normal Chrome window will appear on the screen with an additional message stating that Chrome is being controlled by Selenium.

Chrome Headless Mode

Launching the browser from Selenium, as we just did, is especially useful during development. It allows you to precisely observe what is happening, how the page and browser behave in the context of your code. When everything is satisfactory, it is usually recommended to switch to headless mode in production.

In this mode, Selenium will run Chrome in the background without any visual output or windows. Imagine a production server running multiple Chrome instances simultaneously with all windows open. Well, servers generally disregard how "attentive" people are to their user interface - poor things - but seriously, there is no point in wasting GUI resources for no reason.

Fortunately, enabling headless mode requires only a few flags.

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

options = Options()
options.headless = True
options.add_argument("--window-size=1920,1200")

driver = webdriver.Chrome(options=options, executable_path=DRIVER_PATH)

We just need to instantiate an Options object, set its headless field to True, and pass it to the WebDriver constructor. Done.

WebDriver Page Properties

Building on our headless mode example, let's turn into real Mario and take a look at the Nintendo website.

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

options = Options()
options.headless = True
options.add_argument("--window-size=1920,1200")

driver = webdriver.Chrome(options=options, executable_path=DRIVER_PATH)
driver.get("https://www.nintendo.com/")
print(driver.page_source)
driver.quit()

When you run this script, you'll get some browser-related debug messages and, finally, the HTML code of nintendo.com. That's because our print call accesses the driver's page_source field, which contains the HTML document of the last site we requested.

Two more interesting WebDriver fields:

driver.title - to get the page title
driver.current_url - to get the current URL (this can be useful when there are redirects on the site and you need the final URL)

Locating Elements

To scrape/extract data, you need to know where it is. Therefore, locating page elements is a key feature of web scraping. Naturally, Selenium provides this out of the box (for example, in test scenarios you need to verify the presence/absence of a specific element on the page).

There are quite a few standard ways to find a specific element on a page. For example, you can

search by tag name
filter by a specific HTML class or HTML ID
or use CSS selectors or XPath expressions

As usual, the easiest way to find an element is to open Chrome Developer Tools and inspect the desired element. A handy shortcut is to highlight the element with your mouse, then press Ctrl + Shift + C (or Cmd + Shift + C on macOS) instead of right-clicking and selecting Inspect every time.

Once you find the element in the DOM tree, you can decide the best way to programmatically access it. For example, you can right-click on the element in the inspector and copy its absolute XPath or CSS selector.

find_element Methods

WebDriver provides two main methods for finding elements.

find_element
find_elements

They are quite similar, with the only difference being that the former finds a single element and returns it, while the latter returns a list of all found elements.

Both methods support eight different search types, denoted by the By class.

find_element Examples
Suppose we have the following HTML document.....

<html>
<head>
... some stuff
<\/head>
<body>
<h1 class="someclass" id="greatID">Super title<\/h1>
<\/body>
<\/html>

.... and we want to select the <h1> element. In this case, the following five examples will be identical in the information they return

h1 = driver.find_element(By.NAME, 'h1')
h1 = driver.find_element(By.CLASS_NAME, 'someclass')
h1 = driver.find_element(By.XPATH, '\\\/\\\/h1')
h1 = driver.find_element(By.XPATH, '\\\/html\\\/body\\\/h1')
h1 = driver.find_element(By.ID, 'greatID')

Another example is selecting all anchor/link tags on the page. Since we need multiple elements, we use find_elements here (note the plural)

all_links = driver.find_elements(By.TAG_NAME, 'a')

Some elements are not easy to get using an ID or a simple class, and then you'll need an XPath expression. Also, you may have multiple elements with the same class, and sometimes even the same ID, although the latter should be unique.

XPath is my favorite way to locate elements on a web page. It is a powerful way to extract any element on the page based on its absolute position in the DOM or relative to another element.

Selenium WebElement

WebElement is a Selenium object representing an HTML element.

There are many actions that can be performed with these objects; here are the most useful ones:

Access the element's text using element.text
Click on the element using element.click()
Access an attribute using element.get_attribute('class')
Send text to an input using element.send_keys('mypassword').

There are other interesting methods, such as is_displayed(). It returns True if the element is visible to the user, and can be useful for preventing 'honeypot bots' (e.g., intentionally hidden input elements). Honeypots are mechanisms used by website owners to detect bots. For example, if an HTML input element has a type=hidden attribute, as shown below:

<input type="hidden" id="custId" name="custId" value="">.

This value should be empty. If a bot visits the page and thinks it needs to fill in values for all input elements, it will fill in the hidden input as well. A legitimate user would never provide a value in that hidden field, since it is not displayed by the browser in the first place.

Full Example

Let's provide an example using the Selenium API methods we just covered.

We are going to log in to Hacker News:

Of course, authenticating on Hacker News is not very useful by itself. However, you can imagine creating a bot that automatically posts a link to your latest blog entry.

To authenticate, we need:

Navigate to the login page using driver.get()
Select the username input field using driver.find_element and call element.send_keys() to send text to that field.
Do the same for the password input field
Select the login button (of course using find_element) and click it using element.click().

Everything should be simple, right? Let's look at the code:

driver.get("https:\/\/news.ycombinator.com\/login")

login = driver.find_element_by_xpath("\/\/input").send_keys(USERNAME)
password = driver.find_element_by_xpath("\/\/input[@type='password']").send_keys(PASSWORD)
submit = driver.find_element_by_xpath("\/\/input[@value='login']").click()

Easy, right? But there's one important point missing. How do we know if we are logged in?

We can try a few things:

Check for an error message (e.g., "Invalid password").
Check for an element on the page that is only displayed after logging in.

So we will check for the logout button. The logout button has an id of logout (easy)!

We cannot simply check if the element is None, because find_element raises an exception if the element is not found in the DOM. So we must use a try\/except block and catch the NoSuchElementException:

# dont forget from selenium.common.exceptions import NoSuchElementException
try:
logout_button = driver.find_element_by_id("logout")
print('Successful login')
except NoSuchElementException:
print('Invalid username/password')

Taking Screenshots

The beauty of browser-based approaches, such as Selenium, lies not only in retrieving data and the DOM tree, but also in the fact that, being a browser, it renders the entire page correctly and completely. This naturally allows taking screenshots, and Selenium is fully equipped for that.

driver.save_screenshot('screenshot.png')

A single call - and we have a screenshot of our page. Well, isn't that cool!

Note that when taking a screenshot with Selenium, some things can go wrong or require adjustment. First, you need to make sure the window size is set correctly. Then you need to ensure that all asynchronous HTTP calls made by the front-end JavaScript code are completed and the page is fully rendered.

In our case with Hacker News, it's simple and we don't need to worry about these issues.

Waiting for Element Presence

Working with a site that uses a lot of JavaScript to display content can be challenging. Nowadays, more and more sites use frameworks like Angular, React, and Vue.js for their front-ends. Dealing with these frameworks is difficult because they don't simply provide HTML code; they contain a fairly complex set of JavaScript code that modifies the DOM tree on the fly and sends a lot of information asynchronously in the background via AJAX.

This means we can't just send a request and get data immediately; we have to wait for the JavaScript to finish its work. This can usually be done in two ways:

Use time.sleep() before taking a screenshot.
Use the WebDriverWait object.

If you use time.sleep(), you'll have to pick the most reasonable delay for your case. The problem is that you either wait too long or not long enough, and neither is ideal. Additionally, the site may load slower on your home ISP connection than when your code runs in a data center production environment. With WebDriverWait, you don't need to account for that. It will wait only as long as necessary until the required element appears (or until a timeout occurs).

try:
element = WebDriverWait(driver, 5).until(
EC.presence_of_element_located((By.ID, "mySuperId"))
)
finally:
driver.quit()

This will wait for the element with the HTML ID mySuperId to appear or for the five-second timeout to expire.
The full list of Waits and their expected conditions can, of course, be found in the Selenium documentation. Read more about web scraping in the guide to web scraping for financial data.

But having a full browser engine at our disposal not only allows us to work more or less easily with the JavaScript code executed by the site, but also gives us the ability to run our own custom JavaScript. Let's check that next.

Executing JavaScript

As with screenshots, we can also fully utilize our browser's JavaScript engine. This means we can inject and execute arbitrary code within the site's context. Want to take a screenshot of a fragment located a bit further down the page? Easy, window.scrollBy() and execute_script() have you covered.

javaScript = "window.scrollBy(0, 1000);"
driver.execute_script(javaScript)

Or maybe you want to highlight all anchor tags with a border? That's even simpler.

javaScript = "document.querySelectorAll('a').forEach(e => e.style.border='red 2px solid')"
driver.execute_script(javaScript)

An additional advantage of the execute_script() function is,