Web Scraping Booking.Com

Booking.com, with over 28 million listings, is one of the largest websites for finding a place to stay while traveling. If you are opening a new hotel in any area, you might want to keep an eye on competitors and get notifications about new properties. All of this can be automated using web scraping! In this article, you will learn how to use Python and Selenium to extract data from the Booking.com search results page and handle pagination.

Setting Up the Prerequisites

This guide uses Python 3.10, but it should work with most Python versions. Start by creating a new directory to store all project files, then create a new Python file for the code itself:

$ mkdir booking_scraper
$ cd booking_scraper
$ touch app.py

You will need to install several different libraries:

selenium
selenium-wire
webdriver-manager

You can install both libraries with this command:

$ pip install selenium selenium-wire webdriver-manager

Selenium will provide all the APIs for programmatic access to the browser, and Selenium Wire adds additional capabilities on top. We will discuss these extra capabilities later when we use them. The webdriver-manager helps you easily install the browser driver binary without having to manually download and link it in the code.

Getting the Booking.com Search Results Page

Let's try to get the search results page using Selenium to make sure everything is set up correctly. Save the following code in app.py:

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager

driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()))
url = "https://www.booking.com/searchresults.en-gb.html?ss=San+Francisco%2C+California%2C+United+States"
driver.get(url)

Note: This URL does not contain check-in and check-out dates, so if you want information about properties available on specific dates, select those dates in the search form and copy the resulting URL.

Executing this code will open a Chrome window and navigate to the property search page for San Francisco. Opening the browser window may take a minute because the Chrome driver binary may need to be downloaded.

The code itself is quite simple. It starts by importing the webdriver, service, and ChromeDriverManager. Usually, to initialize the webdriver, you need to provide it with the executable_path to the driver binary for the specific browser:

browser = webdriver.Chrome(executable_path=r "C:\\path\\to\\chromedriver.exe")

The biggest drawback of this approach is that every time the browser is updated, you need to download the updated browser driver binary. This quickly becomes tedious, so the webdriver_manager library simplifies the task by allowing you to pass ChromeDriverManager().install(). It will automatically download the needed binary and return the corresponding path, so you no longer have to worry about it.

The last line of code tells the driver to navigate to the search results page.

How to Extract Property Information from the Search Results Page

Before extracting any data, you need to decide what data to extract. In this guide, you will learn how to extract the following data from the search results page:

Name
Location
Rating
Number of reviews
Thumbnail
Price

To access DOM elements and extract data, you will use the default methods (find_element + find_elements) provided by Selenium. Additionally, CSS selectors and XPath will be used to find DOM elements. Which method you use will depend on the DOM structure and which method is most appropriate.

Extracting the Property Name and Link

Let's start by examining the DOM structure for the property name. Right-click on any property name and select Inspect. The developer tools window will open:

Property title in devtools

As you can see, the hotel name is wrapped in an anchor tag. This anchor tag has a data-testid that we can use to extract the property link. The encapsulating div also defines a data-testid attribute (title) that we can use to extract the hotel name. To simplify things, let's first extract all property cards into a separate list, then work with each property one by one. This is not mandatory, but it helps keep things structured.

You can extract property cards by targeting divs with data-testid set to property-card:

property_cards = driver.find_elements(By.CSS_SELECTOR, 'div[data-testid="property-card"]')

Now you can extract the property name and link using the following code:

for property in property_cards:

name = property.find_element(By.CSS_SELECTOR,'div[data-testid="title"]').text

link = property.find_element(By.CSS_SELECTOR,'a[data-testid="title-link"]').get_attribute('href')

Extracting the Property Address and Price

Similarly, you can extract the property address. Examine the DOM structure in developer tools, and you will again see a unique data-testid:

To extract the address/location, you can use the following code:

for property in property_cards:

# ...

address = property.find_element(By.CSS_SELECTOR, '[data-testid="address"]').text

The same strategy applies to price. You can extract it by targeting the data-testid price-and-discounted-price. In each property listing, this data-testid is set on the final price. It does not include taxes, so if you want to extract those as well, you will need to further inspect the DOM.

To extract the price, use the following code:

for property in property_cards:

price = property.find_element(By.CSS_SELECTOR, '[data-testid="price-and-discounted-price"]').text

Extracting the Property Rating and Review Count

The rating and review count are nested in a div with data-testid review-score.

To extract the rating and review count, you can use one of several strategies. You can extract them separately by targeting each div, or you can be lazy and smart and extract all visible text from that div, then split it accordingly. Here is what the extraction looks like in raw form:

>>> print(property_cards[0].find_element(By.CSS_SELECTOR, '[data-testid="review-score"]').text)

7.8

Good

491 reviews

You can split this text on the newline character (\n), and that will give you the rating and review count. Here is how to do it:

for property in property_cards:

# ...

review_score, _, review_count = property.find_element(By.CSS_SELECTOR, '[data-testid="review-score"]').text.split('\n')

Extracting the Property Thumbnail

The thumbnail is inside an img tag with data-testid image.

You are probably getting used to this by now. Let's write code to target this image tag and extract the src attribute:

for property in property_cards:

# ...

image = property.find_element(By.CSS_SELECTOR, '[data-testid="image"]').

Navigating to the Next Page of Results

Navigating to the next page of search results is done with the following code:

next_page_btn = driver.find_element(By.XPATH, '//button[contains(@aria-label, "Next page")]')
next_page_btn.click()

Now comes the tricky part. By default, Selenium does not provide a way to check if a specific background request has returned. This becomes a serious limitation when dealing with JS-heavy websites. Without such capability, there is no direct way to determine if the DOM has been updated with new results after clicking the next page button. You can compare the list of new properties with those already scanned, which gives you some indication of whether the DOM has been updated with new data. However, there is a simpler method.

This is where Selenium Wire shines. If you followed the installation instructions at the beginning of the article, you already have it installed. Selenium Wire allows you to inspect individual requests and find out whether the response returned successfully. Fortunately, it is a drop-in replacement for Selenium and does not require any code changes. Just update the imp