Implementing Web Scraping in Python with Beautiful Soup

There are two main ways to extract data from a website:

• Use the site's API (if it exists). For example, Facebook has the Facebook Graph API, which allows you to extract data posted on Facebook.
• Access the site's HTML page and extract useful information/data from it. This technique is called web scraping, web harvesting, or web data extraction.

This article covers the steps involved in web scraping, using an example implementation of a web scraping framework in Python called Beautiful Soup. The steps performed in web scraping:

1. Send an HTTP request to the URL of the web page you want to access. The server responds to the request by returning the HTML content of the web page. To accomplish this task, we will use the third-party HTTP library python-requests.
2. After accessing the HTML content, we need to parse the data. Since most HTML data is nested, we cannot extract data simply by string processing. We need a parser capable of creating a nested/tree structure of HTML data. There are many HTML parser libraries, but the most advanced is html5lib.
3. Now we need to navigate and search the parse tree we have created, i.e., traverse the tree. For this task, we will use another third-party Python library - Beautiful Soup. It is a Python library for extracting data from HTML and XML files.

Step 1: Installing Required Third-Party Libraries

The easiest way to install external libraries in Python is using pip. pip is a package management system used to install and manage software packages written in Python. All you need to do is:

pip install requests
pip install html5lib
pip install bs4

Another way is to manually download them from these links:

requests
html5lib
beautifulsoup4

Step 2: Accessing the HTML Content of a Web Page

import requests

URL = "https://www.geeksforgeeks.org/data-structures/"
r = requests.get(URL)
print(r.content)

Let's try to understand this code snippet.

• First, import the requests library.
• Then specify the URL of the web page to scrape.
• Send an HTTP request to the specified URL and save the server response in a response object named r.
• Now, using print(r.content), you get the raw HTML content of the web page. It is of type 'string'.

headers = {'User-Agent': "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36 Edge/12.246"}
# User agent for Edge browser on Windows 10. You can find your browser user agent from the link above.
r = requests.get(url=URL, headers=headers)
print(r.content)

Step 3: Parsing the HTML Content

# This will not run in an online IDE
import requests
from bs4 import BeautifulSoup
  
URL = "http://www.values.com/inspirational-quotes"
r = requests.get(URL)
  
soup = BeautifulSoup(r.content, 'html5lib') # If this line causes an error, run 'pip install html5lib' or install html5lib
print(soup.prettify())

A very nice aspect of the Beautiful Soup library is that it is built on top of HTML parsing libraries such as html5lib, lxml, html.parser, etc. Thus, the BeautifulSoup object and the parser library specification can be created simultaneously. In the example above,

soup = BeautifulSoup(r.content, 'html5lib')

We create a BeautifulSoup object by passing two arguments:

• r.content: This is the raw HTML content.
• html5lib: specifies the HTML parser we want to use.

Now, soup.prettify() is printed, which gives a visual representation of the parse tree created from the raw HTML content.

Step 4: Searching and Navigating the Parse Tree

Now we want to extract some useful data from the HTML content. The soup object contains all the data in a nested structure, which can be extracted programmatically. In our example, we are scraping a web page consisting of some quotes. So we want to create a program to save these quotes (and all necessary information about them).

# Python program to scrape a website
# and save quotes from the site
import requests
from bs4 import BeautifulSoup
import csv
   
URL = "http://www.values.com/inspirational-quotes"
r = requests.get(URL)
   
soup = BeautifulSoup(r.content, 'html5lib')
   
quotes=[]  # a list to store quotes
   
table = soup.find('div', attrs = {'id':'all_quotes'}) 
   
for row in table.findAll('div',
                         attrs = {'class':'col-6 col-lg-3 text-center margin-30px-bottom sm-margin-30px-top'}):
    quote = {}
    quote['theme'] = row.h5.text
    quote['url'] = row.a['href']
    quote['img'] = row.img['src']
    quote['lines'] = row.img['alt'].split(" #")[0]
    quote['author'] = row.img['alt'].split(" #")[1]
    quotes.append(quote)
   
filename = 'inspirational_quotes.csv'
with open(filename, 'w', newline='') as f:
    w = csv.DictWriter(f,['theme','url','img','lines','author'])
    w.writeheader()
    for quote in quotes:
        w.writerow(quote)

It is observed that all quotes are inside a div container with id 'all_quotes'. So we find this div element (referred to as table in the code above) using the find() method:

table = soup.find('div', attrs = {'id':'all_quotes'}) 

The first argument is the HTML tag to search for, and the second is a dictionary type element to specify additional attributes associated with that tag. The find() method returns the first matching element. You can try printing table.prettify() to get an idea of what this section of code does.
Now in the table element, you can see that each quote is inside a div container with class quote. So we iterate over all div containers whose class is quote. Here we use the findAll() method, which is similar in arguments to find but returns a list of all matching elements. Now each quote is iterated using the variable row. For a better understanding, an example of the HTML content of a row is given: Implementing Web Scraping in Python with Beautiful Soup Now let's look at this code snippet:

for row in table.find_all_next('div', attrs = {'class': 'col-6 col-lg-3 text-center margin-30px-bottom sm-margin-30px-top'}):
    quote = {}
    quote['theme'] = row.h5.text
    quote['url'] = row.a['href']
    quote['img'] = row.img['src']
    quote['lines'] = row.img['alt'].split(" #")[0]
    quote['author'] = row.img['alt'].split(" #")[1]
    quotes.append(quote)

We create a dictionary to store all information about the quote. Nested structure can be accessed using dot notation. To access the text inside an HTML element, we use .text:

quote['theme'] = row.h5.text

We can add, delete, modify, and access tag attributes. To do this, the tag is treated as a dictionary:

quote['url'] = row.a['href']

Finally, all quotes are added to a list called quotes.
Finally, we want to save all our data into a CSV file.

filename = 'inspirational_quotes.csv'
with open(filename, 'w', newline='') as f:
    w = csv.DictWriter(f,['theme','url','img','lines','author'])
    w.writeheader()
    for quote in quotes:
        w.writerow(quote)

Here we create a CSV file inspirational_quotes.csv and save all quotes in it for later use.
So, this was a simple example of creating a web scraper in Python. Next, you can try to scrape any other website of your choice.