Web Scraping with Python

Web scraping has become one of the most powerful methods of collecting data from the Internet in the data science community. Often we rely on other people's datasets, but it is important to arm yourself with the necessary skills to create your own datasets. That is why Fortune 500 companies like Amazon, CNN, Target, and Walmart use web scraping to move forward and keep up with data. It is an indispensable growth tool and one of their best-kept secrets, and it can easily become yours.

In this article, you will learn how to search for data using the Python language. By the end of the article, you will understand the most important components of web scraping, giving you the skills to build your own web scraper.

Whether you are a data scientist or machine learning engineer looking to create new datasets, or a web developer interested in automating tasks, this article provides a detailed overview of the fundamentals and approaches to web scraping that you can easily apply in real-world business or personal projects.

What is Web Scraping?

Web scraping, also called data extraction from the Internet, is the automated process of collecting publicly available web data from target websites. Instead of manually collecting data, you can use software to automatically gather large amounts of information, significantly speeding up the process.

Why is Web Scraping Important?

Some websites may contain a very large volume of invaluable data, such as stock prices, product information, sports statistics, etc. If you want to access this information, you would either have to use the format used on the site or manually copy and paste the information into a new document. This can be quite tedious if you need to extract a lot of information from a site, and that's where web scraping can help. Instead of manually collecting data, it is often preferable to use software tools called web scrapers, as they are less costly compared to human labor and work faster. Web scrapers can run on your computer or in a data center.

How Does Web Scraping Work?

The web scraping process involves three main steps:

• Step 1: Retrieve content from the website. You obtain the content of the target website using web scraping software, also called a web scraper, which makes HTTP requests to specific URLs. Depending on your goals, experience, and budget, you can either purchase a web scraping service or build your own web scraper. The web scraper is given one or more URLs to search. The scraper then downloads the entire HTML code of the requested pages. More advanced scrapers render the entire web page, including CSS and JS elements.
• Step 2: Extract the required data. The necessary information from the HTML code is parsed by the web scraper according to your requirements.
• Step 3: Save the parsed data. The final step is to save the parsed data. Typically, the data is saved in CSV or JSON formats for further use.

Use Cases of Web Scraping

Businesses use web scraping for various purposes such as market research, brand protection, price monitoring, SEO monitoring, rate aggregation, review monitoring, etc. Let's look at some of these in more detail:

• Ticket Bots - Web scrapers are increasingly used to perform ticket-related tasks such as searching for price information, checking inventory for new seats, and purchasing tickets. For example, ticket brokers apparently used ticket bots on Ticketmaster to buy most of the tickets for Taylor Swift's "Midnight" album tour.
• Competitive Price Monitoring - Since companies need to keep track of constantly changing market prices, collecting product price information from sites like Amazon or eBay is critical for developing an accurate pricing strategy. For example, with web scraping, you can export a list of product names and prices from Amazon into an Excel spreadsheet and conduct competitive analysis.
• Market Research - Web scraping is widely used for market research. To stay competitive, companies need to understand their market and analyze competitor data.
• SEO Monitoring - Web scraping allows companies to conduct SEO audits to track results and improve search engine rankings.
• Review Monitoring - Web scraping can also be used to track customer reviews and achieve marketing goals.
• Lead Generation - Scraping data from LinkedIn, Yellow Pages, or YELP for lead generation.
• Travel Rate Aggregation - Travel companies use web scrapers to search for deals on multiple sites and publish aggregated results on their own sites.
• Stock Price Scraping - To make better investment decisions.
• Sports Statistics Scraping - For betting or fantasy leagues.

Tools for Web Scraping

Now that you know the basics of web scraping, you're probably wondering which web scraper is best for you? The obvious answer is that it depends! Understanding which web scraper is best for you becomes easier the more you know about your web scraping needs. Nowadays, websites can come in various shapes and formats, and consequently, web scrapers can differ in functionality and capabilities. For example, web scrapers can be in the form of a browser extension or a more powerful desktop application that you can download to your computer for local site scraping using your computer's resources and internet connection, or deployed in the "cloud".

Thus, as you can see, there is a range of automated web scrapers on the market. Among the most common scraping tools are Octoparse and ParseHub. These applications allow you to automate data extraction from various online sources, provided you know what type of content you need.

Is Web Scraping Legal?

With the growing popularity of web scraping, more questions arise about its legality. Although web scraping itself is not illegal and there are no clear laws or regulations governing its use, it is important to comply with all other laws and regulations regarding the sources and the data itself. In general, according to a US appellate court ruling, scraping publicly available data—i.e., data that can be seen without logging into a site—is legal as long as your scraping activity does not harm the operation of the website being scraped.

Here are a few examples of what might make web scraping illegal that you should consider:

• Scraping Intellectual Property - You must ensure you are not violating laws applicable to copyrighted data, such as designs, articles, videos, and anything that can be considered a creative work. Scraping copyrighted data for personal purposes is usually not dangerous, as it may fall under the fair use provision in intellectual property law. However, distributing data for which you do not have rights is illegal.
• Collecting Personal Data - It is important not to collect personally identifiable information (PII), as most people do not like their personal information being collected without their knowledge or consent. Therefore, after scraping, double-check the results for any data that may violate GDPR or CCPA.
• Scraping Confidential Data - Scraping private content without permission can also lead to trouble.

How to Create a Web Scraper in Python?

Feeling adventurous? Just as anyone can create a website, you can build your own web scraper if you're willing to put in a bit of coding work. In the following section, we'll walk through creating a simple scraping bot using Python. Python has many libraries, including requests, beautifulsoup, selenium, scrapy, and pandas, that make it easy to develop scraping programs.

You will write a web scraper in Python that downloads the IMDB Top 250 movie dataset (movie title, original release year, director name, and stars). IMDb, or Internet Movie Database, is an online database containing information about movies, TV shows, home video, video games, and streaming content. It includes data such as cast, crew, biographies, plot summaries, trivia, ratings, and critical reviews.

But first, you need to install several Python libraries:

• requests: The first step in any web scraping workflow is sending an HTTP request to a web server to display data on the target page. Requests is a Python library designed to simplify the process of sending HTTP requests to a given URL. When you make a request to a URI, it returns a response.
• html5lib: A Python library for parsing HTML. It is designed to comply with the WHATWG HTML specification, implemented in all major web browsers.
• bs4: BeautifulSoup is a web scraping framework in Python that allows convenient HTML parsing. Since BeautifulSoup can only parse data and cannot fetch web pages itself, it is often used together with the requests library. Thus, when the requests library sends an HTTP request to a web page, after successful sending and return, BeautifulSoup can be used to parse the data.
• pandas: A library built on top of the NumPy library, providing various data structures and operators for manipulating numerical data.
• lxml: A Python library for processing/parsing XML and HTML.

Steps to Create

Steps to implement web scraping in Python to extract IMDb movies and their ratings:

Step 1. Install the required Python libraries

$ pip install requests beautifulsoup4 html5lib pandas lxml

Step 2: Import the necessary libraries
Open a text editor and paste the following code:

from bs4 import BeautifulSoup
import requests
import re
import pandas as pd

Step 3: Access the HTML content from the web page
You can access the HTML content from the web page by assigning it a URL and creating a soup object as follows:

# Downloading imdb top 250 movie's data from the web page

url = 'http://www.imdb.com/chart/top'
response = requests.get(url)

soup = BeautifulSoup(response.text, "html.parser")

Step 4: Extract movie information
In the code snippet below, we extract data from the BeautifulSoup object using HTML tags such as title, href, etc.

movies = soup.select('td.titleColumn')

crew = [a.attrs.get('title') for a in soup.select('td.titleColumn a')]

ratings = [b.attrs.get('data-value')

        for b in soup.select('td.posterColumn span[name=ir]')]

Step 5: Save the movie information
After extracting the movie information, create an empty list, save the information in a dictionary, and add it to the list.

# create an empty list for storing
# movie details

list = []
 
# Iterating over movies to extract
# each movie's metadata

for index in range(0, len(movies)):

     

    # Separating movie into: 'place',

    # 'title', 'year'

    movie_string = movies[index].get_text()

    movie = (' '.join(movie_string.split()).replace('.', ''))

    movie_title = movie[len(str(index))+1:-7]

    year = re.search('\((.*?)\)', movie_string).group(1)

    place = movie[:len(str(index))-(len(movie))]

    data = {"place": place,

            "movie_title": movie_title,

            "rating": ratings[index],

            "year": year,

            "star_cast": crew[index],

            }

    list.append(data)

Step 6: Display the movie details
Now that our list is filled with the best IMDB movies and their metadata, it's time to display the details.

for movie in list:

    print(movie['place'], '-', movie['movie_title'], '('+movie['year'] +

          ') -', 'Starring:', movie['star_cast'], movie['rating'])

Step 7: Save the data to a CSV file
The following lines of code will save the data to a .csv file.

# saving the list as a dataframe
# converting to .csv file

df = pd.DataFrame(list)

df.to_csv('imdb_top_250_movies.csv',index=False)

Complete code:

from bs4 import BeautifulSoup

import requests

import re

import pandas as pd
 
 
# Downloading imdb top 250 movie's data

url = 'http://www.imdb.com/chart/top'

response = requests.get(url)

soup = BeautifulSoup(response.text, "html.parser")

movies = soup.select('td.titleColumn')

crew = [a.attrs.get('title') for a in soup.select('td.titleColumn a')]

ratings = [b.attrs.get('data-value')

        for b in soup.select('td.posterColumn span[name=ir]')]
 
 
 
 
# create an empty list for storing
# movie information

list = []
 
# Iterating over movies to extract
# each movie's details

for index in range(0, len(movies)):

     

    # Separating movie into: 'place',

    # 'title', 'year'

    movie_string = movies[index].get_text()

    movie = (' '.join(movie_string.split()).replace('.', ''))

    movie_title = movie[len(str(index))+1:-7]

    year = re.search('\((.*?)\)', movie_string).group(1)

    place = movie[:len(str(index))-(len(movie))]

    data = {"place": place,

            "movie_title": movie_title,

            "rating": ratings[index],

            "year": year,

            "star_cast": crew[index],

            }

    list.append(data)
 
# printing movie details with its rating.

for movie in list:

    print(movie['place'], '-', movie['movie_title'], '('+movie['year'] +

        ') -', 'Starring:', movie['star_cast'], movie['rating'])
 
 
##.......##

df = pd.DataFrame(list)

df.to_csv('imdb_top_250_movies.csv',index=False)

Run this code in a terminal or IDE. A csv file with the data shown in the following image is saved:

Conclusion

In summary, web scraping is an automated data collection process. Businesses can use it for various purposes, such as lead generation, competitive data research, stock market analysis, etc. Web scraping is legal as long as it does not violate laws concerning the source objects or the data itself. However, before engaging in any web scraping activity, it is necessary to obtain professional legal advice regarding the specific situation. It is also important to consider all possible risks associated with careless web scraping, such as blocking. That's essentially all there is to web scraping. However, there is still much more to explore in this field, and I encourage you to look into some of the most common scraping techniques and hone your Python programming skills!