Web Scraping with Python: A Beginner's Guide
In the world of big data, web scraping services or data extraction are essential requirements for big data analytics. Obtaining data from the internet has become almost unavoidable for companies to stay in business. The next question that arises is how a beginner can get started with web scraping.
Data can be extracted or collected from a web source using several methods. Popular sites like Google, Facebook, or Twitter offer APIs to view and extract available data in a structured format. This avoids using other methods that may be undesirable to the API provider. However, the need to scrape a website arises when the information is not easily provided by the website.
Python, an open-source programming language, is often used for web scraping due to its simplicity and rich ecosystem. It includes a library called "BeautifulSoup" that performs this task. Let's take a closer look at web scraping with Python.
Setting Up the Python Environment:
To perform web scraping with Python, you first need to install a Python environment that allows you to execute code written in Python.
Libraries for Data Scraping:
Beautiful Soup is a user-friendly Python library. It is one of the best tools for extracting information from a web page. Professionals can extract information from web pages in the form of tables, lists, or paragraphs.
Urllib2 is another library that can be used in conjunction with BeautifulSoup to extract web pages. Filters can be added to extract specific information from web pages. Urllib2 is a Python module that can retrieve URLs.
For macOS:
To install Python libraries on macOS, users need to open the terminal and enter the following commands one at a time:
sudo easy_install pip
pip install BeautifulSoup4
pip install lxml
For Windows Users:
Windows users first need to ensure that the Python environment is installed. After installing the environment, open the command prompt and navigate to the root directory C:\, then enter the following commands:
easy_install BeautifulSoup4
easy_install lxml
Once the libraries are installed, it's time to write the code for data scraping.
Running Python:
Data scraping from websites should be done for a specific purpose, such as viewing current inventory of a retail store. First, use a web browser to navigate to the site containing that data.
Once you locate the table, right-click anywhere on it, then select "Inspect" from the dropdown menu. This will open a window at the bottom or side of the screen showing the site's HTML code.
Ratings are displayed as a table. You may need to browse the HTML data until you find the line of code that highlights the table on the web page.
Besides BeautifulSoup, Python offers several other alternatives for parsing HTML. These libraries include:
- Scrapy
- Scrapemark
- Mechanize
A parser converts unstructured data from HTML code into a structured data form, such as tabular data in an Excel worksheet. Web scraping can be done in various ways, from using Google Docs to programming languages.
People without programming knowledge or technical skills can obtain web data by using web scraping services that provide ready-to-use data from websites of your choice.
HTML Tags:
To implement web scraping, you need at least a basic understanding of HTML tags.
HTML tables are denoted as <Table>, rows as <tr>, and columns and dividers as <td>;
- <!DOCTYPE html> : An HTML document starts with a document type declaration.
- The main body of the HTML document as unformatted plain text is defined by the <body> and </body> tags.
- Headings in HTML are defined using heading tags from <h1> to <h5>.
- Paragraphs in HTML are defined using the <p> tag.
- The entire HTML document is contained between <html> and </html>.
Using BeautifulSoup for Scraping:
In the process of scraping a web page with BeautifulSoup, the key is to define the ultimate goal. For example, if you want to extract a list from a web page, a step-by-step approach is needed:
The first and most important step is to import the necessary libraries:
import urllib2
page = urllib2.urlopen(wiki)
from bs4 import BeautifulSoup
soup = BeautifulSoup(page)
Use the "prettify" function to visualize the nested structure of the HTML page
Extracting information into a DataFrame: You need to go through each row (tr), then assign each tr element (td) to a variable and add it to a list.
Let's analyze the HTML structure of the Table. (Extracting information for the table header <th>)
To access the value of each element, you need to use the "find(text=True)" option with each element. Finally, you have the data in a dataframe.
There are various other ways to collect data using "BeautifulSoup" that reduce manual effort for data collection from web pages. Code written in BeautifulSoup is considered more robust than regular expressions. The scraping method we reviewed uses the "BeautifulSoup" and "urllib2" libraries in Python. This was a brief beginner's guide to get started using Python for web scraping.
Planning to get data from the internet for data science? We are ready to help. Let us know about your tasks.