Implementing Web Scraping in Python with Scrapy

With Scrapy you can:

1. Efficiently scrape millions of data items
2. Run it on a server
3. Extract data
4. Run spider in multiple processes

Scrapy offers entirely new possibilities for creating a spider, running it, and subsequently saving data by scraping. At first it seems quite confusing, but that's for the better.

Let's talk about installation, creating a spider, and testing it.

Step 1: Creating a virtual environment

It is good to create a virtual environment because it isolates the program and does not affect other programs on the machine. To create a virtual environment, first install it using:

sudo apt-get install python3-venv

Create a folder and activate it:

mkdir scrapy-project && cd scrapy-project
python3 -m venv myvenv 

If the above command gives an error, try the following:

python3.5 -m venv myvenv

After creating the virtual environment, activate it using:

source myvenv/bin/activate

Step 2: Installing the Scrapy module

Install the Scrapy module using the command:

pip install scrapy

To install scrapy for a specific version of Python:

python3.5 -m pip install scrapy

Replace version 3.5 with another, e.g., 3.6.

Step 3: Creating a Scrapy project

When working with Scrapy, you need to create a scrapy project.

scrapy startproject gfg

In Scrapy, you always create a spider to help fetch data. To do so, navigate to the spider folder and create a Python file. Create a spider named gfgfetch.py.

Step 4: Creating the Spider

Navigate to the spider folder and create a file gfgfetch.py. When creating a spider, always create a class with a unique name and define the requirements. First, give the spider a name by assigning it to the variable name, then specify the starting URL where the spider will begin crawling. Define some methods to help crawl deeper into the site. For now, let's collect all the URLs present and save them.

import scrapy

 

class ExtractUrls(scrapy.Spider):

 

# This name must be unique always

name = "extract"

 

# Function which will be invoked

def start_requests(self):

 

# enter the URL here

urls = ['https://www.geeksforgeeks.org/', ]

 

for url in urls:

yield scrapy.Request(url = url, callback = self.parse)

The main goal is to get each URL and then request it. Extract all URLs or anchor tags from it. To do this, you need to create another method parse that will get data from the given URL.

Step 5: Getting data from a given page

Before writing the parse function, you need to check a few things, like how to get data from a given page. For this, you can use the scrapy shell. It is a Python interpreter but with the ability to extract data from a given URL. In short, it is a Python interpreter with Scrapy functionality.

scrapy shell URL

Note: Make sure that the scrapy.cfg file is in the same directory, otherwise it will not work.

scrapy shell https://www.geeksforgeeks.org/

Now to get data from a given page, selectors are used. These selectors can be from CSS or Xpath. Now let's try to get all URLs using a CSS selector.

• To get the anchor tag:

response.css('a')

• To extract data:

links = response.css('a').extract()

• For example, for links[0] it will show something like this:

'<a href="https://www.geeksforgeeks.org/" title="GeeksforGeeks" rel="home">GeeksforGeeks</a>'

• To get the href attribute, use the attributes tag.

links = response.css('a::attr(href)').extract()

This will get all href data, which is very useful. Use this link and start requesting it.

Now let's create the parse method and get all links, then yield them. Follow this URL and get other links from that page, and so on repeatedly. In short, we get all URLs present on that page.

Scrapy, by default, filters URLs that have already been visited. So it will not repeat the same URL path. But it is possible that two different pages have two or more identical links. For example, every page will have a link to the header, meaning that header link will appear in every page request. So try to exclude it by checking.

# Parse function
def parse(self, response):
 
# Extra feature to get title
title = response.css('title::text').extract_first()
 
# Get anchor tags
links = response.css('a::attr(href)').extract()
 
for link in links:
yield
{
'title': title,
'links': link
}
 
if 'geeksforgeeks' in link:
yield scrapy.Request(url = link, callback = self.parse)