Web Scraping JavaScript-Enabled Websites with Scrapy-Selenium

Scrapy-selenium is a middleware used for web scraping<\/a>. Scrapy does not support scraping modern websites that use JavaScript frameworks, which is why this middleware is used together with Scrapy to scrape these modern sites<\/a>. Scrapy-selenium provides the capabilities of Selenium that help work with JavaScript-heavy sites. Another advantage is the driver, through which we can see what is happening behind the scenes. Since Selenium is an automated tool<\/a>, it also gives us the ability to work with input tags<\/a> and perform scraping based on what you enter in the input field. Scrapy-selenium was first introduced in 2018 and is open source. An alternative to it is scrapy-splash.<\/p>\n\n

Installing and configuring Scrapy - <\/p>\n\n

\n\t

Install scrapy<\/li>\n\t

Run<\/li>\n<\/ul>\n\n

\n
scrapy startproject projectname (projectname - the project name)<\/p>\n<\/blockquote>\n\n
Now run,<\/p>\n\n
\n
scrapy genspider spidername example.com<\/p>\n<\/blockquote>\n\n
(Replace spidername with your preferred spider name, and example.com with the site you want to scrape). Note: The URL can also be changed later inside your Scrapy <\/p>\n\n
<\/p>\n\n
Install scrapy-selenium and add the following to the settings.py file <\/p>\n\n
\n
# for firefox
\nfrom shutil import which
\n
\nSELENIUM_DRIVER_NAME = 'firefox'
\nSELENIUM_DRIVER_EXECUTABLE_PATH = which('geckodriver')
\nSELENIUM_DRIVER_ARGUMENTS=['-headless']
\n <\/p>\n<\/blockquote>\n\n
\n
# for chrome driver
\nfrom shutil import which
\n
\nSELENIUM_DRIVER_NAME = 'chrome'
\nSELENIUM_DRIVER_EXECUTABLE_PATH = which('chromedriver')
\nSELENIUM_DRIVER_ARGUMENTS=['--headless']
\n DOWNLOADER_MIDDLEWARES = {
\n 'scrapy_selenium.SeleniumMiddleware': 800
\n }<\/p>\n<\/blockquote>\n\n
This project uses the Chrome driver. The Chrome driver must be downloaded according to your Chrome browser version. Go to the Chrome browser help section, then click About Google Chrome and check your version.
\n
\nRun the project:<\/p>\n\n
\n
command: scrapy crawl spidername (in this project, scrapy crawl integratedspider)<\/p>\n<\/blockquote>\n\n
Spider code before scrapy-selenium: <\/p>\n\n
\n
import scrapy
\n
\nclass IntegratedspiderSpider(scrapy.Spider):
\n name = 'integratedspider' # spider name
\n allowed_domains = ['example.com']
\n start_urls = ['http:\/\/example.com\/']
\n
\n def parse(self, response):
\n pass<\/p>\n<\/blockquote>\n\n
Important fields in scrapy-selenium:<\/p>\n\n
\n\t
name -<\/strong> This variable stores the spider name, and each spider is identified by this name. The command to run the spider is scrapy crawl spidername (here spidername refers to the name defined in the spider).<\/li>\n\t
function start_requests -<\/strong> The first requests are made by calling the start_requests() method, which generates a Request for the URL specified in the url field in yield SeleniumRequest, and the parse method as the callback function for the Requests.<\/li>\n\t
url -<\/strong> This specifies the site URL.<\/li>\n\t
screenshot -<\/strong> Using the get_screenshot_as_file() method, you can take a screenshot of the web page, specifying the filename as a parameter, which will be saved in the project.<\/li>\n\t
callback -<\/strong> The function that will be called with the response to this request as the first parameter.<\/li>\n\t
dont_filter -<\/strong> Indicates that this request should not be filtered by the scheduler. If the same URL is sent for scraping, it will not throw an exception that it has already been visited. This means the same URL can be accessed more than once. Default value is false.<\/li>\n\t
wait_time -<\/strong> Scrapy does not wait a fixed time between requests. But with this field, we can assign it during the callback.<\/li>\n<\/ol>\n\n
<\/p>\n\n
General structure of a scrapy-selenium spider: <\/p>\n\n
\n
import scrapy
\nfrom scrapy_selenium import SeleniumRequest
\n
\nclass IntegratedspiderSpider(scrapy.Spider):
\n name = 'integratedspider'
\n def start_requests(self):
\n yield SeleniumRequest(
\n url = "https:\/\/www.geeksforgeeks.org\/",
\n wait_time = 3,
\n screenshot = True,
\n callback = self.parse,
\n dont_filter = True
\n )
\n
\n def parse(self, response):
\n pass<\/p>\n<\/blockquote>\n\n
Code for scraping course data from Geeksforgeeks - <\/p>\n\n
\n
import scrapy
\nfrom scrapy_selenium import SeleniumRequest
\n
\nclass IntegratedspiderSpider(scrapy.Spider):
\n name = 'integratedspider'
\n def start_requests(self):
\n yield SeleniumRequest(
\n url = "https:\/\/practice.geeksforgeeks.org\/courses\/online",
\n wait_time = 3,
\n screenshot = True,
\n callback = self.parse,
\n dont_filter = True
\n )
\n
\n def parse(self, response):
\n # courses make list of all items that came in this xpath
\n # this xpath is of cards containing courses details
\n courses = response.xpath('\/*[@id ="active-courses-content"]\/div\/div\/div')
\n
\n # course is each course in the courses list
\n for course in courses:
\n # xpath of course name is added in the course path
\n # text() will scrape text from h4 tag that contains course name
\n course_name = course.xpath('.\/a\/div[2]\/div\/div[2]\/h4\/text()').get()
\n
\n # course_name is a string containing \\n and extra spaces
\n # these \\n and extra spaces are removed
\n
\n course_name = course_name.split('\\n')[1]
\n course_name = course_name.strip()
\n
\n
\n yield {
\n 'course Name':course_name
\n }<\/p>\n<\/blockquote>

Web Scraping JavaScript-Enabled Websites with Scrapy-Selenium

Why ESK Solutions

Related services

What will your project cost?

Related articles