How to Use the Web Scraper Chrome Extension for Data Extraction
This post is about DIY scraping tools.
How to Use the Web Scraper Chrome Extension for Data Extraction
Website scraping is becoming a vital ingredient in business planning and marketing regardless of industry. There are several ways to find useful data on the Internet depending on your requirements and budget. Did you know that your favorite web browser can also serve as an excellent data collection tool?
You can install the Web Scraper extension from the Chrome Web Store to turn it into an easy-to-use data collection tool.
The best part is that you can stay in the comfort zone of your browser while data collection occurs. It does not require much technical skill, making it a good option when you need to gather data quickly. Let's start with a tutorial on how to use the Web Scraper Chrome extension for data extraction.
About the Web Scraper Chrome Extension
Web Scraper is a web data extraction extension for the Chrome browser, created solely for collecting data from the web. You can set a sitemap to navigate a website and specify the data to extract.
The scraper will navigate the site according to the settings and extract the desired data. It allows you to export extracted data to CSV. This tool can scrape multiple pages, making it even more powerful. It can even extract data from dynamic pages that use Javascript and Ajax.
What You Need
- Google Chrome browser
- Working internet connection
A. Installation and Setup
- Install the Web Scraper Chrome extension
- To download the Web Scraper Chrome extension, click the "Add" button.
Once that is done, you will be ready to start scraping any website with the Chrome browser. You just need to learn how to perform data scraping, which we will cover now.
B. Method
After installation, open the Google Chrome Developer Tools by pressing F12. (Alternatively, you can right-click anywhere on the screen and select "Inspect"). In the Developer Tools, you will find a new tab called "Web Scraper".
Now let's see how to use this on a live web page. For this tutorial, we will use a site called www.awesomegifs.com.
This site contains GIF images, and we will scrape the URLs of these images using our scraper.
Step 1: Create a Sitemap
- Go to https://www.awesomegifs.com/.
- Open Developer Tools by right-clicking anywhere on the screen and selecting "Inspect".
- Go to the "Web Scraper" tab in Developer Tools
- Click "Create new sitemap", then select "Create sitemap".
- Give the sitemap a name and enter the site URL in the start URL field.
- Click the "Create sitemap" button.
To scrape multiple pages of the site, we need to understand the pagination structure of this site. This can easily be done by clicking the "Next" button on the homepage several times.
On Awesomegifs.com, it was found that pages are structured as https://awesomegifs.com/page/1/, https://awesomegifs.com/page/2/, and so on. To navigate to another page, simply change the number at the end of this URL. Now we need the scraper to do this automatically.
To do this, create a new sitemap with the start URL https://awesomegifs.com/page/[001-125]. Now the scraper will open the URL multiple times, incrementing the end value each time. This means the scraper will open pages from 1 to 125 and extract the elements we need on each page.
Step 2: Collect Elements
Each time the scraper opens a page of the site, we need to extract some elements. In this case, it's the URLs of GIF images.
First, we need to find the CSS selector that matches the images. The CSS selector can be found by viewing the webpage source (CTRL+U). A simpler way is to use the selector tool to click and select any element on the screen. Click on the sitemap you just created, then click the "Add new selector" button.
In the selector id field, give the selector a name. In the type field, select the type of data to be extracted. Click the "Select" button and choose any element on the webpage that you want to extract. When you finish selecting, click the "Done" button.
It's as simple as clicking the mouse on the element. You can check the 'multiple' box to indicate that the element you want may appear multiple times on the page and that you want each instance to be captured.
Now you can save the selector if everything looks good. To start the scraping process, simply click on the sitemap tab and select 'Scrape'. A new window will appear that visits each page in the loop and collects the necessary data. If you want to stop the data scraping process, simply close this window, and you will get the data that was extracted up to that point.
Once you stop the scraping process, go to the sitemap tab to view the extracted data or export it to a CSV file. The only drawback of such data extraction software is that you will have to manually search each time, as it lacks many automation features.
If you want to collect data at scale, it is better to use data collection services rather than such free data extraction tools. In the second part of this series, we will show you how to create a MySQL database using the extracted data. Stay tuned!