Web Scraping with Ruby - A Quick Tutorial

The remarkable growth and exponential increase in the volume of data on the Internet have opened up new opportunities for various industries. From manufacturing enterprises to the service sector, data is a critical component used by businesses worldwide to stay relevant in evolving conditions.

Web data not only holds a goldmine of competitive and market information but also offers deep insights that can be used to improve internal processes and operations.

Website parsing services help extract targeted data from the Internet for further use by analytical engines or BI tools. The purpose of parsing remains diverse:

• Data extraction is an effective way to advertise your business and promote products/services.
• Users, consumers, and website visitors can obtain the desired information about a service or product.
• Companies can gain competitive intelligence on strategies and plans to increase their market share.
• Brands can learn the general perception of their brand through people's interactions on social media. This helps marketing teams develop and implement appropriate marketing messages tailored specifically to that audience, thereby increasing the likelihood of conversion.
• Businesses can gain a clearer understanding of their target audience's needs, pain points, and preferences. They can then direct product development in the right direction using this valuable data.

Imagine the benefits if we can structure web data, eliminate noise, and export it into machine-readable formats. Let's see how this can be done using the Ruby language.

Choosing a Coding Script

Data extraction and the real-world application of parsing techniques are not trivial. Basic knowledge of CSS, HTML, and the right choice of library with code will make your journey easy. Choosing a coding script plays a crucial role in this context. Let's find out why Ruby is making waves in the market.

If you are planning to run your first parsing program, Ruby can serve as a reliable scripting language. Quite a few reasons are responsible for the unparalleled popularity of this language, and the following reasons will help you understand why it is so effective!

• Powerful script: Ruby-On-Rails is a very powerful and efficient parsing script. For beginners and novices, this language has proven to be a powerful resource.
• Robust community: Ruby comes with a strong team of developers who form a reliable and highly dependable community. With millions of documentation entries, no question will be unsolvable for you!
• Easy installation: The installation procedure is well-documented and quite simple.

Here are some of the factors that make Ruby an indispensable choice for parsing. Setup and installation must be done optimally, as these processes are critical for executing data extraction processes. Here is a comprehensive guide to help you through this process.

Step-by-Step Guide

Before we start, let's clarify some points. This guide is intended for Mac users; if you are using a different machine, the initial setup process may vary slightly. Secondly, the program uses Nokogiri, which can transform web pages into "Ruby objects," thereby simplifying the parsing process. With these two factors in mind, you can start working on your projects.

In this tutorial, we will collect the titles of the first 100 used car listings on OLX.

Setup Process

Here are the basic requirements for developing a complete setup for parsing information from the web using Ruby.

• Ruby must be installed on your computer, whether it is a desktop or laptop. If you are a Mac user, half the work is already done.
• You will need a text editor. It is necessary for writing program commands. If your computer does not have a built-in option, try downloading Sublime Text. This text editor with exciting features and easy management will make coding fun and interesting.
• Another prerequisite is an in-depth knowledge of HTML and CSS usage. If you plan to master the art of parsing, knowledge of CSS and HTML will be crucial.
• Get knowledge about Ruby. A bit of information is very important in this context. You can check some online courses and enhance your knowledge base. Once all these processes and factors are taken care of, it's time to proceed with the crucial steps.

Step 1: Installing Dependencies

During the installation process, make sure you get full information about three useful Ruby Gems. These three options include:

• Nokogiri
• HTTParty
• Pry

Since we've already discussed Nokogiri a bit, let's talk about HTTParty and Pry. HTTParty is a gem that our parser will use to send HTTP requests to the pages we are scraping.

We will use HTTParty to send GET requests, which will return all the HTML content of the page as a string. For debugging, we use Pry, which is a Ruby gem. It will help us parse the web page code and is an important component in this setup.

Execute the following commands and run them on your machine to install these gems on your computer.

gem install nokogiri

gem install httparty

gem install pry

Step 2: Creating Parser Files

You will need to create a folder named nokogiri_tutorial in any convenient location on your computer. The desktop is an ideal place for this. The next step is to download a text editor like Sublime Text or any other of your choice and save a file in this folder named web_scraper.rb. Once you complete these steps, you can work on the dependencies.

Step 3: Sending HTTP Requests to the Page

Start by creating an operation variable named page and ensure it is equal to an HTTParty GET request of the page we will be parsing.

In this case: https://www.olx.com/all-results/?q=cars.

After this, you can enter Pry.start(binding). Navigate to and find the folder marked as the file web_scraper.rb. Immediately save it on the desktop and open the terminal by entering this command:

cd desktop/nokogiri_tutorial

Your web scraping program is ready to be implemented. You can run this command and execute it:

ruby web_scraper.rb

The terminal should switch to Pry, and it is very important to check the layout before proceeding further. You can move on to the next step. But before doing so, make sure you type exit in the selected terminal, exit Pry, and then return to the original folder location of the program.

Step 4: Transitioning to Nokogiri

The task is to first convert and transform these car listings into Nokogiri objects, as this is crucial for analysis. Creating variables is very important, and you will need to create a new variable named parse_page. Nokogiri has a special way of converting HTML strings into Nokogiri objects. You can leave Pry at the bottom of the code.

The next step is to save the file containing the Ruby command. Pry will open automatically, and you need to enter the new variable parse_page. This will return the OLX page as a Nokogiri object.

Create an HTML file in the same folder named cars.html and copy-paste the results of the parse_page command into this file. These formatted HTML data will be useful later for reference.

Before proceeding to the next step, exit Pry in the terminal.

Step 5: Parsing Data

Parsing data requires basic knowledge of programming as well as coding. Since you want to extract the title texts of all car sale listings, the cars.html file will be useful for cross-checking. Find the important elements in the folder and inspect them using the 'inspect element' tool; you can also view the 'page source code'.

Since we found that the listings are inside a div with the class name content, the subsequent commands will be:

parse_page.css('.content')

parse_page.css('.content').css('.row').css('.hdrlnk')

cars_array

Check the layouts and encoding arrays each time you execute a command. Once parsing is complete, you need to export the datasets to CSV files.

Step 6: Exporting Data Files to CSV

When you reach Step 6, you should have successfully completed the parsing process, and the unstructured data has become structured datasets. Now let's go back to the terminal.

Exit Pry if you are still in it, so that your terminal is in the nokogiri_tutorial folder, which contains the parsing program and the cars.html file. Now enter the following command:

touch cars.csv

Now you will have an empty CSV file into which you can save the data from cars_array. You can then write a simple script to write this data into our new CSV file, and you will have structured data of car listings in the CSV file. This will make processing and manipulation easier whenever you want.

Closing Thoughts

We hope this gave you a rough idea of how you can start building a parsing project with Ruby. It's time to explore and scrape more complex and difficult sites using this newly acquired skill.