cURL: What it is and how to use it for web scraping

What is cURL?

cURL is a command-line tool that can be used to transfer data over network protocols. The name cURL stands for "Client URL" as well as "curl". This popular command uses URL syntax to transfer data to and from servers. Curl works based on 'libcurl', a free and easy-to-use client-side URL transfer library.

Why is using curl beneficial?

The versatility of this command allows you to use curl for various purposes, including:

• User authentication
• HTTP posts
• SSL connections
• Proxy server support
• FTP uploads

The simplest "scenario" for using curl is downloading and uploading entire sites using one of the supported protocols.

cURL protocols

Although curl has a long list of supported protocols, it will default to HTTP if you do not specify a specific protocol. Here is the list of supported protocols:

The cURL command - list of supported protocols - dict, file, ftp, ftps, gopher, http, https, imap, imaps, ldap, pop3, stmp, rtsp, scp, sftp, smb, sms, telnet, tftp

Installing curl

The curl command is installed by default in Linux distributions.

How to check if the curl command is installed?

1. Open a Linux console
2. Type 'curl' and press 'enter'.
3. If curl is already installed, you will see the following message:

4. If curl is not yet installed, the following message will appear: 'command not found'. Then you can refer to your distribution and install it (more on this below).

How to use cURL

Curl syntax is quite simple:

For example, if you want to download a web page: webpage.com, simply run the command:

The command will output the page source code in the terminal window. Keep in mind that if you do not specify a protocol, curl will use HTTP by default. Below is an example of specifying specific protocols:

If you forget to add ://, curl will guess which protocol you want to use.

We briefly covered basic command usage, but the list of options can be found in the curl documentation. Options are possible actions you can perform on a URL. When you choose an option, it tells curl what action to perform on the specified URL. The URL tells cURL where to perform that action. cURL then allows you to list one or more URLs.

To download multiple URLs, prefix each URL with -0 followed by a space. This can be done in one line or on separate lines for each URL. You can also download part of a URL by listing pages. For example:
 

Saving downloads

With curl you can save the contents of a URL to a file using two different methods:

1. The -o method: Allows you to add a filename to which the URL will be saved. This option has the following structure:

2. The -O method: Here you do not need to add a filename, as this option allows you to save the file under the URL's name. To use this option, simply prefix the URL with -O.

Resuming downloads

It may happen that a download stops halfway. In that case, rewrite the command by adding the -C option at the beginning:

Why is curl so popular?

Curl is truly a "swiss army knife" of commands, built to perform complex operations. However, there are alternatives, such as 'wget' or 'Kurly', which are suitable for simpler tasks.

Curl is a favorite among developers because it is available on almost all platforms. Sometimes it is even installed by default. This means that no matter what programs/workstations you run, curl commands should work.

Additionally, chances are if your OS is less than ten years old, curl is already installed. You can also read the documentation in your browser and check the curl documentation. If you are using the latest version of Windows, you likely already have curl installed. Otherwise, check out this Stack Overflow article to learn more about how to do it.

Using cURL with proxy servers

Some users prefer to use cURL together with a proxy server. Benefits include:

Increased ability to successfully manage data requests from different geolocations.
Exponential increase in the number of concurrent data tasks. 
To do this, you can use the built-in cURL options '-x' and '(- - proxy)'.
Good to know: cUrl is compatible with most common proxy types, including HTTP, HTTPS, and SOCKS.

How to change the User-Agent

User-Agent is a characteristic that allows target sites to identify the device requesting information. The target site may require certain criteria from requesters to return the desired data. This can involve device type, operating system, or browser used. In this case, organizations collecting data strive to emulate the ideal "candidate" for the target site. 

For example, suppose the site you are targeting "prefers" Chrome as its browser. To get the desired data set with cURL, you need to emulate this "browser characteristic" as follows: 

curl -A "Goggle/9.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Chrome/103.0.5060.71" https://getfedora.org/.

Scraping with cURL

Tip: Always comply with site rules, and do not attempt to access password-protected content, which is often illegal or at least discouraged.

With curl you can automate the repetitive process of web scraping, helping you avoid tedious tasks. To do this, you need to use PHP. Here is an example we found on GitHub:

When you use curl to scrape a web page, you should use three options:

• curl_init($url) -> Initializes a session
• curl_exec() -> Executes
• curl_close() -> Closes

Other options to use include:

curlopt_url -> Sets the URL to scrape

Curlopt_returntransfer -> Instructs curl to save the scraped page as a variable. (This allows you to get exactly what you wanted to extract from the page).

Conclusion

Although cURL is a powerful web scraping tool, it requires companies to spend valuable developer time both on data collection and data cleaning.