Using Python and Wget to Download Web Pages and Files

In many contexts, such as automation, data science, data engineering, and application development, Python is the lingua franca. It is widely used for downloading images and web pages, with many methods and packages available. One simple yet reliable method is interacting with Wget.

Wget is a free command-line program that has been around for twenty-five years, enabling file retrieval from web services via HTTP, HTTPS, and FTP protocols. When used together with Python, it imposes virtually no limits on downloading and extracting files from the Internet.

This article will demonstrate the advantages of using Wget in Python with simple examples. You will also learn about its limitations and alternatives.

Why Use Wget?

Wget is a convenient and widely supported tool for downloading files over three protocols: HTTP, HTTPS, and FTP. Its popularity stems from two main characteristics: recursiveness and reliability.

Recursiveness: When using the appropriate options, Wget can act as a web crawler. Instead of downloading a single file, it can recursively download files linked from a specific web page until all links are exhausted or until a user-defined recursion depth is reached. In this case, Wget saves the downloaded files in a directory structure that mirrors the server they were downloaded from. This capability is highly customizable:

• Supports wildcards in network locations and file names
• Offers timestamp checking so that only new or updated files are downloaded
• Adheres to the robots exclusion standard

Reliability: Wget can resume interrupted transfers, making it a good solution for downloading files over unstable or slow networks. Wget uses the HTTP Range header to continue downloading from where it was interrupted until the entire file is retrieved. This requires no user intervention.

Wget2 was released in 2021. While it supports roughly the same features, it emphasizes parallelization, making it much faster than its predecessor.

Why Wget with Python?

Python is a general-purpose programming language used in finance, academia, cloud/data engineering, data science, web development, and workflow automation. It is not only widely used across various fields and industries, but also has a huge community, is the most popular programming language at Google, and tops the list of most in-demand programming languages in job postings.

By using Wget, you can easily turn Python scripts into full-fledged solutions for retrieving information from the Internet. Interesting use cases include:

• Creating datasets for academic and business purposes. With Wget, you can easily scrape content from one or multiple websites. Such large datasets can be crucial for machine learning research. For instance, recent NLP models would be impossible without billions of pieces of content.
• Monitoring large websites. Automate Wget to check the availability of web pages and files from various networks and locations around the world.
• Content comparison. Many web pages generate personalized content. By configuring Wget to behave like different personas, you can gain insight into what content is shown to specific users.

Although there are many ways to run shell commands and programs (like Wget) in Python, this tutorial will use the subprocess package to interact with the operating system shell.

Note that the Python wget package, while sharing some common functionality, is unrelated to the Wget command-line program. It is an underdeveloped package that has not been updated in several years and lacks most of Wget's distinctive features.

Using Wget in Python

Next, you will set up Wget for downloading files in Python.

Prerequisites.
First, make sure Wget is installed on your computer. The installation process depends on your operating system.

• If you are using Linux, it may already be preinstalled.
• If you are using a Mac, the easiest way is to install Wget using Homebrew.
• Windows users can download the Wget command-line tool executable from this site. After downloading, make sure it is added to the PATH variable.

Executing Commands with the Subprocess Package

To execute Wget commands inside a Python script, we use the Popen method of the subprocess package. Each time your script calls popen(), it will execute the passed command in an independent instance of the operating system command processor. If you set the verbose argument to True, it will also return the command's output. Feel free to adapt this code to your needs.

All code snippets can be found in this file.

import subprocess

def runcmd(cmd, verbose = False, *args, **kwargs):

    process = subprocess.Popen(
        cmd,
        stdout = subprocess.PIPE,
        stderr = subprocess.PIPE,
        text = True,
        shell = True
    )
    std_out, std_err = process.communicate()
    if verbose:
        print(std_out.strip(), std_err)

runcmd('echo "Hello, World!"', verbose = True)

All commands you will use in this section are structured similarly. You use the wget command, give it a URL, and specify certain options to achieve specific goals.

wget \[options\] url

When Not to Use Wget

Wget is an excellent solution for recursive file downloads from web servers. However, its application is limited due to this narrow focus, so it's worth considering alternatives.

• For downloading files over protocols other than HTTP(S) or FTP(S), it is best to use cURL with Python.
• If you need to parse only specific DOM elements on a web page without saving the file locally, consider using requests in combination with [Beautiful Soup]

Selenium is an excellent solution for simulating click and scroll behavior on a website (e.g., for testing purposes).

Conclusion

Wget is a convenient solution for downloading files over HTTP and FTP protocols. It works well with Python for recursive downloads of multiple files, and this process can be easily automated to save time.

Perhaps Wget's focus is somewhat limited, but it offers many capabilities for file downloading and web scraping.