Web Scraping Tutorial with C#

Web scraping becomes easier when you use the right tools. Let's look at the best NuGet libraries for scraping in C#:

HtmlAgilityPack: the most popular parser library in C#. HtmlAgilityPack provides you with the ability to load web pages, parse their HTML content, select HTML elements, and extract the required data from them.
HttpClient: the most popular HTTP client in C#. HttpClient is especially useful for web crawling, as it allows you to make HTTP requests easily and asynchronously.
Selenium WebDriver — a library that supports multiple programming languages and allows you to write automated tests for web applications. You can also use it for web scraping.
Puppeteer Sharp — a C# port of Puppeteer. Puppeteer Sharp provides headless browser capabilities and allows you to use web scraping for pages with dynamic content.

This tutorial will show you how to perform web scraping using C#, as well as HtmlAgilityPack and Selenium.

Required Prerequisites for Web Scraping with C#

Before writing the first line of code for your C# web scraper, you need to meet some prerequisites:

Visual Studio: the free Community edition of Visual Studio 2022 will work.
.NET 6+: any LTS version starting from 6 will work.

Now you are ready to create a web scraping project in C# using Visual Studio.

Setting Up the Project in Visual Studio

Open Visual Studio and click on the "Create a new project" option.

In the "Create a new project" window, select the "C#" option from the dropdown list. After specifying the programming language, select the "Console App" template and click the "Next" button.

Then name your project StaticWebScraping, click "Select", and choose the .NET version. If you installed .NET 6.0, Visual Studio should already select it for you.

Click the "Create" button to initialize your C# web scraping project. Visual Studio will initialize the StaticWebScraping folder containing the App.cs file. This file will hold the C# web scraping logic:

namespace WebScraping {

public class Program {

public static void Main() {

// scraping logic...

}

}

}

It's time to understand how to create a web scraper in C#!

Web Scraping Static Content Sites with C#

On static sites, the content of web pages is stored in the HTML documents returned by the server. This means that a web page with static content does not make XHR requests to fetch data and does not require JavaScript execution.

Scraping static sites is quite simple. All you need to do is:

Install a C# web scraping library
Load the target web page and parse its HTML document
Use the web scraping library to select the HTML elements you are interested in
Extract data from them

Let's apply all these steps to the Wikipedia page "List of SpongeBob SquarePants episodes":

The goal of the C# web scraper you are about to create is to automatically retrieve all episode data from the static Wikipedia page.

Step 1: Install HtmlAgilityPack

HtmlAgilityPack is an open-source C# library that allows you to parse HTML documents, select elements from the DOM, and extract data from them. In essence, HtmlAgilityPack provides everything needed for web scraping data from a static website.

To install it, right-click on the "Dependencies" option under your project name in Solution Explorer. Then select "Manage NuGet Packages." In the NuGet Package Manager window, search for "HtmlAgilityPack" and click the "Install" button on the right side of the screen.

A popup will ask if you agree to make changes to your project. Click "OK" to install HtmlAgilityPack. You are now ready to perform web scraping with C# on a static website.

Now add the following line to the top of the App.cs file to import HtmlAgilityPack:

using HtmlAgilityPack;

Step 2: Loading the Web Page

You can connect to the target web page using HtmlAgilityPack as follows:

// the URL of the target Wikipedia page

string url = "https://en.wikipedia.org/wiki/List_of_SpongeBob_SquarePants_episodes";

var web = new HtmlWeb();

// downloading to the target page

// and parsing its HTML content

var document = web.Load(url);

An instance of the HtmlWeb class allows you to load a web page via its Load() method. Behind the scenes, this method performs an HTTP GET request to retrieve the HTML document associated with the URL passed as a parameter. Then Load() returns an instance of HtmlAgilityPack HtmlDocument, which can be used to select HTML elements on the page.

Step 3: Select HTML Elements

You can select HTML elements on a web page using XPath selectors. XPath allows you to select one or more specific DOM elements. To get the XPath selector associated with an HTML element, right-click on it, open the browser's developer tools, make sure it refers to the DOM element you're interested in, right-click on it, and select "Copy XPath."

The goal of the web scraper in C# is to extract data related to each episode.

Remember that you want to select all <tr> elements. Therefore, you need to modify the index associated with the row selection element. That is, you do not need to extract the first row of the table because it only contains table headers. In XPath, indices start at 1, so you can select all <tr> elements of the first episode table on the page by adding the XPath syntax position()>1.

Additionally, you want to retrieve data from tables of all seasons. On the Wikipedia page, the tables containing episode data are the second through fifteenth HTML tables in the HTML document. So, here's what the final XPath string will look like:

//*[@id='mw-content-text']/div[1]/table[position()>1 and position()<15]/tbody/tr[position()>1]

Now you can use the SelectNodes() method provided by HtmlAgilityPack to select the HTML elements you're interested in as follows:

var nodes = document.DocumentNode.SelectNodes("//*[@id='mw-content-text']/div[1]/table[position()>1 and position()<15]/tbody/tr[position()>1]");

Note that the SelectNodes() method can only be called on an instance of HtmlNode. Therefore, you need to get the root node of the HTML document using the DocumentNode property.

Also, keep in mind that XPath selectors are just one of many ways to select HTML elements on a web page. CSS selectors are another popular option.

Step 4: Extracting Data from HTML Elements

First, you need a custom class to store the data obtained through web scraping. Create an Episode.cs file in the WebScraping folder and initialize it as follows:

namespace StaticWebScraping {

public class Episode {

public string OverallNumber { get; set; }

public string Title { get; set; }

public string Directors { get; set; }

public string WrittenBy { get; set; }

public string Released { get; set; }

}

}

As you can see, this class has four attributes to store all the most important information about an episode. Note that OverallNumber is a string because the episode number in SpongeBob SquarePants always contains a lowercase character.

Now you can implement the web scraping logic in C# in your App.cs file, as shown below:

using HtmlAgilityPack;

using System;

using System.Collections.Generic;

namespace StaticWebScraping {

public class Program {

public static void Main() {

// the URL of the target Wikipedia page

string url = "https://en.wikipedia.org/wiki/List_of_SpongeBob_SquarePants_episodes";

var web = new HtmlWeb();

// downloading to the target page

// and parsing its HTML content

var document = web.Load(url);



// selecting the HTML nodes of interest

var nodes = document.DocumentNode.SelectNodes("/*[@id='mw-content-text']/div[1]/table[position()>1 and position()<15]/tbody/tr[position()>1]");



// initializing the list of objects that will

// store the scraped data

List<Episode> episodes = new List<Episode>();

// looping over the nodes

// and extract data from them

foreach (var node in nodes) {