Scraping a Web Page in C# with HtmlAgilityPack

Step by step

1. 1

   Install the package

   Run `dotnet add package HtmlAgilityPack` in your project.

2. 2

   Download the HTML

   Use a shared HttpClient instance to get the page as a string. Reuse the client instead of creating one per request.

3. 3

   Load and query

   Call `doc.LoadHtml(html)` and select elements with XPath, for example `//h2[@class='title']`.

4. 4

   Handle missing nodes

   SelectNodes returns null when nothing matches, so check for null before iterating.

Example

using HtmlAgilityPack;

var http = new HttpClient();
var html = await http.GetStringAsync("https://example.com/articles");

var doc = new HtmlDocument();
doc.LoadHtml(html);

var titles = doc.DocumentNode.SelectNodes("//h2[@class='title']");
if (titles != null)
{
    foreach (var node in titles)
        Console.WriteLine(node.InnerText.Trim());
}

Frequently Asked Questions