C# GuideBeginner → Production

Web Scraping with C# — Complete Guide

From HttpClient to production-grade async scrapers. Type-safe web data extraction with .NET.

API Reference

C# and .NET provide a robust foundation for web scraping with excellent async support, strong typing, and mature HTML parsing libraries. HttpClient handles HTTP natively, while HtmlAgilityPack and AngleSharp parse HTML with XPath and CSS selectors respectively. This guide covers the full workflow from fetching pages to building concurrent, production-ready scrapers.

Setting Up Your C# Scraping Environment

Create a .NET console project and install HTML parsing packages. HtmlAgilityPack provides XPath queries, while AngleSharp offers a DOM API with CSS selectors.

dotnet new console -n MyScraper
cd MyScraper
dotnet add package HtmlAgilityPack
dotnet add package AngleSharp

HttpClient is built into .NET — no additional package needed for HTTP requests.

dotnet new console -n MyScraper
cd MyScraper
dotnet add package HtmlAgilityPack
dotnet add package AngleSharp

Fetching a Web Page with C#

Use HttpClient with async/await for non-blocking HTTP requests. Always use a single HttpClient instance (or IHttpClientFactory) to avoid socket exhaustion. Set a timeout and default request headers.

using System.Net.Http;

var client = new HttpClient { Timeout = TimeSpan.FromSeconds(10) };
client.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (compatible; DataBot/1.0)");
client.DefaultRequestHeaders.Add("Accept", "text/html");

var response = await client.GetAsync("https://example.com/products");
response.EnsureSuccessStatusCode();
var html = await response.Content.ReadAsStringAsync();
Console.WriteLine($"Fetched {html.Length} bytes");

Parsing HTML with HtmlAgilityPack

HtmlAgilityPack uses XPath for element selection. Load the HTML, then use SelectNodes() with XPath expressions. It handles malformed HTML gracefully — essential for real-world scraping.

using HtmlAgilityPack;

var doc = new HtmlDocument();
doc.LoadHtml(html);

var cards = doc.DocumentNode.SelectNodes("//div[contains(@class, 'product-card')]");
if (cards != null)
{
    foreach (var card in cards)
    {
        var title = card.SelectSingleNode(".//h2[contains(@class, 'product-title')]")?.InnerText.Trim();
        var price = card.SelectSingleNode(".//span[contains(@class, 'price')]")?.InnerText.Trim();
        var href = card.SelectSingleNode(".//a")?.GetAttributeValue("href", "");
        Console.WriteLine($"{title}: {price} ({href})");
    }
}

Using CSS Selectors with AngleSharp

AngleSharp provides a full DOM implementation with CSS selector support — closer to browser APIs than XPath. It supports querySelector and querySelectorAll just like JavaScript.

using AngleSharp;
using AngleSharp.Html.Parser;

var parser = new HtmlParser();
var document = await parser.ParseDocumentAsync(html);

var cards = document.QuerySelectorAll("div.product-card");
foreach (var card in cards)
{
    var title = card.QuerySelector("h2.product-title")?.TextContent.Trim();
    var price = card.QuerySelector("span.price")?.TextContent.Trim();
    var href = card.QuerySelector("a")?.GetAttribute("href");
    Console.WriteLine($"{title}: {price} ({href})");
}

Concurrent Scraping with Task.WhenAll

Use C#'s async/await with SemaphoreSlim to control concurrency. Task.WhenAll runs multiple scraping tasks in parallel while the semaphore limits the number of simultaneous requests.

var urls = Enumerable.Range(1, 50)
    .Select(p => $"https://example.com/page/{p}")
    .ToList();

var semaphore = new SemaphoreSlim(10);
var results = new System.Collections.Concurrent.ConcurrentBag<Product>();

var tasks = urls.Select(async url =>
{
    await semaphore.WaitAsync();
    try
    {
        var response = await client.GetStringAsync(url);
        var doc = new HtmlDocument();
        doc.LoadHtml(response);
        foreach (var product in ExtractProducts(doc))
            results.Add(product);
    }
    catch (Exception ex)
    {
        Console.Error.WriteLine($"Error: {url} - {ex.Message}");
    }
    finally
    {
        semaphore.Release();
    }
}).ToList();

await Task.WhenAll(tasks);
Console.WriteLine($"Collected {results.Count} products");

Handling JavaScript-Rendered Pages

HttpClient and HTML parsers fetch raw HTML — they do not execute JavaScript. For pages that render content dynamically, you need a rendering solution.

Options: use Playwright for .NET or Selenium WebDriver for browser automation, or use a cloud rendering API. Running browsers adds significant complexity and resource usage.

AlterLab's API handles rendering server-side. Send a POST request with render_js: true and receive fully rendered HTML — no browser dependencies required.

Common Pitfalls and How to Avoid Them

The most common issues in C# web scraping:

**1. HttpClient socket exhaustion** — Creating new HttpClient instances per request exhausts sockets. Use a single instance or IHttpClientFactory.

**2. Null reference exceptions** — HTML elements may not exist. Always use null-conditional operators: node?.InnerText?.Trim().

**3. Encoding issues** — HttpClient auto-detects encoding, but some pages lie about their charset. Use Encoding.GetEncoding() to force the correct encoding.

**4. No timeout set** — The default HttpClient timeout is 100 seconds — too long for scraping. Set it to 10-15 seconds.

**5. Missing error handling** — Use try/catch around HTTP calls and check EnsureSuccessStatusCode(). Implement retry logic with Polly for transient failures.

Complete C# Scraper — Async with CSV Output

Complete working scraper with concurrency, error handling, and CSV output.

using System.Globalization;
using System.Net.Http;
using CsvHelper;
using HtmlAgilityPack;

record Article(string Title, string Url, string Date);

var client = new HttpClient { Timeout = TimeSpan.FromSeconds(10) };
client.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (compatible; DataBot/1.0)");
client.DefaultRequestHeaders.Add("Accept", "text/html");
client.DefaultRequestHeaders.Add("Accept-Language", "en-US,en;q=0.9");

async Task<string> FetchPage(string url)
{
    var response = await client.GetAsync(url);
    response.EnsureSuccessStatusCode();
    return await response.Content.ReadAsStringAsync();
}

List<Article> ParseArticles(string html)
{
    var doc = new HtmlDocument();
    doc.LoadHtml(html);
    var articles = new List<Article>();
    var nodes = doc.DocumentNode.SelectNodes("//article[contains(@class, 'post')]");
    if (nodes == null) return articles;
    foreach (var node in nodes)
    {
        articles.Add(new Article(
            node.SelectSingleNode(".//h2")?.InnerText.Trim() ?? "",
            node.SelectSingleNode(".//a")?.GetAttributeValue("href", "") ?? "",
            node.SelectSingleNode(".//time")?.GetAttributeValue("datetime", "") ?? ""
        ));
    }
    return articles;
}

var allArticles = new List<Article>();
for (int page = 1; page <= 10; page++)
{
    var url = $"https://example.com/articles?page={page}";
    Console.WriteLine($"Scraping page {page}: {url}");
    try
    {
        var html = await FetchPage(url);
        var articles = ParseArticles(html);
        if (articles.Count == 0) { Console.WriteLine("No more results"); break; }
        allArticles.AddRange(articles);
        await Task.Delay(1000);
    }
    catch (Exception ex)
    {
        Console.Error.WriteLine($"Error on page {page}: {ex.Message}");
        break;
    }
}

await using var writer = new StreamWriter("output.csv");
await using var csv = new CsvWriter(writer, CultureInfo.InvariantCulture);
await csv.WriteRecordsAsync(allArticles);
Console.WriteLine($"Saved {allArticles.Count} articles");

Or Skip the Complexity — Use AlterLab

AlterLab handles JavaScript rendering, IP rotation, and website compatibility automatically. One POST request returns rendered HTML — no browser management, no proxy configuration. Starts at $0.0002/request with 5,000 free to start.

using System.Net.Http;
using System.Text;
using System.Text.Json;
using HtmlAgilityPack;

const string ApiKey = "YOUR_API_KEY"; // Get free at alterlab.io

var client = new HttpClient { Timeout = TimeSpan.FromSeconds(30) };

async Task<string> ScrapeWithAlterLab(string targetUrl, bool renderJs = false)
{
    var payload = JsonSerializer.Serialize(new { url = targetUrl, render_js = renderJs });
    var request = new HttpRequestMessage(HttpMethod.Post, "https://api.alterlab.io/api/v1/scrape")
    {
        Content = new StringContent(payload, Encoding.UTF8, "application/json")
    };
    request.Headers.Add("X-API-Key", ApiKey);

    var response = await client.SendAsync(request);
    response.EnsureSuccessStatusCode();
    var json = await response.Content.ReadAsStringAsync();
    using var doc = JsonDocument.Parse(json);
    return doc.RootElement.GetProperty("html").GetString() ?? "";
}

var html = await ScrapeWithAlterLab("https://example.com/products", renderJs: true);

var htmlDoc = new HtmlDocument();
htmlDoc.LoadHtml(html);
var cards = htmlDoc.DocumentNode.SelectNodes("//div[contains(@class, 'product-card')]");
if (cards != null)
{
    foreach (var card in cards)
    {
        var title = card.SelectSingleNode(".//h2")?.InnerText.Trim();
        var price = card.SelectSingleNode(".//*[contains(@class, 'price')]")?.InnerText.Trim();
        Console.WriteLine($"{title}: {price}");
    }
}
API Docs

Choosing Your Approach

HttpClient + HtmlAgilityPack

Pros

  • +Built-in HttpClient — no external HTTP dependency
  • +XPath queries are powerful
  • +Strong typing catches errors at compile time

Cons

  • −No JavaScript execution
  • −XPath is verbose compared to CSS selectors
  • −More boilerplate than Python

AngleSharp

Pros

  • +CSS selectors — familiar for web developers
  • +Full DOM implementation
  • +Async-first API design

Cons

  • −No JavaScript execution
  • −Slightly higher memory usage than HAP
  • −Less widely used than HtmlAgilityPack

AlterLab API

Pros

  • +Handles static, JavaScript, and challenge pages
  • +No browser management
  • +Automatic IP rotation
  • +5-tier auto-escalation
  • +From $0.0002/request

Cons

  • −Per-request cost
  • −Requires network access

Frequently Asked Questions

More C# & Scraping Resources

Web Scraping with Java

Java scraping guide: Jsoup, HttpClient, and virtual thread patterns.

Web Scraping with Python

Complete Python scraping guide: requests, BeautifulSoup, async patterns from beginner to production.

JavaScript Rendering API

Render SPAs and dynamic content with headless Chromium — no browser management.

Pricing

From $0.0002/request. No subscriptions. Balance never expires.

Your first scrape.
Sixty seconds.

$1 free credit — up to 5,000 scrapes. No credit card.Just a POST request.

terminal
curl -X POST https://api.alterlab.io/v1/scrape \
-H "X-API-Key: YOUR_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com", "formats": ["markdown"]}'

No credit card required · $1 free credit, up to 5,000 scrapes · Balance never expires