Web Scraping with Rust — Complete Guide
From reqwest to production-grade async scrapers. Type-safe, memory-safe web data extraction with Rust.
Rust brings memory safety, zero-cost abstractions, and fearless concurrency to web scraping. The reqwest + scraper crate combination handles HTTP and HTML parsing, while tokio provides the async runtime for concurrent requests. This guide covers the full journey from fetching your first page to building production-scale scrapers.
Setting Up Your Rust Scraping Environment
Add the core scraping crates to your Cargo.toml. reqwest handles HTTP requests, scraper parses HTML with CSS selectors, and tokio provides the async runtime.
[dependencies]
reqwest = { version = "0.12", features = ["json"] }
scraper = "0.20"
tokio = { version = "1", features = ["full"] }
serde = { version = "1", features = ["derive"] }
serde_json = "1"For CSV output, add the csv crate: csv = "1".
[dependencies]
reqwest = { version = "0.12", features = ["json"] }
scraper = "0.20"
tokio = { version = "1", features = ["full"] }
serde = { version = "1", features = ["derive"] }
serde_json = "1"Fetching a Web Page with Rust
Use reqwest to send HTTP requests. The async API integrates with tokio for non-blocking I/O. Always set a timeout and user-agent header.
use reqwest::Client;
use std::time::Duration;
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let client = Client::builder()
.timeout(Duration::from_secs(10))
.user_agent("Mozilla/5.0 (compatible; DataBot/1.0)")
.build()?;
let response = client
.get("https://example.com/products")
.header("Accept", "text/html")
.send()
.await?;
let status = response.status();
let body = response.text().await?;
println!("Fetched {} bytes (status {})", body.len(), status);
Ok(())
}Parsing HTML with the scraper Crate
The scraper crate provides CSS selector support built on the html5ever parser. Parse the HTML into a document, create a selector, and iterate over matching elements.
use scraper::{Html, Selector};
let document = Html::parse_document(&body);
let card_selector = Selector::parse("div.product-card").unwrap();
let title_selector = Selector::parse("h2.product-title").unwrap();
let price_selector = Selector::parse("span.price").unwrap();
for card in document.select(&card_selector) {
let title = card.select(&title_selector)
.next()
.map(|el| el.text().collect::<String>())
.unwrap_or_default();
let price = card.select(&price_selector)
.next()
.map(|el| el.text().collect::<String>())
.unwrap_or_default();
println!("{}: {}", title.trim(), price.trim());
}Extracting Structured Data
Define a struct with serde derive macros for serialization. Collect parsed data into a Vec and serialize to JSON or CSV. Rust's type system catches extraction errors at compile time.
use serde::Serialize;
#[derive(Debug, Serialize)]
struct Product {
title: String,
price: String,
url: String,
}
fn extract_products(html: &str) -> Vec<Product> {
let document = Html::parse_document(html);
let card_sel = Selector::parse("div.product-card").unwrap();
let title_sel = Selector::parse("h2").unwrap();
let price_sel = Selector::parse(".price").unwrap();
let link_sel = Selector::parse("a").unwrap();
document.select(&card_sel).map(|card| {
Product {
title: card.select(&title_sel).next()
.map(|e| e.text().collect::<String>().trim().to_string())
.unwrap_or_default(),
price: card.select(&price_sel).next()
.map(|e| e.text().collect::<String>().trim().to_string())
.unwrap_or_default(),
url: card.select(&link_sel).next()
.and_then(|e| e.value().attr("href"))
.unwrap_or("")
.to_string(),
}
}).collect()
}Async Concurrent Scraping with tokio
Use tokio::spawn and a semaphore to limit concurrent requests. Rust's ownership system prevents data races at compile time — no runtime locks needed for read-only data.
use tokio::sync::Semaphore;
use std::sync::Arc;
async fn scrape_many(urls: Vec<String>, concurrency: usize) -> Vec<Product> {
let semaphore = Arc::new(Semaphore::new(concurrency));
let client = Arc::new(Client::builder()
.timeout(Duration::from_secs(10))
.user_agent("Mozilla/5.0 (compatible; DataBot/1.0)")
.build()
.unwrap());
let mut handles = vec![];
for url in urls {
let sem = semaphore.clone();
let client = client.clone();
handles.push(tokio::spawn(async move {
let _permit = sem.acquire().await.unwrap();
match client.get(&url).send().await {
Ok(resp) => {
let html = resp.text().await.unwrap_or_default();
extract_products(&html)
}
Err(e) => {
eprintln!("Error scraping {}: {}", url, e);
vec![]
}
}
}));
}
let mut results = vec![];
for handle in handles {
results.extend(handle.await.unwrap_or_default());
}
results
}Handling JavaScript-Rendered Pages
Rust's HTTP clients fetch raw HTML — they do not execute JavaScript. For pages that render content client-side, you need a rendering solution.
Options: use the headless_chrome crate (Rust bindings for Chrome DevTools Protocol), or use a cloud rendering API. Running browsers from Rust adds significant complexity and binary size.
AlterLab's API handles rendering server-side. Send a POST request with render_js: true and receive fully rendered HTML — no browser dependencies required.
Common Pitfalls and How to Avoid Them
The most common issues in Rust web scraping:
**1. Selector::parse panics** — Selector::parse returns a Result but many examples unwrap it. In production, handle the error or cache selectors as lazy_static/once_cell values.
**2. Blocking the async runtime** — CPU-heavy HTML parsing can block tokio's thread pool. Use tokio::task::spawn_blocking for heavy parse operations.
**3. Missing TLS features** — reqwest needs the rustls-tls or native-tls feature flag. Without it, HTTPS requests fail at compile time.
**4. String encoding issues** — reqwest assumes UTF-8. Use encoding_rs for pages with other encodings.
**5. Large dependency tree** — reqwest + tokio + scraper pull in many crates. Use cargo tree to audit and minimize if binary size matters.
Complete Rust Scraper — Async with CSV Output
Complete working scraper with concurrency, error handling, and CSV output.
use reqwest::Client;
use scraper::{Html, Selector};
use serde::Serialize;
use std::time::Duration;
use tokio::time::sleep;
#[derive(Debug, Serialize)]
struct Article {
title: String,
url: String,
date: String,
}
async fn fetch_page(client: &Client, url: &str) -> Result<String, reqwest::Error> {
let resp = client.get(url)
.header("Accept", "text/html")
.send().await?;
resp.text().await
}
fn parse_articles(html: &str) -> Vec<Article> {
let doc = Html::parse_document(html);
let article_sel = Selector::parse("article.post").unwrap();
let title_sel = Selector::parse("h2").unwrap();
let link_sel = Selector::parse("a").unwrap();
let time_sel = Selector::parse("time").unwrap();
doc.select(&article_sel).map(|el| {
Article {
title: el.select(&title_sel).next()
.map(|e| e.text().collect::<String>().trim().to_string())
.unwrap_or_default(),
url: el.select(&link_sel).next()
.and_then(|e| e.value().attr("href"))
.unwrap_or("").to_string(),
date: el.select(&time_sel).next()
.and_then(|e| e.value().attr("datetime"))
.unwrap_or("").to_string(),
}
}).collect()
}
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let client = Client::builder()
.timeout(Duration::from_secs(10))
.user_agent("Mozilla/5.0 (compatible; DataBot/1.0)")
.build()?;
let mut all_articles = vec![];
for page in 1..=10 {
let url = format!("https://example.com/articles?page={}", page);
println!("Scraping page {}", page);
match fetch_page(&client, &url).await {
Ok(html) => {
let articles = parse_articles(&html);
if articles.is_empty() { break; }
all_articles.extend(articles);
}
Err(e) => { eprintln!("Error on page {}: {}", page, e); break; }
}
sleep(Duration::from_millis(500)).await;
}
let mut wtr = csv::Writer::from_path("output.csv")?;
for article in &all_articles {
wtr.serialize(article)?;
}
wtr.flush()?;
println!("Saved {} articles", all_articles.len());
Ok(())
}Or Skip the Complexity — Use AlterLab
AlterLab handles JavaScript rendering, IP rotation, and website compatibility automatically. One POST request returns rendered HTML — no browser management, no proxy configuration. Starts at $0.0002/request with 5,000 free to start.
use reqwest::Client;
use scraper::{Html, Selector};
use serde_json::json;
use std::time::Duration;
const API_KEY: &str = "YOUR_API_KEY"; // Get free at alterlab.io
async fn scrape_with_alterlab(url: &str, render_js: bool) -> Result<String, reqwest::Error> {
let client = Client::builder()
.timeout(Duration::from_secs(30))
.build()?;
let resp = client.post("https://api.alterlab.io/api/v1/scrape")
.header("X-API-Key", API_KEY)
.json(&json!({"url": url, "render_js": render_js}))
.send().await?;
let result: serde_json::Value = resp.json().await?;
Ok(result["html"].as_str().unwrap_or("").to_string())
}
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let html = scrape_with_alterlab("https://example.com/products", true).await?;
let doc = Html::parse_document(&html);
let card_sel = Selector::parse("div.product-card").unwrap();
let title_sel = Selector::parse("h2").unwrap();
let price_sel = Selector::parse(".price").unwrap();
for card in doc.select(&card_sel) {
let title = card.select(&title_sel).next()
.map(|e| e.text().collect::<String>()).unwrap_or_default();
let price = card.select(&price_sel).next()
.map(|e| e.text().collect::<String>()).unwrap_or_default();
println!("{}: {}", title.trim(), price.trim());
}
Ok(())
}Choosing Your Approach
reqwest + scraper
Pros
- +Memory-safe with zero-cost abstractions
- +Excellent performance — compiled native code
- +Fearless concurrency with tokio
Cons
- −No JavaScript execution
- −Steeper learning curve than Python
- −Longer compile times
headless_chrome crate
Pros
- +Full browser — executes any JavaScript
- +Chrome DevTools Protocol from Rust
Cons
- −Requires Chrome binary
- −High memory usage per instance
- −Complex API compared to reqwest
- −Less mature than Playwright/Puppeteer
AlterLab API
Pros
- +Handles static, JavaScript, and challenge pages
- +No browser management
- +Automatic IP rotation
- +5-tier auto-escalation
- +From $0.0002/request
Cons
- −Per-request cost
- −Requires network access
Frequently Asked Questions
More Rust & Scraping Resources
Web Scraping with Go
Go scraping guide: net/http, goquery, and concurrent pipelines with goroutines.
Web Scraping with Python
Complete Python scraping guide: requests, BeautifulSoup, async patterns from beginner to production.
JavaScript Rendering API
Render SPAs and dynamic content with headless Chromium — no browser management.
Pricing
From $0.0002/request. No subscriptions. Balance never expires.
Your first scrape.
Sixty seconds.
$1 free credit — up to 5,000 scrapes. No credit card.
Just a POST request.
No credit card required · $1 free credit, up to 5,000 scrapes · Balance never expires