Java GuideBeginner → Production

Web Scraping with Java — Complete Guide

From HttpClient to production-grade concurrent scrapers. Reliable web data extraction with Java's mature ecosystem.

API Reference

Java's Jsoup library makes HTML parsing as straightforward as using jQuery, while the built-in HttpClient (Java 11+) handles HTTP requests without external dependencies. With virtual threads (Java 21+), concurrent scraping is simpler than ever. This guide covers fetching pages, parsing HTML, building concurrent scrapers, and handling real-world challenges.

Setting Up Your Java Scraping Environment

Add Jsoup to your project for HTML parsing. Java's HttpClient is built into the JDK since Java 11. For Maven projects:

<dependency>
    <groupId>org.jsoup</groupId>
    <artifactId>jsoup</artifactId>
    <version>1.18.1</version>
</dependency>

For Gradle:

implementation 'org.jsoup:jsoup:1.18.1'

Jsoup handles both HTTP fetching and HTML parsing, so for simple scraping you only need this single dependency.

<dependency>
    <groupId>org.jsoup</groupId>
    <artifactId>jsoup</artifactId>
    <version>1.18.1</version>
</dependency>

Fetching a Web Page with Java

Jsoup can fetch and parse in one step with Jsoup.connect(). For more control, use Java's built-in HttpClient. Always set a timeout and user-agent header.

import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;

// Jsoup's built-in HTTP client — simplest approach
Document doc = Jsoup.connect("https://example.com/products")
    .userAgent("Mozilla/5.0 (compatible; DataBot/1.0)")
    .header("Accept", "text/html")
    .timeout(10_000)
    .get();

System.out.printf("Fetched %d bytes%n", doc.html().length());

Parsing HTML with Jsoup

Jsoup provides CSS selector syntax identical to jQuery. Use select() to find elements and text(), attr(), and html() to extract data. Jsoup handles malformed HTML gracefully.

import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;

Elements cards = doc.select("div.product-card");
for (Element card : cards) {
    String title = card.select("h2.product-title").text();
    String price = card.select("span.price").text();
    String href = card.select("a").attr("href");
    System.out.printf("%s: %s (%s)%n", title, price, href);
}

Extracting Structured Data

Define a record (Java 16+) or class for your data model. Collect parsed elements into a list and serialize to JSON using Jackson or Gson, or write to CSV.

import java.io.*;
import java.util.*;

record Product(String title, String price, String url) {}

List<Product> extractProducts(Document doc) {
    return doc.select("div.product-card").stream()
        .map(card -> new Product(
            card.select("h2").text().trim(),
            card.select(".price").text().trim(),
            card.select("a").attr("href")
        ))
        .toList();
}

// Write to CSV
try (var writer = new PrintWriter("products.csv")) {
    writer.println("title,price,url");
    for (Product p : products) {
        writer.printf(""%s","%s","%s"%n", p.title(), p.price(), p.url());
    }
}

Concurrent Scraping with Virtual Threads

Java 21+ virtual threads make concurrent scraping trivial. Create a virtual-thread executor and submit scraping tasks. Virtual threads are lightweight — you can run thousands simultaneously with minimal memory overhead.

import java.util.concurrent.*;

List<String> urls = IntStream.rangeClosed(1, 50)
    .mapToObj(p -> "https://example.com/page/" + p)
    .toList();

List<Product> allProducts = Collections.synchronizedList(new ArrayList<>());

try (var executor = Executors.newVirtualThreadPerTaskExecutor()) {
    var semaphore = new Semaphore(10);
    List<Future<?>> futures = urls.stream().map(url -> executor.submit(() -> {
        try {
            semaphore.acquire();
            Document doc = Jsoup.connect(url)
                .userAgent("Mozilla/5.0 (compatible; DataBot/1.0)")
                .timeout(10_000).get();
            allProducts.addAll(extractProducts(doc));
        } catch (Exception e) {
            System.err.printf("Error scraping %s: %s%n", url, e.getMessage());
        } finally {
            semaphore.release();
        }
    })).toList();

    futures.forEach(f -> { try { f.get(); } catch (Exception ignored) {} });
}

System.out.printf("Collected %d products%n", allProducts.size());

Handling JavaScript-Rendered Pages

Jsoup and HttpClient fetch raw HTML — they do not execute JavaScript. For pages that render content dynamically, you need a rendering solution.

Options: use Selenium WebDriver or Playwright for Java to run a headless browser, or use a cloud rendering API. Running browsers from Java adds significant dependencies and memory usage.

AlterLab's API handles rendering server-side. Send a POST request with render_js: true and receive fully rendered HTML — no browser dependencies required.

Common Pitfalls and How to Avoid Them

The most common issues in Java web scraping:

**1. No timeout** — Jsoup.connect() without timeout can hang indefinitely. Always call .timeout(10_000).

**2. Character encoding** — Jsoup auto-detects encoding from Content-Type and meta tags. Override with .charset() if auto-detection fails.

**3. SSL certificate errors** — Some sites use self-signed certificates. Use .validateTLSCertificates(false) only for testing — never in production.

**4. Memory management** — Parsing very large HTML documents can consume significant heap. Set appropriate -Xmx values and parse only the sections you need.

**5. Rate limiting** — Add Thread.sleep() between requests. Use a Semaphore to limit concurrent connections to the same host.

Complete Java Scraper — Paginated with CSV Output

Complete working scraper with concurrency, error handling, and CSV output.

import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import java.io.PrintWriter;
import java.util.ArrayList;
import java.util.List;

public class ArticleScraper {

    record Article(String title, String url, String date) {}

    static Document fetchPage(String url) throws Exception {
        return Jsoup.connect(url)
            .userAgent("Mozilla/5.0 (compatible; DataBot/1.0)")
            .header("Accept", "text/html")
            .header("Accept-Language", "en-US,en;q=0.9")
            .timeout(10_000)
            .get();
    }

    static List<Article> parseArticles(Document doc) {
        return doc.select("article.post").stream()
            .map(el -> new Article(
                el.select("h2").text().trim(),
                el.select("a").attr("href"),
                el.select("time").attr("datetime")
            ))
            .toList();
    }

    public static void main(String[] args) throws Exception {
        List<Article> allArticles = new ArrayList<>();

        for (int page = 1; page <= 10; page++) {
            String url = "https://example.com/articles?page=" + page;
            System.out.printf("Scraping page %d: %s%n", page, url);
            try {
                Document doc = fetchPage(url);
                List<Article> articles = parseArticles(doc);
                if (articles.isEmpty()) {
                    System.out.println("No more results");
                    break;
                }
                allArticles.addAll(articles);
                Thread.sleep(1000);
            } catch (Exception e) {
                System.err.printf("Error on page %d: %s%n", page, e.getMessage());
                break;
            }
        }

        try (var writer = new PrintWriter("output.csv")) {
            writer.println("title,url,date");
            for (Article a : allArticles) {
                writer.printf(""%s","%s","%s"%n", a.title(), a.url(), a.date());
            }
        }
        System.out.printf("Saved %d articles%n", allArticles.size());
    }
}

Or Skip the Complexity — Use AlterLab

AlterLab handles JavaScript rendering, IP rotation, and website compatibility automatically. One POST request returns rendered HTML — no browser management, no proxy configuration. Starts at $0.0002/request with 5,000 free to start.

import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import java.net.URI;
import java.net.http.*;
import java.net.http.HttpResponse.BodyHandlers;

public class AlterLabScraper {

    static final String API_KEY = "YOUR_API_KEY"; // Get free at alterlab.io

    static String scrapeWithAlterLab(String targetUrl, boolean renderJs) throws Exception {
        HttpClient client = HttpClient.newBuilder()
            .connectTimeout(java.time.Duration.ofSeconds(30))
            .build();

        String json = String.format(
            "{"url": "%s", "render_js": %b}", targetUrl, renderJs
        );

        HttpRequest request = HttpRequest.newBuilder()
            .uri(URI.create("https://api.alterlab.io/api/v1/scrape"))
            .header("X-API-Key", API_KEY)
            .header("Content-Type", "application/json")
            .POST(HttpRequest.BodyPublishers.ofString(json))
            .build();

        HttpResponse<String> response = client.send(request, BodyHandlers.ofString());

        // Extract html field from JSON response
        String body = response.body();
        int htmlStart = body.indexOf(""html":"") + 8;
        int htmlEnd = body.indexOf(""", htmlStart);
        return body.substring(htmlStart, htmlEnd)
            .replace("\\n", "\n").replace("\\t", "\t");
    }

    public static void main(String[] args) throws Exception {
        String html = scrapeWithAlterLab("https://example.com/products", true);

        Document doc = Jsoup.parse(html);
        doc.select("div.product-card").forEach(card -> {
            String title = card.select("h2").text().trim();
            String price = card.select(".price").text().trim();
            System.out.printf("%s: %s%n", title, price);
        });
    }
}
API Docs

Choosing Your Approach

Jsoup

Pros

  • +jQuery-like CSS selectors
  • +Built-in HTTP client — single dependency
  • +Handles malformed HTML well

Cons

  • −No JavaScript execution
  • −Java is verbose for quick scripts
  • −JVM startup overhead

Selenium / Playwright for Java

Pros

  • +Full browser — executes any JavaScript
  • +Multi-browser support
  • +Mature ecosystem

Cons

  • −Heavy — requires browser binaries
  • −Slow — 5-15 seconds per page
  • −Complex WebDriver management
  • −High memory usage

AlterLab API

Pros

  • +Handles static, JavaScript, and challenge pages
  • +No browser management
  • +Automatic IP rotation
  • +5-tier auto-escalation
  • +From $0.0002/request

Cons

  • −Per-request cost
  • −Requires network access

Frequently Asked Questions

More Java & Scraping Resources

Web Scraping with C#

C# scraping guide: HttpClient, HtmlAgilityPack, and AngleSharp patterns.

Web Scraping with Python

Complete Python scraping guide: requests, BeautifulSoup, async patterns from beginner to production.

JavaScript Rendering API

Render SPAs and dynamic content with headless Chromium — no browser management.

Pricing

From $0.0002/request. No subscriptions. Balance never expires.

Your first scrape.
Sixty seconds.

$1 free credit — up to 5,000 scrapes. No credit card.Just a POST request.

terminal
curl -X POST https://api.alterlab.io/v1/scrape \
-H "X-API-Key: YOUR_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com", "formats": ["markdown"]}'

No credit card required · $1 free credit, up to 5,000 scrapes · Balance never expires