Building Resilient Scrapers: Replacing CSS Selectors with LLMs

TL;DR

Replacing brittle CSS selectors with LLM-powered extraction creates resilient scraping pipelines that survive UI changes. By passing simplified DOM content and a strict JSON schema to a model, you extract data based on semantic meaning rather than structural placement. This eliminates maintenance overhead caused by dynamic classes, A/B testing, and frontend redesigns.

The Fragility of Structural Extraction

Data pipelines built on web scraping share a common failure point: the extraction logic. Traditionally, engineers rely on CSS selectors or XPath expressions to target specific DOM nodes. You inspect the page, find the price inside <span class="price-val-392">, write .price-val-392, and deploy.

This works until it does not. Structural extraction is inherently fragile because it couples your data pipeline to a website's presentation layer.

Three factors guarantee your structural extractors will break:

1. Dynamic CSS-in-JS: Modern frontend frameworks (React, Vue) combined with styling solutions like Tailwind or Styled Components generate dynamic, randomized class names during the build process. A class like .css-1yxg23 will change on the next deployment, silently failing your scraper.
2. A/B Testing: E-commerce and travel sites constantly test layout variations. Your scraper might hit the "control" layout on request one, and the "variant" layout on request two. A static CSS selector cannot handle both without complex conditional logic.
3. DOM Restructuring: A site redesign might move the target data from a <div> to a <dl> list. The data remains visible on the page, but the structural path to reach it is entirely different.

When a selector fails, the pipeline returns null data, triggers alerts, and requires an engineer to manually inspect the target site, update the code, and redeploy. This maintenance burden scales linearly with the number of sites you monitor.

Semantic Extraction via LLMs

Large Language Models (LLMs) solve this fragility by decoupling extraction from structure. Instead of telling the system where the data is located, you tell the system what data you want.

LLMs process the textual representation of the page and understand semantic context. They know that "MSRP: $49.99", "Price: $49.99", and "Buy for $49.99" all represent the same data point, regardless of the HTML tags surrounding them.

By defining a target JSON schema, you instruct the LLM to map the unstructured semantic data of the page into a strict, predictable format.

Implementation: Schema-Driven Extraction

To implement semantic extraction, you define a JSON schema representing your required output. You then pass this schema and the target URL to an extraction API. The API handles fetching the page, rendering the JavaScript, simplifying the DOM to fit context windows, and executing the LLM extraction.

Here is how you execute this using the AlterLab Python SDK. We will extract product details from a generic e-commerce page.

import alterlab
import json
import sys

client = alterlab.Client("YOUR_API_KEY")

# Define the exact structure your pipeline expects
product_schema = {
    "type": "object",
    "properties": {
        "product_name": {"type": "string"},
        "price_usd": {"type": "number"},
        "in_stock": {"type": "boolean"},
        "specifications": {
            "type": "array", 
            "items": {"type": "string"}
        }
    },
    "required": ["product_name", "price_usd", "in_stock"]
}

try:
    # The API handles rendering and extraction in one call
    response = client.scrape(
        "https://example-ecommerce.com/products/wireless-headphones",
        extract={"schema": product_schema}
    )
    
    print(json.dumps(response.data, indent=2))

except alterlab.APIError as e:
    print(f"Extraction failed: {e}")
    sys.exit(1)

If you prefer to integrate directly via HTTP, the identical operation via cURL requires sending the schema in the request payload.

curl -X POST https://api.alterlab.io/v1/scrape \
  -H "X-API-Key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example-ecommerce.com/products/wireless-headphones",
    "extract": {
      "schema": {
        "type": "object",
        "properties": {
          "product_name": {"type": "string"},
          "price_usd": {"type": "number"},
          "in_stock": {"type": "boolean"},
          "specifications": {
            "type": "array",
            "items": {"type": "string"}
          }
        },
        "required": ["product_name", "price_usd", "in_stock"]
      }
    }
  }'

In both examples, the extraction logic is immune to DOM changes. If the target site redesigns their product page entirely, changing every CSS class and HTML tag, this code will still return the exact same JSON structure.

DOM Simplification and Context Limits

While LLMs are powerful, they have finite context windows. Feeding raw, unoptimized HTML from a modern web page directly into an LLM often exceeds token limits and introduces massive latency. Raw HTML contains scripts, inline styles, SVGs, and tracking pixels that provide no semantic value to the extraction task.

Before the HTML reaches the LLM, it must undergo aggressive simplification:

1. Tag Stripping: Remove <script>, <style>, <svg>, <canvas>, and <noscript> blocks.
2. Attribute Pruning: Strip formatting attributes (classes, IDs, styles) while retaining semantic attributes (href, src, alt).
3. Whitespace Normalization: Collapse excess whitespace and empty nodes.

AlterLab's Cortex AI handles this pipeline automatically. By condensing the DOM to its purely semantic components, the token count drops by up to 90%, ensuring fast inference and eliminating context truncation errors.

Overcoming Delivery Architecture Challenges

Semantic extraction requires the data to actually be present in the DOM. Many modern single-page applications (SPAs) load an empty HTML shell and populate the content via client-side JavaScript execution. If you pass the initial HTTP response to the LLM, it will extract nothing, because the data does not yet exist.

Furthermore, public data sources frequently deploy anti-automation systems to manage traffic. Standard HTTP clients will receive CAPTCHA challenges or block pages instead of the target content.

Robust pipelines solve this by pairing LLM extraction with a headless browser infrastructure that executes JavaScript and handles network challenges natively. Ensure your extraction layer incorporates proper anti-bot handling to guarantee the LLM receives the fully rendered, intended page state, strictly for the purpose of accessing publicly available information.

Optimizing Pipeline Unit Economics

LLM inference costs more compute than evaluating a regex or CSS selector. Running an LLM on every page view across a multi-million page scrape requires architectural planning to maintain viable pricing unit economics.

To optimize costs, engineers deploy hybrid extraction architectures:

1. Fallback Extraction

Attempt structural extraction (CSS selectors) first. This path is fast and cheap. If the selector returns null or fails validation, route the HTML to the LLM extraction endpoint as a fallback. This ensures high data quality while keeping average per-page costs low.

2. Auto-Healing Selectors

Use LLMs not for primary extraction, but for pipeline maintenance. When a CSS selector fails, pass the DOM to the LLM to find the new structural path for the target data point. The LLM outputs the updated CSS selector, which you programmatically commit to your configuration database. Subsequent requests use the newly healed selector.

3. Targeted Context

Do not send the entire document to the LLM if you only need data from a specific section. Use a broad, stable CSS selector (like #product-container or main) to isolate the relevant HTML block, and pass only that fragment to the LLM schema engine. This drastically reduces token consumption and latency.

Takeaways

1. Schema Over Structure: Define your output format using JSON schemas, and let LLMs map the semantic HTML to your requirements.
2. Eliminate Maintenance: LLM extraction survives CSS class changes, DOM restructuring, and A/B tests, drastically reducing pipeline breakage.
3. Clean the Input: Strip non-semantic HTML tags and attributes before inference to optimize token usage and speed.
4. Render First: Ensure your system executes JavaScript and handles network challenges before passing the DOM to the extraction layer.
5. Optimize Costs: Use hybrid architectures to balance the reliability of AI with the speed and low cost of structural extraction.