Building LLM-Ready Data Pipelines: From Raw HTML to Structured Records

TL;DR

To build an LLM-ready data pipeline, retrieve pages reliably with anti-bot handling, strip boilerplate, use AI extraction to produce structured JSON, and validate the output against a schema. This yields clean records that language models can consume with minimal preprocessing.

Introduction

Large language models excel at reasoning over structured data, but feeding them raw HTML wastes tokens on markup, scripts, and styling. A robust pipeline converts the noisy HTML of a public page into a concise, typed record—ideal for retrieval‑augmented generation, fine‑tuning, or agent workflows. This guide walks through each stage, emphasizing reliability and ethical collection from publicly accessible content.

Why Raw HTML Isn't Enough for LLMs

HTML documents average 60‑80 % boilerplate: navigation, ads, analytics scripts, and CSS. When passed directly to an LLM, this inflates context size, raises cost, and introduces irrelevant patterns that can distract the model. Structured extraction isolates the salient facts—product specs, article bodies, event details—into a predictable JSON shape that matches downstream schemas.

Step 1: Reliable Retrieval

The foundation is fetching the page without interruptions. AlterLab’s API combines rotating residential proxies, automatic header rotation, and smart rendering to handle JavaScript‑heavy sites and common bot challenges. The request specifies a URL and optional parameters like min_tier to skip unnecessary tiers for faster response.

curl -X POST https://api.alterlab.io/v1/scrape \
  -H "X-API-Key: YOUR_API_KEY" \
  -d '{
    "url": "https://example.com/listings",
    "min_tier": 3,
    "formats": ["html"]
  }'

The response delivers the fully rendered HTML, ready for the next stage. For high‑volume pipelines, reuse the same API key and respect rate limits by implementing exponential backoff on 429 responses.

Step 2: Cleaning and Normalization

Raw HTML still contains scripts, style tags, and comments that add noise. A lightweight cleanup using an HTML parser (e.g., BeautifulSoup or lxml) removes non‑content elements and extracts the main text container.

from bs4 import BeautifulSoup
import re

def clean_html(raw: str) -> str:
    soup = BeautifulSoup(raw, "lxml")
    # Remove boilerplate tags
    for tag in soup(["script", "style", "noscript", "iframe", "svg"]):
        tag.decompose()
    # Optional: keep only main article/product container
    main = soup.find("article") or soup.find("main") or soup.body
    text = main.get_text(separator=" ", strip=True) if main else soup.get_text(separator=" ", strip=True)
    # Collapse whitespace
    return re.sub(r"\s+", " ", text).strip()

The cleaned text reduces size by 50‑70 % while preserving the informational payload. For pages with consistent structure, you can target specific CSS selectors (e.g., .product-description) to further focus extraction.

Step 3: Structured Extraction with AI

Instead of writing fragile XPath or CSS rules, leverage an LLM‑based extractor that understands context. AlterLab’s Cortex AI option takes the cleaned HTML (or raw HTML) and returns JSON matching a user‑defined schema. Provide a simple schema description via the prompt or rely on zero‑shot extraction for common patterns.

import alterlab
import json

client = alterlab.Client("YOUR_API_KEY")

response = client.scrape(
    url="https://example.com/listings",
    formats=["json"],
    ai_extract=True,          # enables Cortex AI
    ai_prompt="Extract each listing as an object with fields: title (str), price (float), availability (bool), url (str).",
)
data = json.loads(response.text)  # already structured JSON
print(json.dumps(data, indent=2))

The AI handles variations in layout, language, and missing fields, emitting null for absent values. This step converts the textual blob into a typed record ready for validation.

Step 4: Validation and Enrichment

Even AI extractors can hallucinate or omit required fields. Validate the output against a JSON Schema or Pydantic model to catch inconsistencies early. Enrichment—such as converting price strings to numbers, normalizing dates, or adding a content hash—ensures downstream consumers receive predictable data.

from pydantic import BaseModel, Field, ValidationError
from typing import List, Optional

class Listing(BaseModel):
    title: str
    price: Optional[float] = Field(None, ge=0)
    availability: Optional[bool] = None
    url: str = Field(..., regex=r"^https?://")

def validate_listings(raw: List[dict]) -> List[Listing]:
    validated = []
    for item in raw:
        try:
            validated.append(Listing(**item))
        except ValidationError as e:
            # Log or send to dead‑letter queue for manual review
            print(f"Invalid item {item.get('url')}: {e}")
    return validated

Validated records can be written to a vector store, posted to a webhook, or batched for LLM fine‑tuning.

Putting It All Together: Example Pipeline

Combining the stages yields a reproducible workflow:

1. Fetch – Call AlterLab API with appropriate tier and format.
2. Clean – Strip scripts, extract main content.
3. Extract – Use Cortex AI to produce structured JSON.
4. Validate – Apply schema checks and enrichment.
5. Store – Persist to your data lake or trigger downstream jobs.

The following script demonstrates a minimal end‑to‑end run for a single URL:

import alterlab
import json
from bs4 import BeautifulSoup
import re
from pydantic import BaseModel, Field, ValidationError
from typing import Optional

class Product(BaseModel):
    name: str
    price: Optional[float]
    in_stock: Optional[bool]
    url: str

def fetch(url: str) -> str:
    client = alterlab.Client("YOUR_API_KEY")
    resp = client.scrape(
        url=url,
        formats=["html"],
        min_tier=3,
    )
    return resp.text

def clean(raw_html: str) -> str:
    soup = BeautifulSoup(raw_html, "lxml")
    for tag in soup(["script", "style", "noscript", "iframe"]):
        tag.decompose()
    main = soup.find("article") or soup.find("main") or soup.body
    text = main.get_text(separator=" ", strip=True) if main else soup.get_text(separator=" ", strip=True)
    return re.sub(r"\s+", " ", text).strip()

def extract_ai(clean_text: str) -> dict:
    client = alterlab.Client("YOUR_API_KEY")
    resp = client.scrape(
        url="data:text/plain," + clean_text,  # faux URL to trigger AI extraction on provided text
        formats=["json"],
        ai_extract=True,
        ai_prompt="Return JSON with keys: name (str), price (float|null), in_stock (bool|null), url (str).",
    )
    return json.loads(resp.text)

def validate(data: dict) -> Product:
    return Product(**data)

if __name__ == "__main__":
    url = "https://example.com/product/123"
    html = fetch(url)
    text = clean(html)
    ai_json = extract_ai(text)
    product = validate(ai_json)
    print(product.json())

This pattern scales: wrap the loop in a worker pool, persist checkpoints, and monitor success rates via the API’s built‑in metrics.

Best Practices and Pitfalls

• Respect usage policies – Only scrape pages that are publicly accessible and permitted by the site’s terms; avoid login‑gated or paywalled content unless you have explicit authorization.
• Handle pagination gracefully – Use API parameters like next_page tokens or infer patterns from URLs; avoid aggressive parallel requests that could trigger rate limits.
• Cache cleaned HTML – If the same page is scraped repeatedly, store the cleaned text to reduce AI extraction calls and cost.
• Monitor AI extraction quality – Sample a fraction of outputs weekly; drift in site design may require prompt tweaks or a fallback to rule‑based selectors.
• Use idempotent writes – Design your storage layer to deduplicate records based on a stable hash (e.g., SHA‑256 of the normalized JSON) to prevent duplicate entries from retries.

Takeaway

A production‑grade LLM‑ready pipeline separates retrieval from transformation: fetch reliably with proxy and rendering layers, strip noise, employ AI extraction for schema‑flexible conversion, and validate rigorously. The result is clean, token‑efficient structured data that powers accurate language‑model applications without brittle scrapers or excessive preprocessing.

Check out the Python SDK for a batteries‑included client, and review the API docs for details on ai_extract and rendering tiers.

Sign up to obtain an API key and begin building your own pipelines.