Building Scalable RAG Pipelines: Reducing LLM Token Waste with Markdown Extraction and Structured JSON

TL;DR

Extract web pages as markdown and structured JSON to cut LLM token usage in RAG pipelines. This approach removes unnecessary HTML, preserves semantic structure, and yields predictable fields for faster, cheaper retrieval‑augmented generation.

Why Token Waste Matters in RAG

Retrieval‑augmented generation pipelines feed large language models with context pulled from external sources. If that context often that context arrives as raw HTML, the model must process tags, scripts, and styling that add no semantic value. Each extra token raises latency and cost, especially when processing thousands of documents per day.

By converting pages to markdown, we keep headings, lists, code blocks, and emphasis while dropping tags like <div>, <script>, and inline styles. Structured JSON then isolates discrete data points (product name, price, availability) into named fields. The LLM receives less noise and more signal, which directly reduces the token count needed for each prompt.

How AlterLab Delivers Markdown and JSON

AlterLab’s scraping API can return page content in multiple formats via the formats parameter. Setting formats=['markdown'] yields a cleaned markdown version. Setting formats=['json'] returns a JSON object with fields such as text, title, and metadata. Both modes use the same anti‑bot handling and rendering pipeline, so you get reliable extraction without managing headless browsers yourself.

Check out the Python SDK for a batteries‑included client that handles authentication, retries, and response parsing.

Step‑by‑Step Pipeline

Code Example: Python SDK

import alterlab
from sentence_transformers import SentenceTransformer
import numpy as np

client = alterlab.Client("YOUR_API_KEY")  # Initialize with your key

def fetch_and_prepare(url: str):
    # Request both markdown and JSON formats in one call
    resp = client.scrape(
        url,
        formats=["markdown", "json"],
        # Optional: set min_tier to skip unnecessary rendering steps
        min_tier=2
    )
    markdown_text = resp.markdown   # Clean markdown version
    structured = resp.json          # Dict with title, metadata, etc.
    return markdown_text, structured

def chunk_markdown(text: str, max_tokens: int = 200):
    # Simple splitter on headings; replace with tiktoken‑aware split for prod
    lines = text.split("\n")
    chunks = []
    current = []
    for line in lines:
        if line.startswith("#") and current:
            chunks.append("\n".join(current))
            current = [line]
        else:
            current.append(line)
    if current:
        chunks.append("\n".join(current))
    return chunks

# Example usage
url = "https://example.com/product-list"
md, meta = fetch_and_prepare(url)
chunks = chunk_markdown(md)
print(f"Extracted {len(chunks)} chunks from {meta.get('title','unknown')}")

Code Example: cURL

curl -X POST https://api.alterlab.io/v1/scrape \
  -H "X-API-Key: YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/product-list",
    "formats": ["markdown","json"],
    "min_tier": 2
  }' | jq '.'

The response includes markdown and json fields ready for the pipeline steps above.

Try It Yourself

Token‑Saving Techniques

1. Head‑based chunking – Split markdown at heading levels (#, ##) so each chunk starts with a clear topic. This improves retrieval relevance and reduces redundant context.
2. Token counting – Use a tokenizer (e.g., tiktoken for GPT models) to enforce a maximum token window per chunk before embedding.
3. Metadata enrichment – Pass fields from the JSON output (like price or availability) as separate filterable attributes in your vector database. The LLM can then receive a compact prompt: “Given price $49.99 and description …”.
4. Cache frequent pages – Store the markdown+JSON blob for URLs that change infrequently. Subsequent requests hit your cache, saving both API calls and rendering time.
5. Dynamic min_tier – For sites that need JavaScript, start with min_tier=1 and increase only if the returned text length is below a threshold. This avoids unnecessary headless browser usage.

Comparison: Raw HTML vs Markdown vs JSON

Real‑World Impact

A team processing 10 000 product pages daily saw:

• API cost drop from $120 to $55 per day (≈55% reduction)
• Average latency per query fall from 1.4 s to 0.9 s