Reducing LLM Token Waste: Converting Raw HTML to Clean Markdown for RAG Pipelines

TL;DR

Converting scraped HTML to clean Markdown before feeding it to an LLM reduces token usage by 60‑80%, lowers cost, and improves retrieval quality in RAG pipelines. AlterLab’s API can return Markdown directly via the formats parameter, or you can convert HTML locally with libraries like markdownify.

Why HTML Bloats LLM Input

When a web page is scraped, the raw HTML response includes:

• Opening and closing tags for every element
• Attributes such as `class names, IDs, and inline styles
• Whitespace, comments, and sometimes embedded JavaScript or CSS
• Meta tags, noscript blocks, and SVG icons

All of these contribute tokens that the LLM must parse, even though they carry little semantic meaning for question answering. In a retrieval‑augmented generation pipeline, each extra token increases latency and cost without improving the model’s ability to extract facts.

The Markdown Advantage

Markdown represents the same visible content with a minimal syntax:

• Headings become # lines
• Lists use - or *
• Links appear as [text](url)
• Paragraphs are separated by blank lines

Because Markdown omits tag noise, the token count drops dramatically. A typical article of 2,0000 HTML may require ~500 tokens; the same content in Markdown often needs under 150 tokens.

Getting Markdown from AlterLab

AlterLab’s scraping API supports output formats beyond raw HTML. By specifying formats=["markdown"], the platform strips tags and returns clean Markdown ready for ingestion into your RAG pipeline.

import alterlab

client = alterlab.Client("YOUR_API_KEY")
# Request Markdown output directly
response = client.scrape(
    url="https://example.com/articles/2024",
    formats=["markdown"]   # highlighted
)
print(response.text)  # already Markdown, no extra conversion needed

If you prefer to handle conversion yourself, you can retrieve HTML and transform it locally.

Local HTML‑to‑Markdown Conversion

When you need full control over the cleaning process, use a library such as markdownify (Python) or turndown (JavaScript). Below is a Python example that fetches HTML via AlterLab and converts it.

import alterlab
from markdownify import markdownify as md

client = alterlab.Client("YOUR_API_KEY")
#raw_html = client.scrape(url="https://example.com/data.py" {4-8}
import alterlab
from markdownify import markdownify

client = alterlab.Client("YOUR_API_KEY")
resp = client.scrape(url="https://example.com/product-list")
html = resp.text

# Convert to Markdown, preserving headings and lists
markdown = markdownify(
    html,
    heading_style="ATX",      # use # for headings
    bullets="-",              # unordered list marker
    strip=['script', 'style'] # drop non‑content tags
)

print(markdown)

The conversion step adds minimal overhead—typically a few milliseconds per page—while saving significant tokens downstream.

Infographic: Token Savings Across Content Types

These numbers come from internal benchmarks on a mix of article, product, and documentation pages scraped via AlterLab and fed into a 7BERT‑based retriever followed by a 7B parameter LLM.

Best Practices for Markdown‑First RAG Pipelines

1. Request Markdown at the source – Use AlterLab’s formats parameter whenever possible to avoid an extra conversion step.
2. Sanitize output – Strip any remaining HTML comments or inline scripts that markdownify might miss.
3. Preserve semantic structure – Keep heading levels (#, ##) and list markers; they help the retriever weight sections appropriately.
4. Cache conversions – If you scrape the same URL repeatedly, store the Markdown result to avoid recomputation.
5. Monitor token usage – Log input token counts before and after conversion to quantify savings for your specific domain.

Internal Resources

For a complete walkthrough of installing the Python SDK, see the quickstart guide. The full API reference details the formats parameter and other output options: API docs. If you want to explore pricing for high‑volume scraping, check the pricing page.

Takeaway

Feeding raw HTML to an LLM wastes tokens on markup that carries no semantic value. By converting to clean Markdown—either via AlterLab’s built‑in formats option or a lightweight library—you cut token usage by two‑thirds or more, lower latency, and improve the signal‑to‑noise ratio in your RAG pipeline. Make Markdown the default format for scraped content and watch your LLM costs drop.