Building Scalable RAG Pipelines with Markdown Extraction

TL;DR

Extracting Markdown from web pages before feeding them to an LLM cuts token waste, lowers cost, and improves retrieval quality in RAG pipelines. Use AlterLab’s formats=["markdown"] option to get clean, structured output ready for embedding or direct LLM consumption.

Why Markdown Helps RAG Pipelines

Large language models process text token by token. When you feed raw HTML, a significant portion of tokens are consumed by tags, attributes, JavaScript, and CSS that carry no semantic meaning for the model. This inflates costs, slows inference, and can dilute the signal‑to‑noise ratio in retrieval steps.

Markdown strips away presentation while preserving semantic structure:

• Headings become # syntax`) retain hierarchy.
• Lists and code blocks stay intact.
• Boilerplate like navigation menus, footers, and ads are removed.

The result is a denser representation of the same information, often reducing token count by 40‑60% on typical article pages.

How AlterLab Delivers Markdown Extraction

AlterLab’s scraping API handles anti‑bot challenges, rotates proxies, and renders JavaScript automatically. Adding formats=["markdown"] to the request instructs the pipeline to run an HTML‑to‑Markdown conversion after retrieval and return the cleaned content.

You can retrieve Markdown via the Python SDK or a raw HTTP call. Both approaches are shown below.

Python SDK Example

import alterlab

client = alterlab.Client("YOUR_API_KEY")   # highlighted
response = client.scrape(
    url="https://example.com/article",
    formats=["markdown"]                  # highlighted
)                                         # highlighted
print(response.text)                      # highlighted

The formats parameter tells AlterLab to return Markdown instead of raw HTML. The SDK handles authentication, retries, and error handling.

cURL Example

curl -X POST https://api.alterlab.io/v1/scrape \
  -H "X-API-Key: YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com/article","formats":["markdown"]}'  # highlighted

The JSON payload includes the formats array. The response body is the Markdown string, ready for chunking and embedding.

Infographic: Markdown Extraction Flow

Cost Savings in Practice

Consider a pipeline that ingests 10,000 article pages per day. Average raw HTML size is 120 KB (~30 k tokens). After Markdown extraction, average size drops to 45 KB (~11 k tokens).

Assuming a language model costs $0.00001 per token:

• Raw HTML: 10,000 × 30,000 × $0.00001 = $3,000/day
• Markdown: 10,000 × 11,000 × $0.00001 = $1,100/day

That’s a 63% reduction in LLM API spend, with no loss of meaningful content.

Integrating Markdown into Your RAG Workflow

1. Scrape – Use AlterLab to get Markdown for each source URL.
2. Chunk – Split the Markdown on headings or fixed token windows (e.g., 500‑token chunks).
3. Embed – Generate vector embeddings for each chunk with your preferred embedding model.
4. Store – Save embeddings in a vector database (FAISS, Pinecone, Weaviate, etc.).
5. Retrieve – At query time, fetch top‑k similar chunks.
6. Generate – Feed the retrieved Markdown snippets to the LLM as context.

Because the Markdown already excludes navigation and ads, the retrieved chunks are more likely to contain the exact information the user needs, improving answer relevance.

Try It Yourself

Internal Resources

• For a full walkthrough of the Python SDK, see the Python scraping API.
• Review the API docs for all supported output formats.
• Check the pricing page to estimate cost savings from reduced token usage.

Takeaway

Markdown extraction is a simple, high‑impact optimization for any RAG pipeline that relies on web‑sourced content. By requesting Markdown directly from AlterLab, you cut LLM token waste, lower expenses, and preserve the semantic structure that models need for accurate retrieval and generation. Implement the two‑line formats=["markdown"] change today and measure the reduction in your next billing cycle.