Optimizing RAG Pipelines: Using Markdown and Structured JSON to Reduce LLM Token Waste

TL;DR

Using Markdown or structured JSON as the output format from your web scraper cuts LLM token consumption in RAG pipelines by 30‑50%. This reduces cost, speeds up retrieval, and improves answer quality by feeding the model cleaner, more relevant data.

Why Token Waste Happens in RAG

Retrieval‑Augmented Generation pipelines typically fetch raw HTML from target pages, then pass that HTML to an LLM for summarization or question answering. HTML contains tags, attributes, whitespace, and boilerplate navigation that add tokens without contributing semantic value. For a typical article, HTML can be 2‑3× larger than its Markdown equivalent. When you multiply that across dozens or hundreds of retrieved documents per query, the token bill grows quickly.

The Solution: Clean Structured Output

Instead of scraping raw HTML, request the data in a format that already strips noise:

• Markdown preserves headings, lists, and emphasis while dropping tags.
• Structured JSON lets you select only the fields you need (title, body, price, etc.) and discard the rest.

Both formats reduce the token count sent to the LLM, which directly lowers cost (most LLM providers charge per token) and latency (fewer tokens to process).

How AlterLab Delivers Clean Formats

AlterLab’s scraping API includes a formats parameter that tells the engine to post‑process the retrieved page 1/45. You can ask for Markdown, JSON, or plain text. The service handles anti‑bot challenges, JavaScript rendering, and proxy rotation behind the scenes, so you receive clean data without worrying about getting blocked.

Check out the Python SDK for a batteries‑included client that makes this call trivial.

import alterlab

client = alterlab.Client("YOUR_API_KEY")
# Request Markdown output; the API strips HTML and returns clean text
response = client.scrape(
    url="https://example.com/articles/tech-trends",
    formats=["markdown"]   # highlighted
)
print(response.text[:500])  # first 500 chars of Markdown

curl -X POST https://api.alterlab.io/v1/scrape \
  -H "X-API-Key: YOUR_KEY" \
  -d '{
    "url": "https://example.com/articles/tech-trends",
    "formats": ["json"]
  }' | jq '.text'   # highlighted

Infographic: Token Savings by Format

The table above shows typical size reductions for a news article. Translating HTML (1) -> 68) -> JSON 24). This means you send to the LLM token usage follows a similar pattern because tokenizers roughly count characters/words.

Building a Token‑Efficient RAG Pipeline

1. Scrape with format selection – Use Markdown for full‑text retrieval or JSON when you only need specific fields.
2. Chunk the cleaned text – Split Markdown on headings or JSON objects into passages of ~200‑300 tokens.
3. Embed and store – Generate embeddings on the cleaned passages; store them in your vector database.
4. Retrieve and rerank – At query time, fetch the top‑k passages; because they are already clean, the reranker works with higher signal‑to‑noise.
5. Generate answer – Pass the retrieved passages to the LLM. With fewer input tokens, the model focuses on relevant content, improving accuracy and cutting cost.

Code Example: Building the Index

from alterlab import Client
import json
import faiss
import numpy as np
from sentence_transformers import SentenceTransformer

client = Client("YOUR_API_KEY")
model = SentenceTransformer("all-MiniLM-L6-v2")

def scrape_and_chunk(url):
    resp = client.scrape(url, formats=["markdown"])
    text = resp.text
    # simple chunking by double newline (paragraphs)
    chunks = [p.strip() for p in text.split("\n\n") if len(p) > 50]
    return chunks

urls = [
    "https://example.com/articles/ai-trends",
    "https://example.com/articles/ml-frameworks",
    # add more as needed
]

all_chunks = []
for u in urls:
    all_chunks.extend(scrape_and_chunk(u))

embeddings = model.encode(all_chunks, show_progress_bar=True)
index = faiss.IndexFlatL2(embeddings.shape[1])
index.add(np.array(embeddings))

# store chunks and index for later use
with open("chunks.json", "w") as f:
    json.dump(all_chunks, f)
faiss.write_index(index, "vectors.index")

Code Example: Querying the Pipeline

from alterlab import Client
import json
import faiss
import numpy as np
from sentence_transformers import SentenceTransformer

client = Client("YOUR_API_KEY")
model = SentenceTransformer("all-MiniLM-L6-v2")
index = faiss.read_index("vectors.index")
with open("chunks.json") as f:
    chunks = json.load(f)

def retrieve(query, k=3):
    q_emb = model.encode([query])
    D, I = index.search(np.array(q_emb), k)
    return [chunks[i] for i in I[0]]

def generate_answer(query):
    passages = retrieve(query)
    context = "\n\n".join(passages)
    prompt = f"Answer the question using only the following context:\n\n{context}\n\nQuestion: {query}\nAnswer:"
    # call your LLM provider (OpenAI, Anthropic, etc.)
    # For illustration, we show a placeholder
    response = alterlab.llm.complete(
        model="gpt-4o-mini",
        prompt=prompt,
        max_tokens=250,
        temperature=0.2
    )
    return response.text

print(generate_answer("What are the main benefits of using Markdown in RAG?"))

Balancing Completeness and Brevity

While Markdown and JSON strip noise, ensure you do not remove information critical to your use case. For example:

• If you need exact HTML attribute values (e.g., data-price), keep those fields in a JSON schema.
• If visual layout matters (e.g., multi‑column tables), consider retaining a minimal HTML snippet alongside the Markdown body.

AlterLab lets you request multiple formats in one call:

response = client.scrape(
    url="https://example.com/product-page",
    formats=["markdown", "json", "raw_html"]   # get all three
)
markdown = response.formats["markdown"]
json_data = response.formats["json"]
html = response.formats["raw_html"]

Cost Impact Estimate

Assume a RAG system processes 10,000 queries per day, each retrieving 5 passages of 800 tokens when using raw HTML. That’s 40 M tokens/day. Switching to Markdown reduces average passage size to 400 tokens, halving the token volume to 20 M tokens/day. At $0.01 per 1k tokens (a typical LLM rate), daily cost drops from $400 to $200—a 50% saving.

Best Practices

• Select fields deliberately – When using JSON, request only the fields you will actually embed.
• Validate output – After scraping, run a quick sanity check (e.g., ensure Markdown headings are present) before indexing.
• Cache cleaned data – Store the Markdown/JSON versions; you rarely need to re‑scrape the same page for the same format.
• Monitor token usage – Log input token counts per query to verify savings over time.

Final Takeaway

Feeding your RAG pipeline clean, structured data from the scraper stage is a straightforward way to cut LLM token consumption, lower expenses, and improve response quality. By leveraging AlterLab’s built‑in format conversion, you get anti‑bot handling, JavaScript rendering, and clean output in a single API call—no extra parsing pipelines required. Start small: switch one source to Markdown, measure the token drop, then roll the change across your ingestion workflow.

Learn more about AlterLab’s pricing to see how reduced token usage translates into lower overall scraping and LLM costs.