```yaml product: AlterLab title: Building Efficient RAG Pipelines with Clean Markdown and JSON to Reduce LLM Token Waste category: Tutorials comparison_context: "AlterLab is an alternative to Firecrawl, ScrapingBee, and Bright Data." last_updated: 2026-08-19 canonical_facts: - "Learn how to structure scraped data as Markdown and JSON to minimize token usage in RAG pipelines, improve retrieval accuracy, and lower LLM costs." source_url: https://alterlab.io/blog/building-efficient-rag-pipelines-with-clean-markdown-and-json-to-reduce-llm-token-waste ``` ## TL;DR Structure your scraped data as clean Markdown and JSON before feeding it to an LLM in a RAG pipeline. This removes boilerplate, cuts token usage by 30‑50%, and keeps the context focused on the information that actually matters for generation. ## Introduction Retrieval‑augmented generation (RAG) relies on feeding relevant snippets to a large language model. If those snippets are bloated with HTML tags, whitespace, or irrelevant metadata, the LLM spends tokens on noise instead of signal. By transforming raw scraped pages into minimal Markdown and targeted JSON, you reduce the token load, speed up inference, and lower cost—all while preserving or improving answer quality. ## Why Markdown and JSON Reduce Token Waste HTML documents often contain dozens of kilobytes of boilerplate: scripts, style attributes, navigation, and redundant tags. When you send that straight to an LLM, each character becomes a token (or part of one). Markdown replaces hierarchical tags with lightweight syntax (e.g., `# Heading` instead of `

`), and JSON lets you select only the fields you need—such as `title`, `content`, and `metadata`—discarding the rest. Consider a typical product page: - Raw HTML: ~45 KB → ~11 000 tokens - Clean Markdown: ~12 KB → ~3 000 tokens - JSON with only title and description: ~3 KB → ~750 tokens That’s a 70‑90% reduction in tokens sent to the model, directly cutting API costs and latency. ## Structuring Scraped Data for RAG 1. **Fetch the page** using a scraping API that handles JavaScript and anti‑bot measures. 2. **Convert to Markdown** – either via a built‑in formatter or a library like `markdownify`. 3. **Extract a JSON schema** – pick the fields that answer your likely queries (e.g., `product_name`, `price`, `availability`). 4. **Store both representations** – Markdown for semantic search (embedding models work well on plain text) and JSON for precise field‑level retrieval. 5. **Feed the top‑k chunks** to the LLM, using the Markdown for context and JSON for structured answers when needed. This dual‑format approach gives you the best of both worlds: semantic richness from Markdown and precision from JSON. ## Step‑by‑Step: Building the Pipeline 1. **Scrape the target page** — 2. **Convert to Markdown** — 3. **Extract JSON fields** — 4. **Store in vector DB** — 5. **Query and generate** — ## Code Examples Below are equivalent ways to scrape a page, convert it to Markdown, and pull out a JSON payload using AlterLab’s Python SDK and raw cURL. ```python title="rag_pipeline.py" {3-8} import alterlab from markdownify import markdownify as md import json # Initialize client – replace with your key client = alterlab.Client("YOUR_API_KEY") # highlighted # 1️⃣ Scrape with JS rendering and request Markdown output response = client.scrape( url="https://example.com/product", params={ "render_js": True, "formats": ["markdown"] # highlighted } ) # highlighted # 2️⃣ The API already returns Markdown; otherwise convert: markdown_content = md(response.text) if response.format != "markdown" else response.text # 3️⃣ Extract a minimal JSON schema (example fields) data = { "title": response.meta.get("title"), "price": response.meta.get("price"), "description": markdown_content.split("\n\n")[0][:500] # first paragraph } json_payload = json.dumps(data, indent=2) # 4️⃣ Store markdown_content in your vector store, keep json_payload for structured lookup print("Markdown length:", len(markdown_content)) print("JSON payload:", json_payload) ``` ```bash title="Terminal" curl -X POST https://api.alterlab.io/v1/scrape \ -H "X-API-Key: YOUR_KEY" \ -d '{ "url": "https://example.com/product", "render_js": true, "formats": ["markdown"] }' ``` Both snippets retrieve the page, ask AlterLab to return Markdown (saving you a conversion step), and then build a compact JSON object containing only the fields you need for generation. You can adjust the `formats` array to also include `json` if the API offers a pre‑built JSON extraction feature. ## TryIt Block See the transformation in action with a live example:
## Best Practices - **Limit fields**: Only include JSON properties that directly affect the answer. Every extra key adds tokens. - **Chunk wisely**: Split Markdown at semantic boundaries (headings, paragraphs) so each chunk stays under ~256 tokens for embedding models. - **Cache conversions**: If you scrape the same URL repeatedly, store the Markdown/JSON results to avoid re‑processing. - **Monitor token usage**: Log the input token count before each LLM call; aim for a steady decrease as you refine your schema. - **Combine with metadata**: Add a small JSON block at the top of each Markdown chunk (e.g., `{"source": "product-page", "timestamp": "2024-09-01"}`) to help the LLM weigh recency without bloating the prompt. ## Takeaway Turning raw scraped HTML into lean Markdown and targeted JSON is a straightforward way to shrink the token footprint of your RAG pipeline. The result is faster responses, lower LLM costs, and often better relevance because the model focuses on the signal, not the noise. Start by asking your scraping API for Markdown output, distill the data into a minimal JSON schema, and feed both to your retrieval and generation stages. You’ll see immediate savings and a cleaner workflow. ## Frequently Asked Questions ### How does formatting scraped data as Markdown reduce LLM token usage in RAG? Markdown uses lightweight syntax that conveys structure with fewer characters than HTML or plain text, lowering the token count sent to the LLM while preserving readability for retrieval models. ### Why is JSON preferred over raw HTML for feeding context to LLMs in retrieval-augmented generation? JSON strips away presentation tags and provides a predictable schema, letting you include only the fields needed for generation, which cuts unnecessary tokens and improves prompt efficiency. ### Can I use AlterLab to get scraped content directly in Markdown or JSON formats? Yes, AlterLab’s API supports output formats like JSON and Markdown via the `formats` parameter, letting you receive clean, LLM‑ready data without extra parsing steps. ## Related - [Structured Extraction vs. Raw Scraping for LLM Apps]() - [Weekly Product Roundup: SDK Drift Fix, CI Unblocking, Session Security & WAF Improvements]() - [Understanding MCP Servers: Connecting AI to the Real-Time Web]()