Cleaning Scraped Pages into Markdown and Chunking Them for RAG
Short answer
How do you clean scraped web pages and chunk them for a RAG pipeline?
Convert each page to markdown, strip navigation, footers and cookie banners, then split along headings into chunks of roughly 300-800 tokens with a small overlap. Store the URL, title and heading path with every chunk so answers can cite sources. AlterLab can return main-content markdown directly, which removes most of the cleaning step.
Step by step
- 1
Fetch as markdown
Request markdown rather than HTML so headings, lists and links survive while scripts and styling are dropped.
- 2
Remove boilerplate
Drop repeated navigation, footer and consent text. Lines that appear on most pages of a site are usually boilerplate.
- 3
Split on headings
Chunk at heading boundaries first, then by size, so each chunk covers one topic.
- 4
Attach metadata
Keep URL, page title, heading path and fetch date with every chunk for citations and refreshes.
Example
import re
def chunk_markdown(md: str, url: str, max_chars: int = 3000) -> list[dict]:
sections = re.split(r"(?m)^(?=#{1,3} )", md)
chunks = []
for section in sections:
heading = section.splitlines()[0] if section.strip() else ""
for i in range(0, len(section), max_chars):
chunks.append({
"url": url,
"heading": heading.lstrip("# ").strip(),
"text": section[i:i + max_chars].strip(),
})
return [c for c in chunks if c["text"]]Frequently Asked Questions
What chunk size works best?
Around 300-800 tokens suits most embedding models. Smaller chunks retrieve more precisely, larger ones keep more context. Test with your own questions.
How do I keep the index fresh?
Store a content hash per page, re-fetch on a schedule and re-embed only pages whose hash changed.
Read the full guide
Web data for AI agents and RAGTry it with AlterLab
Send a URL, get the page back as HTML, markdown or JSON. Rendering, proxy rotation and retries are handled for you.
Your first scrape.
Sixty seconds.
Up to 5,000 free requests — about 5,000 Light, 500 Protected or 250 Dynamic requests. No credit card.
Just a POST request.
No credit card required · Up to 5,000 free requests (about 5,000 Light, 500 Protected or 250 Dynamic requests) · Balance never expires