Building Scalable RAG Pipelines with Real-Time Web Data via MCP Servers

TL;DR

Combine Model Context Protocol (MCP) servers with agentic browsing and a reliable scraping API to fetch fresh web data for RAG pipelines. This approach decouples data retrieval from model inference, scales horizontally, and handles anti‑bot measures without custom browser management.

Introduction

Retrieval‑augmented generation (RAG) improves LLM outputs by grounding them in external knowledge. When that knowledge lives on the public web, pipelines must fetch, parse, and embed pages on demand. Doing this at scale introduces three challenges: handling bot defenses, keeping latency low, and maintaining data freshness. This post shows how MCP servers expose scraping as a callable tool, how agentic browsing drives the retrieval logic, and how AlterLab’s API provides the anti‑bot‑protected fetch layer.

What is MCP and Why It Matters for RAG

MCP defines a lightweight JSON‑RPC‑style protocol where a server advertises tools (functions) and resources (data). An LLM‑agent can invoke a tool by name, passing arguments, and receive a structured response. For RAG, the tool is typically “scrape_url” which returns cleaned HTML, text, or JSON. Benefits include:

• Loose coupling: The LLM does not need to know HTTP details; it calls a named tool.
• Reusability: Multiple agents or workflows share the same MCP endpoint.
• Observability: Tool calls are logged, making cost and latency tracking straightforward.

Agentic Browsing Basics

Agentic browsing replaces static CSS‑selector scripts with an LLM that decides what to click, fill, or wait for. The agent receives a goal (e.g., “extract the latest price from the product listing”) and interacts with a headless browser via a DOM‑access API. Key properties:

• Adaptability: Handles pagination, infinite scroll, and minor UI changes without code updates.
• Goal‑driven: The agent can back‑off, retry, or switch strategies if an element is missing.
• Auditability: Each action (click, type, wait) can be logged for debugging.

When paired with an MCP server, the agent’s “scrape” action becomes a remote tool call, keeping the browsing logic inside the agent while the actual network request lives in a trusted, scalable service.

Architecture Overview

+----------------+       +------------------+       +---------------------+
| LLM Agent      | --->  | MCP Server       | --->  | AlterLab API        |
| (Agentic)      |       | (scrape_url tool)|       | (anti‑bot fetch)    |
+----------------+       +------------------+       +---------------------+
        ^                         |                         |
        |                         v                         v
   User Request          +----------------+          +-----------------+
                         | Headless       |          | Proxy Pool,    |
                         | Browser (Playwright) |   | CAPTCHA Solver |
                         +----------------+          +-----------------+

1. The agent receives a query and decides it needs fresh web data.
2. It calls the MCP scrape_url tool with a target URL.
3. The MCP server forwards the request to AlterLab, which returns cleaned content.
4. The agent extracts the needed snippet, returns it to the LLM, which generates the final answer.

Setting Up an MCP Server for Web Data

A minimal MCP server can be built with any language that supports JSON‑RPC. Below is a Python example using the mcp library (hypothetical; replace with your preferred framework).

from mcp import MCPServer, Tool
import alterlab
import asyncio

client = alterlab.Client("YOUR_API_KEY")  # altered for brevity

async def scrape_url(args: dict) -> dict:
    url = args.get("url")
    if not url:
        raise ValueError("Missing 'url' argument")
    resp = await client.scrape(
        url,
        formats=["text"],          # ask for plain text to reduce payload
        javascript=True,           # enable headless rendering
    )
    return {"text": resp.text[:8000]}  # truncate to fit context window

server = MCPServer()
server.register_tool(
    Tool(
        name="scrape_url",
        description="Fetch and clean a web page via AlterLab",
        parameters={
            "type": "object",
            "properties": {"url": {"type": "string"}},
            "required": ["url"],
        },
        handler=scrape_url,
    )
)

if __name__ == "__main__":
    asyncio.run(server.run(host="0.0.0.0", port=8000))

Line 3: Initialize the AlterLab client with your API key.
Lines 7‑16: Define the tool implementation that calls AlterLab, requests plain text, and caps the response size.
Lines 18‑27: Register the tool and start the server on port 8000.

Integrating with AlterLab for Reliable Scraping

AlterLab automatically rotates residential proxies, solves CAPTCHAs, and retries failed requests. By delegating these concerns to the API, the MCP server stays focused on protocol handling and observability. To use AlterLab effectively:

• Enable javascript=True for SPA content.
• Choose the formats parameter (["json"], ["markdown"], or ["text"]) based on downstream parsing needs.
• Set a reasonable timeout (e.g., 30 s) to avoid hanging agent turns.

Code Example: Python SDK

The following snippet shows how an agent (or any client) would call the MCP server from Python.

import httpx
import json

MCP_ENDPOINT = "http://mcp-server:8000"

async def scrape_via_mcp(url: str) -> str:
    payload = {
        "jsonrpc": "2.0",
        "method": "scrape_url",
        "params": {"url": url},
        "id": 1,
    }
    async with httpx.AsyncClient() as client:
        resp = await client.post(MCP_ENDPOINT, json=payload, timeout=30.0)
        resp.raise_for_status()
        result = resp.json()
        return result["result"]["text"]

# Usage
text = await scrape_via_mcp("https://example.com/products")
print(text[:500])

Lines 4‑9: Build a JSON‑RPC request for the scrape_url tool.
Lines 10‑14: Send the request via HTTPX and extract the returned text.

Code Example: cURL

You can test the MCP endpoint directly with cURL:

curl -X POST http://mcp-server:8000 \
  -H "Content-Type: application/json" \
  -d '{
        "jsonrpc": "2.0",
        "method": "scrape_url",
        "params": {"url": "https://example.com"},
        "id": 1
      }'

The response mirrors the JSON‑RPC format, with the scraped text under result.text.

Step Flow Infographic

TryIt Block

Best Practices for Scalability

1. Horizontal MCP servers: Run multiple instances behind a load balancer; each is stateless aside from API key configuration.
2. Batching: If an agent needs several URLs, send a single MCP call that returns an array of results to reduce round‑trips.
3. Caching layer: Store recently fetched pages (with TTL) in Redis or Memcached to spare AlterLab calls for repeat queries.
4. Rate limiting: Respect AlterLab’s tier limits; use the X-RateLimit-Remaining header to throttle MCP workers.
5. Observability: Log each tool call with latency, status code, and URL length; alert on error spikes.

Conclusion

By treating web scraping as a tool exposed through an MCP server, agentic browsing pipelines gain modularity, scalability, and resilience to anti‑bot measures. AlterLab handles the heavy lifting of proxy management and browser automation, letting engineers focus on the retrieval logic and LLM integration. This pattern scales to thousands of concurrent requests while keeping data fresh and