Building Agentic Web Browsing Tools with Real-Time Data and MCP Servers

TL;DR

Agentic web browsing tools let LLMs autonomously retrieve and act on live web data by treating a scraping API as a callable tool via an MCP server. This approach removes the need for custom scraper code while giving agents up-to-date information for decision-making.

Introduction

Modern LLM applications often need current facts that lie outside the model’s training data. Instead of relying on stale knowledge or frequent retraining, you can equip the model with a tool that fetches fresh web content on demand. By wrapping a scraping API in an MCP (Model Context Protocol) server, you expose a standardized interface that the LLM can invoke just like any other function. The result is an agent that reasons, decides when to scrape, extracts the needed data, and continues its workflow—all without you writing brittle scraper logic.

Why Agentic Browsing Matters

Static prompts limit LLMs to the information they were trained on. When a user asks for the latest price, a recent news headline, or up-to-date product specifications, the model must consult an external source. Building agentic browsing solves this by:

• Enabling real‑time lookups without manual data pipelines
• Keeping the scraping logic centralized and maintainable
• Allowing the LLM to control frequency and scope based on context
• Reducing engineering overhead because the scraping API handles anti‑bot measures, JavaScript rendering, and proxy rotation

Architecture Overview

The system consists of three loosely coupled pieces:

1. LLM Agent – decides when to invoke the scraping tool and processes the returned data
2. MCP Server – wraps the scraping API in a JSON‑RPC‑style endpoint that the LLM can call
3. Scraping API – provides reliable access to public web pages with automatic header rotation, JavaScript rendering, and CAPTCHA bypass

The LLM never sees the raw HTML; it receives a cleaned JSON payload (or markdown/text) that it can immediately use in its reasoning loop.

Setting Up the MCP Server

You can run the MCP server locally or in the cloud. Below is a minimal Python implementation using FastAPI. It exposes a single /tool/scrape endpoint that forwards the request to AlterLab.

from fastapi import FastAPI, HTTPException
import httpx
import os

app = FastAPI()
ALTERLAB_KEY = os.getenv("ALTERLAB_API_KEY")
ALTERLAB_ENDPOINT = "https://api.alterlab.io/v1/scrape"

@app.post("/tool/scrape")
async def scrape_tool(payload: dict):
    url = payload.get("url")
    if not url:
        raise HTTPException(status_code=400, detail="Missing 'url' field")
    async with httpx.AsyncClient() as client:
        response = await client.post(
            ALTERLAB_ENDPOINT,
            json={"url": url, "formats": ["json"]},
            headers={"X-API-Key": ALTERLAB_KEY},
            timeout=30.0,
        )
        if response.status_code != 200:
            raise HTTPException(status_code=502, detail="Scraping failed")
        return response.json()

The server expects a JSON body with a url field and returns the raw JSON from AlterLab. You can extend it to accept additional parameters like min_tier or formats.

Integrating with LLM Tool Use

Most LLM frameworks (e.g., LangChain, LlamaIndex, or custom agents) let you define tools as functions that return a string or structured object. You point the tool at your MCP server’s endpoint. Below is an example using a pseudo‑agent loop; replace the HTTP call with your framework’s tool invocation method.

import httpx
import json

MCP_ENDPOINT = "http://localhost:8000/tool/scrape"

def scrape_tool(url: str) -> dict:
    resp = httpx.post(MCP_ENDPOINT, json={"url": url})
    resp.raise_for_status()
    return resp.json()

# Example agent reasoning step
user_query = "What is the latest headline on the tech news site?"
if "latest" in user_query.lower():
    # LLM decides to scrape
    data = scrape_tool("https://example-news-site.com/latest")
    headline = data.get("title", "No title found")
    print(f"Fetched headline: {headline}")
else:
    print("Answering from internal knowledge")

The agent checks the user intent, calls the scraping tool when fresh data is needed, and incorporates the result into its response.

Code Examples: Python SDK and cURL

You can also call AlterLab directly from your agent if you prefer not to run an MCP server. The SDK handles retries, header rotation, and response parsing.

import alterlab

client = alterlab.Client(os.getenv("ALTERLAB_API_KEY"))
response = client.scrape(
    "https://example-news-site.com/latest",
    formats=["json"],
    # Optionally force a higher tier for Java