Building Agentic Web Browsing Workflows with Markdown Extraction and Headless Browsers

TL;DR

Combine a headless browser to render pages and extract markdown to feed clean, structured text into an LLM. This grounds the model’s responses in current web data, reduces hallucinations, and enables reliable AI agents that can browse, read, and act on live information.

Introduction

AI agents often need up‑to‑date facts that lie outside their training data. Retrieving a live web page, stripping noise, and presenting the core content to an LLM improves answer quality. This tutorial shows how to build a workflow that uses a headless browser for rendering and markdown extraction for LLM‑friendly input, all via AlterLab’s scraping API.

Why Ground LLMs in Live Web Data

LLMs generate text based on patterns learned during training. When a query asks about recent events, product specs, or evolving data, the model may hallucinate or give outdated answers. Fetching the relevant source at request time grounds the response in verifiable facts. The workflow consists of three steps: render the page, extract usable text, and prompt the LLM with that text plus the user query.

Challenges of Raw HTML for LLMs

Raw HTML contains tags, scripts, style attributes, and large amounts of boilerplate. Feeding this directly to an LLM wastes tokens on irrelevant markup and can confuse the model. Additionally, many sites rely on JavaScript to load content; a simple HTTP GET returns an incomplete page. A headless browser solves the rendering problem, but we still need to transform the rendered DOM into a concise text format.

Using Markdown Extraction

Markdown offers a lightweight markup language that preserves headings, lists, and code blocks while discarding most HTML noise. Converting the DOM to markdown yields a compact representation that retains semantic structure. AlterLab provides a formats=['markdown'] option that returns the page content as markdown after rendering, making it ideal for LLM consumption.

Headless Browser Integration

Modern sites often require JavaScript execution, cookie handling, or user interactions to reveal content. A headless browser (e.g., Chromium) runs in a server environment, loads the page, waits for network idle, and returns the final DOM. AlterLab’s API automatically selects the appropriate rendering tier based on the page’s complexity, handling rotating proxies and anti‑bot challenges transparently.

Building the Workflow

The workflow is stateless and can be wrapped in a function or microservice that accepts a URL and a question, returns the LLM’s answer.

Code Example: Python SDK

import alterlab

client = alterlab.Client("YOUR_API_KEY")   # highlighted

def answer_with_web(url: str, question: str) -> str:
    # Render page and get markdown
    scrape_res = client.scrape(
        url,
        formats=["markdown"],          # highlighted
        wait_for_network_idle=True
    )
    markdown_text = scrape_res.text

    # Build prompt for LLM (pseudo‑code)
    prompt = f"Using the following information, answer the question:\n\n{markdown_text}\n\nQuestion: {question}"
    # Call your LLM here (e.g., OpenAI, Anthropic)
    llm_response = call_llm(prompt)
    return llm_response

The formats=["markdown"] flag tells AlterLab to run the headless browser, wait for content, and convert the DOM to markdown before returning it.

Code Example: cURL

curl -X POST https://api.alterlab.io/v1/scrape \
  -H "X-API-Key: YOUR_KEY" \
  -d '{
        "url": "https://example.com/page",
        "formats": ["markdown"],
        "wait_for_network_idle": true
      }'  # highlighted

The JSON payload requests markdown output after rendering. The response body contains the markdown string ready for LLM prompting.

TryIt Block

Replace the URL with any publicly accessible page to test the workflow.

Best Practices

• Cache when appropriate: If the same page is queried repeatedly within a short window, cache the markdown to reduce API calls and latency.
• Limit token usage: Truncate overly long markdown sections or summarize before sending to the LLM to stay within model context limits.
• Handle errors gracefully: Check for HTTP status codes, retry on transient failures, and validate that markdown is not empty before prompting.
• Respect site policies: Only scrape publicly accessible content, respect rate limits, and consider adding a reasonable delay between requests to the same domain.
• Secure API keys: Store your AlterLab key in environment variables or a secret manager; never commit it to source control.

Conclusion

By rendering pages with a headless browser and extracting markdown, you create a clean, structured feed for LLMs that grounds their responses in fresh web data. This approach improves factual reliability, reduces hallucinations, and enables agents to interact with the web as a knowledgeable source. Integrate the pattern into your AI pipelines to build more trustworthy, data‑driven applications.

Takeaway

Use AlterLab’s markdown format with headless browser rendering to turn live web pages into LLM‑ready text. Combine the output with your LLM prompt to ground answers in current, verifiable information. This simple two‑step process—render then extract—forms the foundation of effective agentic web browsing workflows.