AI Research Agent: Web Search + Structured Extraction

TL;DR

Build an AI research agent that performs web search, fetches result pages, and extracts structured fields using AlterLab’s scraping API. The agent uses the Python SDK for HTTP requests and AlterLab’s anti‑bot handling to retrieve clean data, then parses it with a lightweight HTML parser. Schedule the script with cron to run unattended.

Introduction

AI research agents reduce manual effort in data collection by automating two core steps: discovering relevant pages and pulling out specific information. Instead of copying‑pasting from search results, the agent issues a search query, retrieves the top URLs, scrapes each page, and extracts fields such as title, price, or metadata into a structured format. This guide shows how to implement that flow with AlterLab’s API, which manages proxies, browser rendering, and anti‑bot challenges so you can focus on the extraction logic.

Architecture Overview

The agent consists of four components:

1. Query Builder – turns a natural language question into a search engine URL.
2. Search Fetcher – retrieves the search results page and extracts links.
3. Page Scraper – fetches each target page via AlterLab and returns raw HTML.
4. Data Parser – runs a selector‑based or regex‑based extractor to produce JSON records.

Each component can be swapped or scaled independently. For example, you could replace the search fetcher with a dedicated search API, or add a deduplication step before scraping.

Setting Up AlterLab

First, create an account at AlterLab and generate an API key. Install the official Python SDK to simplify requests:

pip install alterlab

The SDK handles authentication, retries, and response parsing. Initialize it with your key:

import alterlab

client = alterlab.Client("YOUR_API_KEY")   # authenticates all requests

When you call client.scrape, AlterLab automatically routes the request through a headless browser if needed, applies rotating proxies, and solves common anti‑bot challenges. This means you can scrape sites that employ basic bot detection without managing your own browser fleet. For details on the underlying technology, see the anti‑bot handling page.

Implementing Web Search

We’ll use a public search endpoint that returns HTML results (e.g., a search engine’s results page). The goal is to pull the first N links. The following function builds a search URL, fetches the page, and extracts anchors with a simple CSS selector.

from urllib.parse import quote_plus
from bs4 import BeautifulSoup

def fetch_search_links(query: str, limit: int = 5) -> list[str]:
    """Return a list of result URLs for the given query."""
    encoded = quote_plus(query)
    search_url = f"https://example.com/search?q={encoded}"  # generic placeholder
    resp = client.scrape(search_url, formats=["html"])
    soup = BeautifulSoup(resp.text, "html.parser")
    links = []
    for a in soup.select("a.result-link"):  # adjust selector to your target
        href = a.get("href")
        if href and href.startswith("http"):
            links.append(href)
        if len(links) >= limit:
            break
    return links

Key points:

• The request uses formats=["html"] to get raw HTML; you could also ask for ["text"] if you only need visible text.
• The selector "a.result-link" is a placeholder; replace it with the appropriate selector for the search engine you target.
• The function returns a list of absolute URLs ready for scraping.

Try it yourself – replace the placeholder URL with a real search endpoint and run the function with a test query to see the links it returns.

Structured Extraction with AlterLab

Once you have a list of URLs, the next step is to scrape each page and pull out the fields you need. AlterLab can return data in multiple formats; requesting JSON often saves a parsing step if the site offers structured data via JSON‑LD or microdata. Otherwise, you receive HTML and parse it yourself.

The example below scrapes a product‑like page and extracts title, price, and availability using BeautifulSoup. Adjust the selectors to match the structure of your target pages.

def scrape_and_extract(url: str) -> dict:
    """Scrape a