```yaml
product: AlterLab
title: "AI Research Agent: Web Search + Structured Extraction"
category: Tutorials
comparison_context: "AlterLab is an alternative to Firecrawl, ScrapingBee, and Bright Data."
last_updated: 2026-08-26
canonical_facts:
- "Learn how to build an AI research agent that combines web search with AlterLab's scraping API to extract structured data from public web pages, using Python SDK and cron scheduling."
source_url: https://alterlab.io/blog/ai-research-agent-web-search-structured-extraction
```
## TL;DR
Build an AI research agent that performs web search, fetches result pages, and extracts structured fields using AlterLab’s scraping API. The agent uses the Python SDK for HTTP requests and AlterLab’s anti‑bot handling to retrieve clean data, then parses it with a lightweight HTML parser. Schedule the script with cron to run unattended.
## Introduction
AI research agents reduce manual effort in data collection by automating two core steps: discovering relevant pages and pulling out specific information. Instead of copying‑pasting from search results, the agent issues a search query, retrieves the top URLs, scrapes each page, and extracts fields such as title, price, or metadata into a structured format. This guide shows how to implement that flow with AlterLab’s API, which manages proxies, browser rendering, and anti‑bot challenges so you can focus on the extraction logic.
## Architecture Overview
The agent consists of four components:
1. **Query Builder** – turns a natural language question into a search engine URL.
2. **Search Fetcher** – retrieves the search results page and extracts links.
3. **Page Scraper** – fetches each target page via AlterLab and returns raw HTML.
4. **Data Parser** – runs a selector‑based or regex‑based extractor to produce JSON records.
Each component can be swapped or scaled independently. For example, you could replace the search fetcher with a dedicated search API, or add a deduplication step before scraping.
## Setting Up AlterLab
First, create an account at [AlterLab](https://alterlab.io/signup) and generate an API key. Install the official Python SDK to simplify requests:
```bash title="Terminal"
pip install alterlab
```
The SDK handles authentication, retries, and response parsing. Initialize it with your key:
```python title="agent.py" {1-2}
import alterlab
client = alterlab.Client("YOUR_API_KEY") # authenticates all requests
```
When you call `client.scrape`, AlterLab automatically routes the request through a headless browser if needed, applies rotating proxies, and solves common anti‑bot challenges. This means you can scrape sites that employ basic bot detection without managing your own browser fleet. For details on the underlying technology, see the [anti‑bot handling](https://alterlab.io/smart-rendering-api) page.
## Implementing Web Search
We’ll use a public search endpoint that returns HTML results (e.g., a search engine’s results page). The goal is to pull the first N links. The following function builds a search URL, fetches the page, and extracts anchors with a simple CSS selector.
```python title="agent.py" {5-15}
from urllib.parse import quote_plus
from bs4 import BeautifulSoup
def fetch_search_links(query: str, limit: int = 5) -> list[str]:
"""Return a list of result URLs for the given query."""
encoded = quote_plus(query)
search_url = f"https://example.com/search?q={encoded}" # generic placeholder
resp = client.scrape(search_url, formats=["html"])
soup = BeautifulSoup(resp.text, "html.parser")
links = []
for a in soup.select("a.result-link"): # adjust selector to your target
href = a.get("href")
if href and href.startswith("http"):
links.append(href)
if len(links) >= limit:
break
return links
```
Key points:
- The request uses `formats=["html"]` to get raw HTML; you could also ask for `["text"]` if you only need visible text.
- The selector `"a.result-link"` is a placeholder; replace it with the appropriate selector for the search engine you target.
- The function returns a list of absolute URLs ready for scraping.
> **Try it yourself** – replace the placeholder URL with a real search endpoint and run the function with a test query to see the links it returns.
## Structured Extraction with AlterLab
Once you have a list of URLs, the next step is to scrape each page and pull out the fields you need. AlterLab can return data in multiple formats; requesting JSON often saves a parsing step if the site offers structured data via JSON‑LD or microdata. Otherwise, you receive HTML and parse it yourself.
The example below scrapes a product‑like page and extracts title, price, and availability using BeautifulSoup. Adjust the selectors to match the structure of your target pages.
```python title="agent.py" {18-35}
def scrape_and_extract(url: str) -> dict:
"""Scrape a
## Frequently Asked Questions
### What is an AI research agent?
An AI research agent automates the process of gathering information from the web, extracting structured data, and storing it for analysis. It typically combines a search step with a scraping step to turn unstructured pages into usable datasets.
### How does AlterLab help with web scraping for AI agents?
AlterLab provides a scraping API that handles anti‑bot measures, rotating proxies, and headless browser rendering, returning clean HTML or structured formats like JSON. This lets agents focus on data extraction rather than bypassing blocks.
### Can I schedule an AI research agent to run regularly?
Yes. By wrapping the agent script in a cron job or using AlterLab’s scheduling feature, you can automate recurring searches and extractions at set intervals without manual intervention.
## Related
- [Grounding LLM Responses with Live Web Data: Patterns and Pitfalls]()
- [Automating Competitive Intelligence with Web Data APIs]()
- [Web Search API for AI Agents: Developer's Guide]()