Use Case

Market Research & Competitive Intelligence

Gather competitive intelligence, analyze market trends, and research industries at scale with web scraping and structured extraction.

The Problem

Market research requires gathering data from hundreds of sources — competitor websites, industry publications, review sites, and public databases. Doing this manually is slow and incomplete:

Competitor websites change frequently and are hard to track
Industry data is scattered across many sites with different formats
Search results need to be scraped and analyzed, not just viewed
Bulk data collection requires infrastructure you don't want to manage

Solution Architecture

Use AlterLab's search, scrape, and extract endpoints to build a complete research pipeline:

1. Search

POST /search to find relevant pages across the web for your research topic. Returns URLs, titles, and snippets.

2. Scrape & Extract

POST /scrape each result with extraction_schema to pull structured insights from each source.

3. Batch

POST /batch to process all search results in parallel. Combine results into a research dataset.

Quick Example

Search for a topic and extract structured data from the top results:

Python

import requests

# Step 1: Search for relevant pages
search_resp = requests.post(
    "https://api.alterlab.io/api/v1/search",
    headers={"X-API-Key": "YOUR_API_KEY"},
    json={
        "query": "enterprise web scraping API pricing 2026",
        "num_results": 10
    }
)

results = search_resp.json().get("results", [])
print(f"Found {len(results)} results")

# Step 2: Extract structured data from each result
for result in results[:5]:
    scrape_resp = requests.post(
        "https://api.alterlab.io/api/v1/scrape",
        headers={"X-API-Key": "YOUR_API_KEY"},
        json={
            "url": result["url"],
            "extraction_schema": {
                "type": "object",
                "properties": {
                    "company_name": {"type": "string"},
                    "pricing_model": {"type": "string"},
                    "starting_price": {"type": "string"},
                    "key_features": {"type": "array"}
                }
            }
        }
    )
    data = scrape_resp.json().get("filtered_content", {})
    print(f"{data.get('company_name')}: {data.get('starting_price')}")

Advanced Patterns

Competitive Analysis

Build a competitor tracking system that monitors pricing pages, feature lists, and positioning across multiple competitors:

Python

import requests
import json

competitors = [
    {"name": "Competitor A", "pricing_url": "https://competitor-a.com/pricing"},
    {"name": "Competitor B", "pricing_url": "https://competitor-b.com/pricing"},
    {"name": "Competitor C", "pricing_url": "https://competitor-c.com/pricing"},
]

pricing_schema = {
    "type": "object",
    "properties": {
        "plans": {"type": "array"},
        "starting_price": {"type": "string"},
        "enterprise_available": {"type": "boolean"},
        "free_tier": {"type": "boolean"}
    }
}

def analyze_competitors(api_key):
    """Scrape and compare competitor pricing pages."""
    results = []
    for comp in competitors:
        resp = requests.post(
            "https://api.alterlab.io/api/v1/scrape",
            headers={"X-API-Key": api_key},
            json={
                "url": comp["pricing_url"],
                "cache": False,
                "extraction_schema": pricing_schema
            }
        )
        data = resp.json().get("filtered_content", {})
        results.append({
            "competitor": comp["name"],
            **data
        })
    return results

report = analyze_competitors("YOUR_API_KEY")
for entry in report:
    print(f"{entry['competitor']}: starts at {entry.get('starting_price', 'N/A')}")

Bulk Research Pipeline

Use batch scraping to process large lists of URLs from search results or curated source lists:

Python

import requests

def bulk_research(api_key, urls, schema, webhook_url=None):
    """Submit a batch of URLs for structured extraction."""
    response = requests.post(
        "https://api.alterlab.io/api/v1/batch",
        headers={"X-API-Key": api_key},
        json={
            "requests": [
                {"url": url, "extraction_schema": schema}
                for url in urls
            ],
            "webhook_url": webhook_url
        }
    )
    return response.json().get("job_id")

# Research schema for industry articles
article_schema = {
    "type": "object",
    "properties": {
        "title": {"type": "string"},
        "author": {"type": "string"},
        "date": {"type": "string"},
        "summary": {"type": "string"},
        "key_points": {"type": "array"}
    }
}

# Submit batch of 50 industry articles
urls = ["https://source1.com/article", "https://source2.com/report"]  # ...
job_id = bulk_research(
    "YOUR_API_KEY",
    urls,
    article_schema,
    webhook_url="https://your-app.com/webhooks/research"
)
print(f"Research batch submitted: {job_id}")

Caching for Research

Research queries often hit the same pages across runs. Enable caching (the default) to avoid re-scraping pages you've already collected. Cache hits are free.

Batch Scraping Guide

Submit up to 1,000 URLs in a single API call.

Structured Extraction Tutorial

Use AI to extract structured JSON data from any web page.

Caching Guide

Reduce costs with intelligent caching. Cache hits are free.

Pricing Reference

Understand tier costs and optimize for bulk research workloads.

← Previous: Content Monitoring Next: Lead Generation →

Last updated: June 2026

Use Case

Market Research & Competitive Intelligence

Gather competitive intelligence, analyze market trends, and research industries at scale with web scraping and structured extraction.

The Problem

Market research requires gathering data from hundreds of sources — competitor websites, industry publications, review sites, and public databases. Doing this manually is slow and incomplete:

Competitor websites change frequently and are hard to track
Industry data is scattered across many sites with different formats
Search results need to be scraped and analyzed, not just viewed
Bulk data collection requires infrastructure you don't want to manage

Solution Architecture

Use AlterLab's search, scrape, and extract endpoints to build a complete research pipeline:

1. Search

POST /search to find relevant pages across the web for your research topic. Returns URLs, titles, and snippets.

2. Scrape & Extract

POST /scrape each result with extraction_schema to pull structured insights from each source.

3. Batch

POST /batch to process all search results in parallel. Combine results into a research dataset.

Quick Example

Search for a topic and extract structured data from the top results:

Python

import requests

# Step 1: Search for relevant pages
search_resp = requests.post(
    "https://api.alterlab.io/api/v1/search",
    headers={"X-API-Key": "YOUR_API_KEY"},
    json={
        "query": "enterprise web scraping API pricing 2026",
        "num_results": 10
    }
)

results = search_resp.json().get("results", [])
print(f"Found {len(results)} results")

# Step 2: Extract structured data from each result
for result in results[:5]:
    scrape_resp = requests.post(
        "https://api.alterlab.io/api/v1/scrape",
        headers={"X-API-Key": "YOUR_API_KEY"},
        json={
            "url": result["url"],
            "extraction_schema": {
                "type": "object",
                "properties": {
                    "company_name": {"type": "string"},
                    "pricing_model": {"type": "string"},
                    "starting_price": {"type": "string"},
                    "key_features": {"type": "array"}
                }
            }
        }
    )
    data = scrape_resp.json().get("filtered_content", {})
    print(f"{data.get('company_name')}: {data.get('starting_price')}")

Advanced Patterns

Competitive Analysis

Build a competitor tracking system that monitors pricing pages, feature lists, and positioning across multiple competitors:

Python

import requests
import json

competitors = [
    {"name": "Competitor A", "pricing_url": "https://competitor-a.com/pricing"},
    {"name": "Competitor B", "pricing_url": "https://competitor-b.com/pricing"},
    {"name": "Competitor C", "pricing_url": "https://competitor-c.com/pricing"},
]

pricing_schema = {
    "type": "object",
    "properties": {
        "plans": {"type": "array"},
        "starting_price": {"type": "string"},
        "enterprise_available": {"type": "boolean"},
        "free_tier": {"type": "boolean"}
    }
}

def analyze_competitors(api_key):
    """Scrape and compare competitor pricing pages."""
    results = []
    for comp in competitors:
        resp = requests.post(
            "https://api.alterlab.io/api/v1/scrape",
            headers={"X-API-Key": api_key},
            json={
                "url": comp["pricing_url"],
                "cache": False,
                "extraction_schema": pricing_schema
            }
        )
        data = resp.json().get("filtered_content", {})
        results.append({
            "competitor": comp["name"],
            **data
        })
    return results

report = analyze_competitors("YOUR_API_KEY")
for entry in report:
    print(f"{entry['competitor']}: starts at {entry.get('starting_price', 'N/A')}")

Bulk Research Pipeline

Use batch scraping to process large lists of URLs from search results or curated source lists:

Python

import requests

def bulk_research(api_key, urls, schema, webhook_url=None):
    """Submit a batch of URLs for structured extraction."""
    response = requests.post(
        "https://api.alterlab.io/api/v1/batch",
        headers={"X-API-Key": api_key},
        json={
            "requests": [
                {"url": url, "extraction_schema": schema}
                for url in urls
            ],
            "webhook_url": webhook_url
        }
    )
    return response.json().get("job_id")

# Research schema for industry articles
article_schema = {
    "type": "object",
    "properties": {
        "title": {"type": "string"},
        "author": {"type": "string"},
        "date": {"type": "string"},
        "summary": {"type": "string"},
        "key_points": {"type": "array"}
    }
}

# Submit batch of 50 industry articles
urls = ["https://source1.com/article", "https://source2.com/report"]  # ...
job_id = bulk_research(
    "YOUR_API_KEY",
    urls,
    article_schema,
    webhook_url="https://your-app.com/webhooks/research"
)
print(f"Research batch submitted: {job_id}")

Caching for Research

Research queries often hit the same pages across runs. Enable caching (the default) to avoid re-scraping pages you've already collected. Cache hits are free.

Last updated: June 2026

Market Research & Competitive Intelligence

The Problem

Solution Architecture

1. Search

2. Scrape & Extract

3. Batch

Quick Example

Advanced Patterns

Competitive Analysis

Bulk Research Pipeline

Related Guides

Batch Scraping Guide

Structured Extraction Tutorial

Caching Guide

Pricing Reference

Market Research & Competitive Intelligence

The Problem

Solution Architecture

1. Search

2. Scrape & Extract

3. Batch

Quick Example

Advanced Patterns

Competitive Analysis

Bulk Research Pipeline

Related Guides

Batch Scraping Guide

Structured Extraction Tutorial

Caching Guide

Pricing Reference