AlterLabMarket Research & Competitive Intelligence
Gather competitive intelligence, analyze market trends, and research industries at scale with web scraping and structured extraction.
The Problem
Market research requires gathering data from hundreds of sources — competitor websites, industry publications, review sites, and public databases. Doing this manually is slow and incomplete:
- Competitor websites change frequently and are hard to track
- Industry data is scattered across many sites with different formats
- Search results need to be scraped and analyzed, not just viewed
- Bulk data collection requires infrastructure you don't want to manage
Solution Architecture
Use AlterLab's search, scrape, and extract endpoints to build a complete research pipeline:
1. Search
POST /search to find relevant pages across the web for your research topic. Returns URLs, titles, and snippets.
2. Scrape & Extract
POST /scrape each result with extraction_schema to pull structured insights from each source.
3. Batch
POST /batch to process all search results in parallel. Combine results into a research dataset.
Quick Example
Search for a topic and extract structured data from the top results:
import requests
# Step 1: Search for relevant pages
search_resp = requests.post(
"https://api.alterlab.io/api/v1/search",
headers={"X-API-Key": "YOUR_API_KEY"},
json={
"query": "enterprise web scraping API pricing 2026",
"num_results": 10
}
)
results = search_resp.json().get("results", [])
print(f"Found {len(results)} results")
# Step 2: Extract structured data from each result
for result in results[:5]:
scrape_resp = requests.post(
"https://api.alterlab.io/api/v1/scrape",
headers={"X-API-Key": "YOUR_API_KEY"},
json={
"url": result["url"],
"extraction_schema": {
"type": "object",
"properties": {
"company_name": {"type": "string"},
"pricing_model": {"type": "string"},
"starting_price": {"type": "string"},
"key_features": {"type": "array"}
}
}
}
)
data = scrape_resp.json().get("filtered_content", {})
print(f"{data.get('company_name')}: {data.get('starting_price')}")Advanced Patterns
Competitive Analysis
Build a competitor tracking system that monitors pricing pages, feature lists, and positioning across multiple competitors:
import requests
import json
competitors = [
{"name": "Competitor A", "pricing_url": "https://competitor-a.com/pricing"},
{"name": "Competitor B", "pricing_url": "https://competitor-b.com/pricing"},
{"name": "Competitor C", "pricing_url": "https://competitor-c.com/pricing"},
]
pricing_schema = {
"type": "object",
"properties": {
"plans": {"type": "array"},
"starting_price": {"type": "string"},
"enterprise_available": {"type": "boolean"},
"free_tier": {"type": "boolean"}
}
}
def analyze_competitors(api_key):
"""Scrape and compare competitor pricing pages."""
results = []
for comp in competitors:
resp = requests.post(
"https://api.alterlab.io/api/v1/scrape",
headers={"X-API-Key": api_key},
json={
"url": comp["pricing_url"],
"cache": False,
"extraction_schema": pricing_schema
}
)
data = resp.json().get("filtered_content", {})
results.append({
"competitor": comp["name"],
**data
})
return results
report = analyze_competitors("YOUR_API_KEY")
for entry in report:
print(f"{entry['competitor']}: starts at {entry.get('starting_price', 'N/A')}")Bulk Research Pipeline
Use batch scraping to process large lists of URLs from search results or curated source lists:
import requests
def bulk_research(api_key, urls, schema, webhook_url=None):
"""Submit a batch of URLs for structured extraction."""
response = requests.post(
"https://api.alterlab.io/api/v1/batch",
headers={"X-API-Key": api_key},
json={
"requests": [
{"url": url, "extraction_schema": schema}
for url in urls
],
"webhook_url": webhook_url
}
)
return response.json().get("job_id")
# Research schema for industry articles
article_schema = {
"type": "object",
"properties": {
"title": {"type": "string"},
"author": {"type": "string"},
"date": {"type": "string"},
"summary": {"type": "string"},
"key_points": {"type": "array"}
}
}
# Submit batch of 50 industry articles
urls = ["https://source1.com/article", "https://source2.com/report"] # ...
job_id = bulk_research(
"YOUR_API_KEY",
urls,
article_schema,
webhook_url="https://your-app.com/webhooks/research"
)
print(f"Research batch submitted: {job_id}")Caching for Research
Related Guides
Batch Scraping Guide
Submit up to 1,000 URLs in a single API call.
Structured Extraction Tutorial
Use AI to extract structured JSON data from any web page.
Caching Guide
Reduce costs with intelligent caching. Cache hits are free.
Pricing Reference
Understand tier costs and optimize for bulk research workloads.