Automating Competitive Intelligence with Web Data APIs

TL;DR

Automate competitive intelligence by scraping public web pages with a reliable API, feeding the raw content into an LLM for summarization, and storing the results for analysis. This pipeline reduces manual data gathering and turns unstructured pages into concise, actionable insights.

Why Automate Competitive Intelligence

Manual competitive research involves visiting dozens of sites, copying data, and writing summaries—a process that scales poorly. An automated pipeline handles three core tasks: fetching up-to-date pages, extracting relevant information, and condensing it into a readable format. By using a web data API that manages anti-bot measures, rotating proxies, and headless rendering, you avoid the operational overhead of maintaining your own scraping infrastructure. The LLM step adds value by turning noisy HTML into focused briefings that highlight price changes, product updates, or sentiment shifts.

Architecture Overview

A typical pipeline consists of four stages:

1. Scheduler – Triggers scraping jobs on a cron-like schedule.
2. Fetcher – Calls a web scraping API to retrieve page content.
3. Processor – Strips scripts, ads, and navigation; optionally extracts structured fields.
4. Summarizer – Sends cleaned text to an LLM and stores the output.

Each stage can run independently, allowing you to scale the fetcher horizontally while keeping the summarizer lightweight. The design also makes it easy to swap components—for example, replacing the LLM with a rule‑based extractor for simple metrics.

Step 1: Scrape Target Pages with AlterLab

AlterLab provides a programmable interface that handles proxy rotation, JavaScript rendering, and automatic retries. Below is a Python example that fetches a product listing page and returns raw HTML.

import alterlab
import os

client = alterlab.Client(os.getenv("ALTERLAB_API_KEY"))   # initialized with env var
response = client.scrape(
    url="https://example-shop.com/category/laptops",
    params={"render_js": True, "wait_for": ".product-card"}  # highlighted
)

if response.status_code == 200:
    html = response.text
    print(f"Fetched {len(html)} bytes")
else:
    raise RuntimeError(f"Request failed: {response.status_code}")

The render_js flag ensures that dynamically loaded content is present before returning the response. You can adjust the wait_for selector to match an element that indicates the page has finished loading.

Note: Only scrape pages that are publicly accessible and do not require authentication. Respect the site’s crawl‑delay and rate‑limit guidelines.

Step 2: Clean and Prepare the Text

Raw HTML contains boilerplate that distracts an LLM. A quick cleanup using BeautifulSoup removes scripts, styles, and navigation elements, leaving the main content.

from bs4 import BeautifulSoup
import re

def clean_html(html: str) -> str:
    soup = BeautifulSoup(html, "html.parser")
    # remove non‑content tags
    for tag in soup(["script", "style", "nav", "footer", "iframe"]):
        tag.decompose()
    # get text and collapse whitespace
    text = soup.get_text(separator=" ", strip=True)
    text = re.sub(r"\s+", " ", text)
    return text

# usage
cleaned = clean_html(html)
print(f"Cleaned to {len(cleaned)} characters")

The cleaned text is significantly shorter and focuses on the substantive information the LLM needs to summarize.

Step 3: LLM Summarization

With the cleaned text in hand, you can call any LLM provider—OpenAI, Anthropic, or a self‑hosted model. The prompt below asks for a bullet‑point summary of key market signals.

import openai
import os

openai.api_key = os.getenv("OPENAI_API_KEY")

def summarize(text: str, max_tokens: int = 200) -> str:
    response = openai.ChatCompletion.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "You are an analyst summarizing competitive web pages."},
            {"role": "user", "content": f"Provide a concise bullet‑point summary of the following page content, highlighting price changes, product launches, and availability notes:\n\n{text}"}
        ],
        max_tokens=max_tokens,
        temperature=0.3,
    )
    return response.choices[0].message["content"].strip()

# example usage
summary = summarize(cleaned)
print(summary)

The temperature setting keeps the output factual and reduces hallucination. Adjust max_tokens based on the expected length of the summary.

Putting It All Together: Example Pipeline

The following script ties the three steps into a repeatable job. It reads a list of target URLs from a file, processes each one, and appends the summary to a results log.

undefined