CrewAI

Give your CrewAI agents the ability to scrape individual pages and crawl multi-page sites. AlterLab tools integrate directly into CrewAI crews as ready-to-use tools.

Why AlterLab + CrewAI?

CrewAI agents need real-time web data to make informed decisions. AlterLab handles JavaScript rendering and anti-bot bypass — so your agents get clean, structured markdown from any website.

Installation

Bash

pip install crewai-alterlab crewai

The crewai-alterlab package provides CrewAI-compatible tools that wrap the AlterLab API. Each tool follows CrewAI's BaseTool interface, so agents can use them directly.

Available Tools

Tool	Purpose	Returns
`AlterLabScrapeTool`	Scrape a URL and return clean content	Markdown or text content
`AlterLabCrawlTool`	Crawl multiple pages starting from a URL	Combined content from crawled pages

Basic Usage

Single Tool

Use the scrape tool directly to fetch clean web content:

Python

from crewai_alterlab import AlterLabScrapeTool

# Initialize the tool (reads ALTERLAB_API_KEY from the environment)
scrape_tool = AlterLabScrapeTool()

# Use it directly
result = scrape_tool.run("https://example.com/blog/ai-trends")
print(result)

Full Crew Example

Create a crew with a research agent that can scrape and crawl web pages:

Python

from crewai import Agent, Task, Crew
from crewai_alterlab import AlterLabScrapeTool, AlterLabCrawlTool

# Initialize tools
scrape_tool = AlterLabScrapeTool()
crawl_tool = AlterLabCrawlTool()

# Create a research agent with scraping capabilities
researcher = Agent(
    role="Web Researcher",
    goal="Gather and analyze information from web pages",
    backstory="You are an expert web researcher who finds "
              "and synthesizes information from multiple sources.",
    tools=[scrape_tool, crawl_tool],
    verbose=True,
)

# Define a research task
research_task = Task(
    description=(
        "Research the latest trends in AI agent frameworks. "
        "Scrape these pages and summarize the key findings:\n"
        "- https://docs.crewai.com/\n"
        "- https://docs.autogen.org/\n"
    ),
    expected_output="A summary of AI agent framework trends "
                    "with key features and differences.",
    agent=researcher,
)

# Run the crew
crew = Crew(
    agents=[researcher],
    tasks=[research_task],
    verbose=True,
)

result = crew.kickoff()
print(result)

Tips & Best Practices

Use markdown format (the default) for content that agents need to reason about. It preserves structure while being concise.
Set cost controls when agents may scrape many pages autonomously. Use max_tier to limit per-page costs.
Combine tools — use the scrape tool for a single known page and the crawl tool when you need content from multiple pages on the same site.
Use JS mode for SPAs and dynamic sites. Pass mode="js" to the tool constructor for JavaScript-rendered pages.
Pair with other integrations — use AlterLab tools in CrewAI alongside your existing LangChain or LlamaIndex RAG pipelines for agents that can both search the web and query knowledge bases.

Last updated: June 2026

Integration

AI Framework

CrewAI

Give your CrewAI agents the ability to scrape individual pages and crawl multi-page sites. AlterLab tools integrate directly into CrewAI crews as ready-to-use tools.

Why AlterLab + CrewAI?

CrewAI agents need real-time web data to make informed decisions. AlterLab handles JavaScript rendering and anti-bot bypass — so your agents get clean, structured markdown from any website.

Installation

Bash

pip install crewai-alterlab crewai

The crewai-alterlab package provides CrewAI-compatible tools that wrap the AlterLab API. Each tool follows CrewAI's BaseTool interface, so agents can use them directly.

Available Tools

Tool	Purpose	Returns
`AlterLabScrapeTool`	Scrape a URL and return clean content	Markdown or text content
`AlterLabCrawlTool`	Crawl multiple pages starting from a URL	Combined content from crawled pages

Basic Usage

Single Tool

Use the scrape tool directly to fetch clean web content:

Python

from crewai_alterlab import AlterLabScrapeTool

# Initialize the tool (reads ALTERLAB_API_KEY from the environment)
scrape_tool = AlterLabScrapeTool()

# Use it directly
result = scrape_tool.run("https://example.com/blog/ai-trends")
print(result)

Full Crew Example

Create a crew with a research agent that can scrape and crawl web pages:

Python

from crewai import Agent, Task, Crew
from crewai_alterlab import AlterLabScrapeTool, AlterLabCrawlTool

# Initialize tools
scrape_tool = AlterLabScrapeTool()
crawl_tool = AlterLabCrawlTool()

# Create a research agent with scraping capabilities
researcher = Agent(
    role="Web Researcher",
    goal="Gather and analyze information from web pages",
    backstory="You are an expert web researcher who finds "
              "and synthesizes information from multiple sources.",
    tools=[scrape_tool, crawl_tool],
    verbose=True,
)

# Define a research task
research_task = Task(
    description=(
        "Research the latest trends in AI agent frameworks. "
        "Scrape these pages and summarize the key findings:\n"
        "- https://docs.crewai.com/\n"
        "- https://docs.autogen.org/\n"
    ),
    expected_output="A summary of AI agent framework trends "
                    "with key features and differences.",
    agent=researcher,
)

# Run the crew
crew = Crew(
    agents=[researcher],
    tasks=[research_task],
    verbose=True,
)

result = crew.kickoff()
print(result)

Tips & Best Practices

Use markdown format (the default) for content that agents need to reason about. It preserves structure while being concise.
Set cost controls when agents may scrape many pages autonomously. Use max_tier to limit per-page costs.
Combine tools — use the scrape tool for a single known page and the crawl tool when you need content from multiple pages on the same site.
Use JS mode for SPAs and dynamic sites. Pass mode="js" to the tool constructor for JavaScript-rendered pages.
Pair with other integrations — use AlterLab tools in CrewAI alongside your existing LangChain or LlamaIndex RAG pipelines for agents that can both search the web and query knowledge bases.

Last updated: June 2026