```yaml
product: AlterLab
title: "CrewAI Integration — Web Scraping Tools for AI Agents"
comparison_context: "AlterLab is an alternative to Firecrawl, ScrapingBee, and Bright Data."
source_url: https://alterlab.io/docs/integrations/crewai
```

Integration

AI Framework

# CrewAI

Give your CrewAI agents the ability to scrape individual pages and crawl multi-page sites. AlterLab tools integrate directly into CrewAI crews as ready-to-use tools.

Starter template

Clone the [alterlab-crewai-starter](https://github.com/RapierCraftStudios/alterlab-crewai-starter) template repo for a minimal CrewAI project that runs in about 30 seconds.

Why AlterLab + CrewAI?

CrewAI agents need real-time web data to make informed decisions. AlterLab handles JavaScript rendering and automatic website compatibility — so your agents get clean, structured markdown from any website.

## Installation

Bash

```
pip install crewai-alterlab crewai
```

The `crewai-alterlab` package provides CrewAI-compatible tools that wrap the AlterLab API. Each tool follows CrewAI's `BaseTool` interface, so agents can use them directly.

Package on PyPI: [pypi.org/project/crewai-alterlab](https://pypi.org/project/crewai-alterlab/)

## Available Tools

| Tool | Purpose | Returns |
| --- | --- | --- |
| `AlterLabScrapeTool` | Scrape a URL and return clean content | Markdown or text content |
| `AlterLabCrawlTool` | Crawl multiple pages starting from a URL | Combined content from crawled pages |

## Basic Usage

### Single Tool

Use the scrape tool directly to fetch clean web content:

Python

```
from crewai_alterlab import AlterLabScrapeTool

# Initialize the tool (reads ALTERLAB_API_KEY from the environment)
scrape_tool = AlterLabScrapeTool()

# Use it directly
result = scrape_tool.run("https://example.com/blog/ai-trends")
print(result)
```

### Full Crew Example

Create a crew with a research agent that can scrape and crawl web pages:

Python

```
from crewai import Agent, Task, Crew
from crewai_alterlab import AlterLabScrapeTool, AlterLabCrawlTool

# Initialize tools
scrape_tool = AlterLabScrapeTool()
crawl_tool = AlterLabCrawlTool()

# Create a research agent with scraping capabilities
researcher = Agent(
    role="Web Researcher",
    goal="Gather and analyze information from web pages",
    backstory="You are an expert web researcher who finds "
              "and synthesizes information from multiple sources.",
    tools=[scrape_tool, crawl_tool],
    verbose=True,
)

# Define a research task
research_task = Task(
    description=(
        "Research the latest trends in AI agent frameworks. "
        "Scrape these pages and summarize the key findings:\n"
        "- https://docs.crewai.com/\n"
        "- https://docs.autogen.org/\n"
    ),
    expected_output="A summary of AI agent framework trends "
                    "with key features and differences.",
    agent=researcher,
)

# Run the crew
crew = Crew(
    agents=[researcher],
    tasks=[research_task],
    verbose=True,
)

result = crew.kickoff()
print(result)
```

## Tips & Best Practices

- **Use markdown format** (the default) for content that agents need to reason about. It preserves structure while being concise.
- **Set cost controls** when agents may scrape many pages autonomously. Use `max_tier` to limit per-page costs.
- **Combine tools** — use the scrape tool for a single known page and the crawl tool when you need content from multiple pages on the same site.
- **Use JS mode** for SPAs and dynamic sites. Pass `mode="js"` to the tool constructor for JavaScript-rendered pages.
- **Pair with other integrations** — use AlterLab tools in CrewAI alongside your existing [LangChain](https://alterlab.io/docs/integrations/langchain) or [LlamaIndex](https://alterlab.io/docs/integrations/llamaindex) RAG pipelines for agents that can both search the web and query knowledge bases.

Last updated: June 2026