```yaml
product: AlterLab
title: "Scrapy — Web Scraping Glossary | AlterLab"
comparison_context: "AlterLab is an alternative to Firecrawl, ScrapingBee, and Bright Data."
source_url: https://alterlab.io/glossary/scrapy
```

tool

# Scrapy

An open-source Python framework for large-scale web crawling and scraping with async request scheduling, deduplication, and output pipelines.

Scrapy is an open-source Python web crawling and scraping framework designed for large-scale data extraction. It provides a complete, production-ready architecture: asynchronous request scheduling via Twisted, URL deduplication to avoid revisiting pages, middleware pipelines for processing requests and responses, item pipelines for cleaning and storing extracted data, and built-in support for exporting to JSON, CSV, XML, and databases.

Scrapy's architecture is built around Spiders — Python classes that define how to crawl a site (start URLs, following rules) and how to extract data (CSS or XPath selectors). Spiders yield Item objects that flow through configurable pipelines: data cleaning, validation, deduplication, and storage. The framework handles the async I/O, request scheduling, and retry logic automatically.

For JavaScript-rendered pages, Scrapy alone is insufficient — it sends plain HTTP requests. The common integration pattern is to use Scrapy as the crawling and pipeline framework while routing requests through AlterLab's API for rendering. The `scrapy-playwright` and `scrapy-splash` integrations provide alternative browser rendering backends. AlterLab can serve as a transparent HTTP rendering proxy for Scrapy spiders.

### What is Scrapy?

An open-source Python framework for large-scale web crawling and scraping with async request scheduling, deduplication, and output pipelines.

### How does AlterLab handle Scrapy?

For JavaScript-rendered pages, Scrapy alone is insufficient — it sends plain HTTP requests. The common integration pattern is to use Scrapy as the crawling and pipeline framework while routing requests through AlterLab's API for rendering. The `scrapy-playwright` and `scrapy-splash` integrations provide alternative browser rendering backends. AlterLab can serve as a transparent HTTP rendering proxy for Scrapy spiders.

## Examples

```
import scrapy

class ProductSpider(scrapy.Spider):
    name = 'products'

    def start_requests(self):
        # Route through AlterLab for rendering
        yield scrapy.Request(
            'https://api.alterlab.io/v1/scrape',
            method='POST',
            body='{"url": "https://example.com", "render_js": "auto"}',
            headers={'X-API-Key': 'sk_live_...'}
        )
```

## Related Terms

[BeautifulSoupA Python library for parsing HTML and XML and navigating the parse tree using CSS selectors or tag methods.](https://alterlab.io/glossary/beautifulsoup)[JavaScript RenderingThe process of executing a page's JavaScript in a browser engine before extracting content, required for React, Vue, and Angular SPAs.](https://alterlab.io/glossary/javascript-rendering)[Web CrawlerA program that automatically navigates websites by following hyperlinks, systematically indexing pages across an entire domain.](https://alterlab.io/glossary/web-crawler)[CSS SelectorA pattern used to match HTML elements by tag, class, ID, or attribute, used in web scraping to extract specific nodes from parsed HTML.](https://alterlab.io/glossary/css-selector)[XPathXML Path Language — a query syntax for selecting nodes in HTML or XML using path expressions, more expressive than CSS selectors for complex traversals.](https://alterlab.io/glossary/xpath)

## Extract Scrapy data from any website

AlterLab returns clean, structured data from any public URL — no scraper infrastructure needed. Start free, no credit card required.

[View API docs](https://alterlab.io/docs/api/rest)

[Back to Glossary](https://alterlab.io/glossary)