```yaml
product: AlterLab
title: "How to Scrape Paginated Results — All Pages | AlterLab | AlterLab"
comparison_context: "AlterLab is an alternative to Firecrawl, ScrapingBee, and Bright Data."
source_url: https://alterlab.io/how-to/scrape-paginated-results
```

Intermediate4 steps

# How to Scrape Paginated Results

Data spread across multiple pages requires iterating through each page systematically. Pagination patterns vary widely — some use page numbers in the URL, others use query parameters, cursors, or infinite scroll — each requiring a different approach.

## Step-by-Step Guide

1

### Identify the pagination pattern

Inspect the URL as you move between pages. Look for page=N, offset=N, or cursor tokens in the URL or in API responses embedded in the page.

2

### Fetch the first page and extract data

Send the first page URL to AlterLab and parse the returned HTML to extract both your target data and the next-page URL or pagination controls.

3

### Build a pagination loop

Extract the next page URL from the current page's pagination controls. Continue fetching and extracting until no next-page link is found.

4

### Handle rate limits gracefully

Add a short delay between requests. AlterLab manages proxy rotation, but respecting the target site's request cadence improves reliability.

## Code Example

Python

```
import requests
from bs4 import BeautifulSoup

def scrape_all_pages(start_url: str, api_key: str, next_selector: str):
    url = start_url
    results = []

    while url:
        response = requests.post(
            "https://alterlab.io/api/v1/scrape",
            headers={"X-API-Key": api_key, "Content-Type": "application/json"},
            json={"url": url},
        )
        html = response.json().get("html", "")
        soup = BeautifulSoup(html, "html.parser")

        # Extract your data here
        results.extend(soup.select(".result-item"))

        # Find next page link
        next_link = soup.select_one(next_selector)
        url = next_link["href"] if next_link and next_link.get("href") else None

    return results
```

Replace `YOUR_API_KEY` with your key from the . No credit card required.

## Try this yourself with AlterLab

Run this tutorial on live websites with AlterLab's API. New accounts get up to 5,000 free requests — no credit card required.

[View API docs](https://alterlab.io/docs/api/rest)

## Frequently Asked Questions

### How do I handle infinite scroll pagination?

For infinite scroll, use `render_js: true` and the `wait_for` parameter to wait for new content to load. Then extract a cursor or offset value from the rendered page to construct the next request.

### What if the pagination uses POST requests with a cursor token?

Many sites expose their data through embedded API calls. Use your browser's Network tab to find the underlying XHR/fetch requests — you may be able to call those directly rather than scraping the rendered HTML.

### Responsible Use

AlterLab is designed for extracting publicly available data. Always review the terms of service for any website you access, respect `robots.txt` directives, and ensure your use case complies with applicable laws in your jurisdiction.

### More tutorials

Browse all [how-to guides](https://alterlab.io/how-to) for web scraping — from beginner extractions to advanced multi-page pipelines.

## Your first scrape. Sixty seconds.

Up to 5,000 free requests — about 5,000 Light, 500 Protected or 250 Dynamic requests. No credit card. 
Just a POST request.

terminal

curl -X POST https://api.alterlab.io/v1/scrape \

-H "X-API-Key: YOUR_KEY" \

-H "Content-Type: application/json" \

-d '{"url": "https://example.com", "formats": ["markdown"]}'

[Start building free](https://alterlab.io/signin)

No credit card required · Up to 5,000 free requests (about 5,000 Light, 500 Protected or 250 Dynamic requests) · Balance never expires