Intermediate4 steps

How to Scrape Paginated Results

Data spread across multiple pages requires iterating through each page systematically. Pagination patterns vary widely — some use page numbers in the URL, others use query parameters, cursors, or infinite scroll — each requiring a different approach.

Step-by-Step Guide

1

Identify the pagination pattern

Inspect the URL as you move between pages. Look for page=N, offset=N, or cursor tokens in the URL or in API responses embedded in the page.

2

Fetch the first page and extract data

Send the first page URL to AlterLab and parse the returned HTML to extract both your target data and the next-page URL or pagination controls.

3

Build a pagination loop

Extract the next page URL from the current page's pagination controls. Continue fetching and extracting until no next-page link is found.

4

Handle rate limits gracefully

Add a short delay between requests. AlterLab manages proxy rotation, but respecting the target site's request cadence improves reliability.

Code Example

Python
import requests
from bs4 import BeautifulSoup

def scrape_all_pages(start_url: str, api_key: str, next_selector: str):
    url = start_url
    results = []

    while url:
        response = requests.post(
            "https://alterlab.io/api/v1/scrape",
            headers={"X-API-Key": api_key, "Content-Type": "application/json"},
            json={"url": url},
        )
        html = response.json().get("html", "")
        soup = BeautifulSoup(html, "html.parser")

        # Extract your data here
        results.extend(soup.select(".result-item"))

        # Find next page link
        next_link = soup.select_one(next_selector)
        url = next_link["href"] if next_link and next_link.get("href") else None

    return results

Replace YOUR_API_KEY with your key from the . No credit card required.

Try this yourself with AlterLab

Run this tutorial on live websites with AlterLab's API. Free tier includes 5,000 requests — no credit card required.

View API docs

Frequently Asked Questions

How do I handle infinite scroll pagination?

For infinite scroll, use `render_js: true` and the `wait_for` parameter to wait for new content to load. Then extract a cursor or offset value from the rendered page to construct the next request.

What if the pagination uses POST requests with a cursor token?

Many sites expose their data through embedded API calls. Use your browser's Network tab to find the underlying XHR/fetch requests — you may be able to call those directly rather than scraping the rendered HTML.

Responsible Use

AlterLab is designed for extracting publicly available data. Always review the terms of service for any website you access, respect robots.txt directives, and ensure your use case complies with applicable laws in your jurisdiction.

More tutorials

Browse all how-to guides for web scraping — from beginner extractions to advanced multi-page pipelines.

Your first scrape.
Sixty seconds.

$1 free credit — up to 5,000 scrapes. No credit card.Just a POST request.

terminal
curl -X POST https://api.alterlab.io/v1/scrape \
-H "X-API-Key: YOUR_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com", "formats": ["markdown"]}'

No credit card required · $1 free credit, up to 5,000 scrapes · Balance never expires