Advanced4 steps

How to Handle Infinite Scroll When Scraping

Infinite scroll pages load new content dynamically as the user scrolls down — meaning a standard HTTP fetch only returns the initially visible content. Collecting the full dataset requires either simulating scroll actions or intercepting the underlying data requests.

Step-by-Step Guide

1

Check for a hidden API endpoint first

Open browser DevTools → Network tab and filter for XHR/Fetch requests while scrolling. Many sites load infinite scroll data from a clean JSON API endpoint — fetching that directly is faster than scraping the HTML.

2

Use JavaScript rendering with scroll simulation

If no clean API endpoint exists, use AlterLab with render_js enabled. Some targets support a scroll parameter that triggers additional content to load.

3

Extract a cursor or offset from each response

Infinite scroll content typically includes a cursor, next_cursor, or offset field in either the HTML data attributes or the underlying API response. Extract this to construct the next request.

4

Loop until no more content is returned

Continue fetching with the updated cursor until the response returns an empty results list or no next_cursor value.

Code Example

Python
import requests

def scrape_paginated_api(base_url: str, api_key: str) -> list[dict]:
    """For sites with a discoverable JSON API endpoint."""
    results = []
    cursor = None

    while True:
        params = {"limit": 20}
        if cursor:
            params["cursor"] = cursor

        response = requests.post(
            "https://alterlab.io/api/v1/scrape",
            headers={"X-API-Key": api_key, "Content-Type": "application/json"},
            json={"url": f"{base_url}?{'&'.join(f'{k}={v}' for k,v in params.items())}"},
        )
        data = response.json()
        items = data.get("items", [])
        results.extend(items)
        cursor = data.get("next_cursor")
        if not cursor or not items:
            break

    return results

Replace YOUR_API_KEY with your key from the . No credit card required.

Try this yourself with AlterLab

Run this tutorial on live websites with AlterLab's API. Free tier includes 5,000 requests — no credit card required.

View API docs

Frequently Asked Questions

What if the infinite scroll API requires authentication?

If the underlying API requires auth headers, you will need to capture session tokens from a logged-in browser session and include them in your API requests. This is more complex and requires maintaining active session state.

Responsible Use

AlterLab is designed for extracting publicly available data. Always review the terms of service for any website you access, respect robots.txt directives, and ensure your use case complies with applicable laws in your jurisdiction.

More tutorials

Browse all how-to guides for web scraping — from beginner extractions to advanced multi-page pipelines.

Your first scrape.
Sixty seconds.

$1 free credit — up to 5,000 scrapes. No credit card.Just a POST request.

terminal
curl -X POST https://api.alterlab.io/v1/scrape \
-H "X-API-Key: YOUR_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com", "formats": ["markdown"]}'

No credit card required · $1 free credit, up to 5,000 scrapes · Balance never expires