```yaml
product: AlterLab
title: "How to Handle Infinite Scroll Web Scraping | AlterLab | AlterLab"
comparison_context: "AlterLab is an alternative to Firecrawl, ScrapingBee, and Bright Data."
source_url: https://alterlab.io/how-to/handle-infinite-scroll
```

Advanced4 steps

# How to Handle Infinite Scroll When Scraping

Infinite scroll pages load new content dynamically as the user scrolls down — meaning a standard HTTP fetch only returns the initially visible content. Collecting the full dataset requires either simulating scroll actions or intercepting the underlying data requests.

## Step-by-Step Guide

1

### Check for a hidden API endpoint first

Open browser DevTools → Network tab and filter for XHR/Fetch requests while scrolling. Many sites load infinite scroll data from a clean JSON API endpoint — fetching that directly is faster than scraping the HTML.

2

### Use JavaScript rendering with scroll simulation

If no clean API endpoint exists, use AlterLab with render_js enabled. Some targets support a scroll parameter that triggers additional content to load.

3

### Extract a cursor or offset from each response

Infinite scroll content typically includes a cursor, next_cursor, or offset field in either the HTML data attributes or the underlying API response. Extract this to construct the next request.

4

### Loop until no more content is returned

Continue fetching with the updated cursor until the response returns an empty results list or no next_cursor value.

## Code Example

Python

```
import requests

def scrape_paginated_api(base_url: str, api_key: str) -> list[dict]:
    """For sites with a discoverable JSON API endpoint."""
    results = []
    cursor = None

    while True:
        params = {"limit": 20}
        if cursor:
            params["cursor"] = cursor

        response = requests.post(
            "https://alterlab.io/api/v1/scrape",
            headers={"X-API-Key": api_key, "Content-Type": "application/json"},
            json={"url": f"{base_url}?{'&'.join(f'{k}={v}' for k,v in params.items())}"},
        )
        data = response.json()
        items = data.get("items", [])
        results.extend(items)
        cursor = data.get("next_cursor")
        if not cursor or not items:
            break

    return results
```

Replace `YOUR_API_KEY` with your key from the . No credit card required.

## Try this yourself with AlterLab

Run this tutorial on live websites with AlterLab's API. New accounts get up to 5,000 free requests — no credit card required.

[View API docs](https://alterlab.io/docs/api/rest)

## Frequently Asked Questions

### What if the infinite scroll API requires authentication?

If the underlying API requires auth headers, you will need to capture session tokens from a logged-in browser session and include them in your API requests. This is more complex and requires maintaining active session state.

### Responsible Use

AlterLab is designed for extracting publicly available data. Always review the terms of service for any website you access, respect `robots.txt` directives, and ensure your use case complies with applicable laws in your jurisdiction.

### More tutorials

Browse all [how-to guides](https://alterlab.io/how-to) for web scraping — from beginner extractions to advanced multi-page pipelines.

## Your first scrape. Sixty seconds.

Up to 5,000 free requests — about 5,000 Light, 500 Protected or 250 Dynamic requests. No credit card. 
Just a POST request.

terminal

curl -X POST https://api.alterlab.io/v1/scrape \

-H "X-API-Key: YOUR_KEY" \

-H "Content-Type: application/json" \

-d '{"url": "https://example.com", "formats": ["markdown"]}'

[Start building free](https://alterlab.io/signin)

No credit card required · Up to 5,000 free requests (about 5,000 Light, 500 Protected or 250 Dynamic requests) · Balance never expires