How to Handle Infinite Scroll When Scraping
Infinite scroll pages load new content dynamically as the user scrolls down — meaning a standard HTTP fetch only returns the initially visible content. Collecting the full dataset requires either simulating scroll actions or intercepting the underlying data requests.
Step-by-Step Guide
Check for a hidden API endpoint first
Open browser DevTools → Network tab and filter for XHR/Fetch requests while scrolling. Many sites load infinite scroll data from a clean JSON API endpoint — fetching that directly is faster than scraping the HTML.
Use JavaScript rendering with scroll simulation
If no clean API endpoint exists, use AlterLab with render_js enabled. Some targets support a scroll parameter that triggers additional content to load.
Extract a cursor or offset from each response
Infinite scroll content typically includes a cursor, next_cursor, or offset field in either the HTML data attributes or the underlying API response. Extract this to construct the next request.
Loop until no more content is returned
Continue fetching with the updated cursor until the response returns an empty results list or no next_cursor value.
Code Example
import requests
def scrape_paginated_api(base_url: str, api_key: str) -> list[dict]:
"""For sites with a discoverable JSON API endpoint."""
results = []
cursor = None
while True:
params = {"limit": 20}
if cursor:
params["cursor"] = cursor
response = requests.post(
"https://alterlab.io/api/v1/scrape",
headers={"X-API-Key": api_key, "Content-Type": "application/json"},
json={"url": f"{base_url}?{'&'.join(f'{k}={v}' for k,v in params.items())}"},
)
data = response.json()
items = data.get("items", [])
results.extend(items)
cursor = data.get("next_cursor")
if not cursor or not items:
break
return resultsReplace YOUR_API_KEY with your key from the dashboard. No credit card required.
Ready to try it?
Run this tutorial on live websites with AlterLab's API. Start free — no credit card required.
Frequently Asked Questions
Responsible Use
AlterLab is designed for extracting publicly available data. Always review the terms of service for any website you access, respect robots.txt directives, and ensure your use case complies with applicable laws in your jurisdiction.
More tutorials
Browse all how-to guides for web scraping — from beginner extractions to advanced multi-page pipelines.
Your first scrape.
Sixty seconds.
$1 free balance. No credit card. No SDK.
Just a POST request.
No credit card required · Up to 5,000 free scrapes · Balance never expire