Pagination is the technique websites use to split large data sets across multiple pages to reduce initial page load size and server response times. Common pagination patterns include: numbered page links (`/products?page=1`, `/products?page=2`), next/previous buttons with a cursor or token parameter, infinite scroll that loads more content as the user scrolls, and load-more buttons that append items on click.

For scraping paginated content, the strategy must match the pagination mechanism. For numbered pages, enumerate page numbers up to the last page (which may be indicated in the HTML or must be discovered dynamically). For cursor-based pagination, extract the cursor value from each response and pass it in the next request. For infinite scroll and load-more buttons, a JavaScript-rendering browser session is required to trigger the DOM updates.

Pagination traps to avoid: detecting the last page to stop scraping (avoid infinite loops), handling pages with no results gracefully, dealing with sites that change page content dynamically (product addition/removal between page fetches causing items to shift between pages), and respecting rate limits when scraping hundreds of pages from a single domain.

# Scrape all pages of a paginated list for page_num in range(1, total_pages + 1): response = scrape(f"https://example.com/products?page={page_num}") items = extract(response['html']) # Check if we've reached the last page if not items: break

Pagination

Examples

Related Terms