general

Infinite Scroll

Infinite scroll is a UX pattern where additional content loads automatically as the user scrolls toward the bottom of the page, replacing traditional pagination.

Social media feeds, image galleries, and news aggregators commonly use infinite scroll to create a seamless browsing experience. From a scraping perspective, the page's initial HTML contains only the first batch of items; subsequent batches are loaded via JavaScript-triggered XHR or Fetch API calls as the viewport approaches the document's bottom edge.

Scraping infinite-scroll pages requires either a headless browser that can trigger scroll events and wait for network responses to settle, or direct interception of the underlying API calls that power the lazy loading. The latter is more efficient — once the API endpoint and parameters are identified, content can be fetched programmatically without rendering overhead.

AlterLab's network interception capability captures these background API calls, allowing callers to retrieve all paginated content without simulating scroll gestures. When the underlying API is not publicly accessible, the browser-based tier scrolls the page programmatically and captures DOM updates.

Examples

// Playwright: scroll to bottom to trigger infinite load
async function scrapeInfiniteScroll(page) {
  let prevHeight = 0;
  while (true) {
    await page.evaluate(() => window.scrollTo(0, document.body.scrollHeight));
    await page.waitForTimeout(1500);
    const newHeight = await page.evaluate(() => document.body.scrollHeight);
    if (newHeight === prevHeight) break;
    prevHeight = newHeight;
  }
}

Related Terms

Pagination
The division of large data sets across multiple pages, requiring scrapers to follow next-page links or increment page parameters to retrieve complete data.
Dynamic Content
Page content generated client-side by JavaScript after the initial HTML is served, requiring a browser engine to render.
JavaScript Rendering
The process of executing a page's JavaScript in a browser engine before extracting content, required for React, Vue, and Angular SPAs.
Network Interception
Capturing XHR/fetch API responses made by a page's JavaScript during browser rendering to extract structured data directly from network calls.
Headless Browser
A web browser that runs without a graphical user interface, executing JavaScript and rendering pages exactly as a real browser would.

Extract Infinite Scroll data from any website

AlterLab returns clean, structured data from any public URL — no scraper infrastructure needed. Start free, no credit card required.

View API docs
Back to Glossary

Your first scrape.
Sixty seconds.

$1 free balance. No credit card. No SDK.Just a POST request.

terminal
curl -X POST https://api.alterlab.io/v1/scrape \
-H "X-API-Key: YOUR_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com", "formats": ["markdown"]}'

No credit card required · Up to 5,000 free scrapes · Balance never expires

    Infinite Scroll — Web Scraping Glossary | AlterLab