browser

SPA Scraping

SPA scraping targets Single-Page Applications, where the HTML shell is static but all content is rendered by JavaScript after the initial page load.

A Single-Page Application (SPA) loads a minimal HTML shell on first request and uses a JavaScript framework (React, Vue, Angular, Svelte) to render all subsequent views client-side. Navigating between 'pages' within the SPA does not trigger a full server round-trip; instead, the framework updates the DOM based on client-side routing.

Scraping an SPA with a plain HTTP client retrieves only the empty HTML shell — the data of interest has not yet been rendered. A headless browser is required to wait for JavaScript to execute and the target elements to appear in the DOM. The browser must also handle client-side routing correctly, as the URL path changes via the History API rather than a server redirect.

An efficient alternative is to intercept the API calls the SPA makes to its backend. SPAs almost always fetch data from a JSON API; those API endpoints can often be called directly, bypassing the rendering layer entirely and returning structured data without any HTML parsing.

Examples

// Playwright: wait for SPA content to render
await page.goto("https://spa-example.com/products");
await page.waitForSelector(".product-card"); // wait for JS render
const products = await page.$$eval(".product-card", els =>
  els.map(el => ({ name: el.querySelector("h2").textContent }))
);