Caching is the storage of a copy of a web response so that subsequent identical requests can be served from the cache rather than regenerated by the origin server. Caches exist at multiple levels in the web stack: the origin server (application-level caching with Redis or Memcached), the CDN edge (HTTP caching based on Cache-Control and Vary headers), the browser (local cache), and proxy caches in between.
For web scraping, caching creates two distinct problems. First, cached responses may be stale — a scraper retrieving prices or inventory may receive data from a cached response that is hours or days old rather than the current live data. Cache headers (`Cache-Control: max-age=86400`, `Expires`) indicate the intended freshness window. Second, cached responses bypass some anti-bot checks — a CDN serving a cached response from its own infrastructure may not trigger the same bot detection as a live origin request.
Strategies for bypassing caches to receive live data include: adding a random query parameter (`?_t=<timestamp>`) to prevent cache key matching, using cache-busting headers (`Cache-Control: no-cache`, `Pragma: no-cache`), or submitting POST requests rather than GETs (POST responses are typically not cached). AlterLab passes cache-control headers that request fresh responses from origins.