Web Scraping with PHP — Complete Guide
From cURL to production-ready scrapers. Extract web data with PHP's built-in tools and modern libraries.
PHP has built-in HTML parsing capabilities that many developers overlook. With cURL or Guzzle for HTTP and DOMDocument for parsing, you can build effective scrapers without leaving the PHP ecosystem. This guide covers fetching pages, parsing HTML, handling common challenges, and scaling your scraping operations.
Setting Up Your PHP Scraping Environment
PHP's built-in cURL extension handles HTTP requests, and DOMDocument parses HTML natively. For a more modern HTTP client, install Guzzle via Composer.
composer require guzzlehttp/guzzleFor CSS selector support (instead of XPath), install Symfony's DomCrawler:
composer require symfony/dom-crawler symfony/css-selectorMake sure the dom, libxml, and curl PHP extensions are enabled in your php.ini.
composer require guzzlehttp/guzzle
composer require symfony/dom-crawler symfony/css-selectorFetching a Web Page with PHP
Use Guzzle for a clean, modern HTTP client. Always set a timeout and user-agent header. Guzzle throws exceptions on HTTP errors by default, which makes error handling straightforward.
<?php
require 'vendor/autoload.php';
use GuzzleHttp\Client;
$client = new Client([
'timeout' => 10,
'headers' => [
'User-Agent' => 'Mozilla/5.0 (compatible; DataBot/1.0)',
'Accept' => 'text/html',
],
]);
$response = $client->get('https://example.com/products');
$html = (string) $response->getBody();
echo "Fetched " . strlen($html) . " bytes\n";Parsing HTML with DOMDocument
PHP's built-in DOMDocument class parses HTML and provides XPath queries. Suppress HTML5 parsing warnings with libxml_use_internal_errors. DOMXPath lets you query the document with XPath expressions.
<?php
libxml_use_internal_errors(true);
$doc = new DOMDocument();
$doc->loadHTML($html);
libxml_clear_errors();
$xpath = new DOMXPath($doc);
$cards = $xpath->query('//div[contains(@class, "product-card")]');
foreach ($cards as $card) {
$title = $xpath->query('.//h2[contains(@class, "product-title")]', $card);
$price = $xpath->query('.//span[contains(@class, "price")]', $card);
echo ($title->length ? $title->item(0)->textContent : 'N/A')
. ': '
. ($price->length ? $price->item(0)->textContent : 'N/A')
. "\n";
}Using CSS Selectors with DomCrawler
Symfony's DomCrawler provides jQuery-like CSS selectors — much more readable than XPath for most scraping tasks. Use filter() with CSS selectors and text() or attr() to extract data.
<?php
use Symfony\Component\DomCrawler\Crawler;
$crawler = new Crawler($html);
$products = $crawler->filter('div.product-card')->each(function (Crawler $node) {
return [
'title' => $node->filter('h2.product-title')->count()
? trim($node->filter('h2.product-title')->text())
: '',
'price' => $node->filter('span.price')->count()
? trim($node->filter('span.price')->text())
: '',
'url' => $node->filter('a')->count()
? $node->filter('a')->attr('href')
: '',
];
});
// Save to JSON
file_put_contents('products.json', json_encode($products, JSON_PRETTY_PRINT));Concurrent Scraping with Guzzle Promises
Guzzle supports async requests via Promises. Use Pool to send concurrent requests with a configurable concurrency limit. This is significantly faster than sequential fetching for large URL lists.
<?php
use GuzzleHttp\Client;
use GuzzleHttp\Pool;
use GuzzleHttp\Psr7\Request;
$client = new Client(['timeout' => 10]);
$urls = array_map(fn($p) => "https://example.com/page/$p", range(1, 50));
$requests = function () use ($urls) {
foreach ($urls as $url) {
yield new Request('GET', $url, [
'User-Agent' => 'Mozilla/5.0 (compatible; DataBot/1.0)',
]);
}
};
$results = [];
$pool = new Pool($client, $requests(), [
'concurrency' => 10,
'fulfilled' => function ($response, $index) use (&$results) {
$results[$index] = (string) $response->getBody();
},
'rejected' => function ($reason, $index) {
echo "Failed request $index: {$reason->getMessage()}\n";
},
]);
$pool->promise()->wait();
echo "Fetched " . count($results) . " pages\n";Handling JavaScript-Rendered Pages
PHP's HTTP clients fetch raw HTML — they do not execute JavaScript. For pages that render content dynamically, you need a rendering solution.
Options: use a headless browser library like chrome-php/chrome (PHP bindings for Chrome DevTools Protocol), or use a cloud rendering API. Running browsers from PHP adds significant complexity and memory usage.
AlterLab's API handles rendering server-side. Send a POST request with render_js: true and receive fully rendered HTML — no browser dependencies required.
Common Pitfalls and How to Avoid Them
The most common issues in PHP web scraping:
**1. HTML5 parsing warnings** — DOMDocument generates warnings on modern HTML5. Always use libxml_use_internal_errors(true) before loadHTML().
**2. Character encoding** — Use mb_convert_encoding() to convert non-UTF-8 pages before parsing. DOMDocument expects UTF-8 input.
**3. Memory limits** — PHP's default memory limit (128MB) may be too low for large pages. Set ini_set('memory_limit', '512M') for scraping scripts.
**4. Missing error handling** — Guzzle throws GuzzleException on errors. Always wrap requests in try/catch blocks.
**5. Session cookies** — Use Guzzle's CookieJar to persist cookies across requests for sites that require authentication.
Complete PHP Scraper — Guzzle + DomCrawler
Complete working scraper with concurrency, error handling, and CSV output.
<?php
require 'vendor/autoload.php';
use GuzzleHttp\Client;
use Symfony\Component\DomCrawler\Crawler;
$client = new Client([
'timeout' => 10,
'headers' => [
'User-Agent' => 'Mozilla/5.0 (compatible; DataBot/1.0)',
'Accept' => 'text/html,application/xhtml+xml',
'Accept-Language' => 'en-US,en;q=0.9',
],
]);
function fetchPage(Client $client, string $url): string {
$response = $client->get($url);
return (string) $response->getBody();
}
function parseArticles(string $html): array {
$crawler = new Crawler($html);
return $crawler->filter('article.post')->each(function (Crawler $node) {
return [
'title' => $node->filter('h2')->count()
? trim($node->filter('h2')->text()) : '',
'url' => $node->filter('a')->count()
? $node->filter('a')->attr('href') : '',
'date' => $node->filter('time')->count()
? $node->filter('time')->attr('datetime') : '',
];
});
}
$allArticles = [];
for ($page = 1; $page <= 10; $page++) {
$url = "https://example.com/articles?page=$page";
echo "Scraping page $page: $url\n";
try {
$html = fetchPage($client, $url);
$articles = parseArticles($html);
if (empty($articles)) {
echo "No more results\n";
break;
}
$allArticles = array_merge($allArticles, $articles);
usleep(500000); // 500ms delay
} catch (\Exception $e) {
echo "Error on page $page: {$e->getMessage()}\n";
break;
}
}
$fp = fopen('output.csv', 'w');
fputcsv($fp, ['title', 'url', 'date']);
foreach ($allArticles as $article) {
fputcsv($fp, $article);
}
fclose($fp);
echo "Saved " . count($allArticles) . " articles\n";Or Skip the Complexity — Use AlterLab
AlterLab handles JavaScript rendering, IP rotation, and website compatibility automatically. One POST request returns rendered HTML — no browser management, no proxy configuration. Starts at $0.0002/request with 5,000 free to start.
<?php
require 'vendor/autoload.php';
use GuzzleHttp\Client;
use Symfony\Component\DomCrawler\Crawler;
$apiKey = 'YOUR_API_KEY'; // Get free at alterlab.io
$client = new Client(['timeout' => 30]);
function scrapeWithAlterLab(Client $client, string $apiKey, string $url, bool $renderJs = false): string {
$response = $client->post('https://api.alterlab.io/api/v1/scrape', [
'headers' => [
'X-API-Key' => $apiKey,
'Content-Type' => 'application/json',
],
'json' => [
'url' => $url,
'render_js' => $renderJs,
],
]);
$result = json_decode((string) $response->getBody(), true);
return $result['html'] ?? '';
}
$html = scrapeWithAlterLab($client, $apiKey, 'https://example.com/products', true);
$crawler = new Crawler($html);
$products = $crawler->filter('div.product-card')->each(function (Crawler $node) {
return [
'title' => $node->filter('h2')->count() ? trim($node->filter('h2')->text()) : '',
'price' => $node->filter('.price')->count() ? trim($node->filter('.price')->text()) : '',
];
});
echo json_encode($products, JSON_PRETTY_PRINT) . "\n";Choosing Your Approach
cURL + DOMDocument
Pros
- +Built into PHP — no external dependencies
- +XPath queries are powerful
- +Low memory usage
Cons
- −No JavaScript execution
- −Verbose XPath syntax
- −HTML5 parsing warnings
Guzzle + DomCrawler
Pros
- +Clean, modern API
- +CSS selectors via DomCrawler
- +Async support with Promises
Cons
- −No JavaScript execution
- −Requires Composer dependencies
- −Less performant than cURL for simple requests
AlterLab API
Pros
- +Handles static, JavaScript, and challenge pages
- +No browser management
- +Automatic IP rotation
- +5-tier auto-escalation
- +From $0.0002/request
Cons
- −Per-request cost
- −Requires network access
Frequently Asked Questions
More PHP & Scraping Resources
Web Scraping with Python
Complete Python scraping guide: requests, BeautifulSoup, async patterns from beginner to production.
Web Scraping with Node.js
Node.js scraping guide: Axios, Cheerio, and TypeScript patterns.
JavaScript Rendering API
Render SPAs and dynamic content with headless Chromium — no browser management.
Pricing
From $0.0002/request. No subscriptions. Balance never expires.
Your first scrape.
Sixty seconds.
$1 free credit — up to 5,000 scrapes. No credit card.
Just a POST request.
No credit card required · $1 free credit, up to 5,000 scrapes · Balance never expires