protocol

Content Negotiation

Content negotiation is an HTTP mechanism where the client specifies preferred response formats (JSON, HTML, XML) via Accept headers and the server responds in the best matching format.

HTTP content negotiation lets a client declare what media types, languages, and encodings it can accept using headers like `Accept: application/json, text/html;q=0.9`. The server selects the best matching format from its available representations and returns the content with a `Content-Type` header indicating what was sent.

For scrapers, content negotiation is a powerful technique: requesting `Accept: application/json` for a URL that normally serves HTML may reveal that the site's server also exposes a JSON endpoint for the same data, returning clean structured data without HTML parsing. Many CMS platforms and e-commerce backends support this dual-format serving.

Content negotiation also applies to compression: sending `Accept-Encoding: gzip, br` causes the server to compress the response body, reducing transfer size significantly for large HTML documents. Most HTTP client libraries decompress gzip responses automatically.

Examples

import requests

# Request JSON if available; fall back to HTML
response = requests.get("https://example.com/product/123",
    headers={"Accept": "application/json, text/html;q=0.9"})
if response.headers.get("Content-Type", "").startswith("application/json"):
    data = response.json()
else:
    # Fall back to HTML parsing
    from bs4 import BeautifulSoup
    data = BeautifulSoup(response.text, "html.parser")