Scraped Data Validation: A Practical Guide
Best Practices

Scraped Data Validation: A Practical Guide

Build reliable data pipelines with scraped data validation: schemas, type checks, freshness tests, anomaly detection, and quarantine workflows in Python.

H
Herald Blog Service
11 min read

AlterLab handles this automatically — scrape any URL with one API call. No infrastructure required.

Try it free

TL;DR

Scraped data validation checks whether fetched content is complete, correctly typed, semantically plausible, and fresh enough for downstream use. Validate at the retrieval, normalized-record, and dataset boundaries, then publish only records that pass the contract while sending failures to quarantine.

What is scraped data validation?

Scraped data validation is the process of comparing fetched content with explicit quality rules before storing or distributing it. A successful HTTP response does not prove that the page contains usable data. The response may contain an empty template, an access notice, stale content, or a layout change that breaks extraction.

A reliable validation pipeline separates five concerns:

  1. Transport validation checks whether the request returned usable content.
  2. Content validation checks whether the response contains the expected page or document.
  3. Schema validation checks fields, types, formats, and required values.
  4. Semantic validation checks whether values make sense together.
  5. Dataset validation checks duplicates, completeness, freshness, and drift across records.

Treat validation as a publish gate. Records that fail should remain available for inspection and replay rather than disappearing inside a parser.

After understanding the validation boundary, test the retrieval step with a permitted public page:

Try it yourself

Try scraping this page with AlterLab

Validation does not change collection permissions. Collect only publicly accessible content that you are allowed to access, respect site policies and applicable directives, and do not treat rendering or anti-bot handling as a way around login walls, paywalls, or access controls.

Validate data at three boundaries

1. Validate the retrieved response

Start with checks that do not depend on the page parser:

  • Confirm the response completed without a transport error.
  • Check that the content type matches the expected document.
  • Reject empty bodies and unexpectedly small responses.
  • Record the final URL when redirects are relevant.
  • Detect obvious access notices, challenge pages, or generic error documents.
  • Record the collection timestamp and source URL.

A response can pass the transport layer and still fail content validation. For example, an HTML response may contain a generic error page where the pipeline expected a product listing, article, or documentation page.

Avoid using a single string check as the entire solution. Access notices can vary by locale, template, and rendering mode. Combine markers with structural checks, such as the presence of required headings, repeated records, or expected metadata.

2. Validate the normalized record

Raw HTML is difficult to validate consistently. Normalize values before applying business rules:

  • Trim whitespace and normalize Unicode.
  • Convert dates to timezone-aware timestamps.
  • Convert numeric text to a consistent decimal representation.
  • Normalize URLs before deduplication.
  • Standardize empty values as null or a documented equivalent.
  • Preserve the original value when normalization may lose information.

A normalized record should have a stable contract. For a generic listing dataset, that contract might include a non-empty title, an HTTP or HTTPS URL, a non-negative price, a three-letter currency code, and a collection timestamp.

Do not silently coerce malformed values into valid-looking defaults. Turning an invalid price into zero makes the record harder to detect and can corrupt downstream calculations.

3. Validate the dataset

Record-level checks cannot detect every failure. Add checks across the collection:

  • Compare the number of records with recent collection baselines.
  • Detect duplicate canonical URLs or stable identifiers.
  • Check that pagination does not repeat the same page.
  • Measure freshness lag between collection time and source update time.
  • Compare distributions for fields such as currency, category, or status.
  • Detect sudden increases in null values.
  • Compare field-level changes rather than relying only on full-page hashes.

Dataset checks should produce alerts and quarantine decisions, not just logs. A parser can return valid-looking records while losing half the source items. A count check is often the first signal that this happened.

For larger collection jobs, the batch scraping guide provides a related pattern for handling multiple URLs and collection runs.

Define a versioned data contract

A data contract describes what the pipeline promises to produce. Write it before implementing the parser.

A useful contract documents:

Contract areaExample ruleFailure action
Required fieldstitle, url, and collected_at must existQuarantine
Typesprice is a decimal, collected_at is an ISO timestampQuarantine
FormatsURLs use HTTP or HTTPS, currency uses uppercase ISO-style codesQuarantine
Semantic rangePrice cannot be negativeQuarantine
Relationshipsmin_value cannot exceed max_valueQuarantine
FreshnessCollection age stays below the dataset thresholdHold or alert
CompletenessRecord count stays within the expected rangeAlert or quarantine

Version the contract when fields, meanings, or accepted formats change. A version lets you distinguish a source change from an intentional schema update.

Keep raw and normalized data separate. The raw response supports debugging and replay. The normalized record supports validation and downstream use. Storing only the final record makes it difficult to determine whether a failure came from retrieval, parsing, normalization, or source drift.

Retrieve content before validating it

The Python SDK and API request below perform the same retrieval operation against a public example page. The validation layer runs after the response is returned.

The Python SDK is useful when retrieval and validation live in the same worker process:

Python
from alterlab import AlterLabSync

BLOCK_MARKERS = ("access denied", "enable javascript to continue")

def validate_content(content: str) -> list[str]:
    errors = []

    if not isinstance(content, str) or not content.strip():
        errors.append("empty content")
        return errors

    if len(content) > 5_000_000:
        errors.append("unexpectedly large content")

    first_chunk = content[:2000].lower()
    if any(marker in first_chunk for marker in BLOCK_MARKERS):
        errors.append("possible access notice or incomplete document")

    return errors

client = AlterLabSync(api_key="YOUR_API_KEY")
result = client.scrape("https://example.com")
errors = validate_content(result["content"])

if errors:
    raise ValueError("; ".join(errors))

print("content passed validation")

The marker list is only an example. Maintain source-specific markers carefully because a legitimate page can mention terms such as “access denied” in its content. Combine content markers with structural and dataset checks.

The equivalent cURL request is:

Bash
curl -X POST https://api.alterlab.io/api/v1/scrape \
  -H "X-API-Key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'

For production use, parse the API response according to the API docs, then pass the retrieved content into the same validation layer. Keep retrieval and validation separate so you can test the validator with saved fixtures without making a network request.

Validate normalized records with typed rules

Once the parser creates a normalized object, apply field-level and semantic rules. The following validator uses only Python's standard library and expects an internal record contract.

Python
import re
from datetime import datetime
from decimal import Decimal, InvalidOperation
from urllib.parse import urlparse

CURRENCY_PATTERN = re.compile(r"^[A-Z]{3}$")

def validate_record(record: dict) -> list[str]:
    errors = []

    for field in ("title", "url", "price", "currency", "collected_at"):
        value = record.get(field)
        if value is None or value == "":
            errors.append(f"missing required field: {field}")

    title = record.get("title")
    if not isinstance(title, str) or not title.strip():
        errors.append("title must be a non-empty string")

    parsed_url = urlparse(str(record.get("url", "")))
    if parsed_url.scheme not in {"http", "https"} or not parsed_url.netloc:
        errors.append("url must be an absolute HTTP or HTTPS URL")

    try:
        price = Decimal(str(record["price"]))
        if price < 0:
            errors.append("price cannot be negative")
    except (KeyError, InvalidOperation, TypeError):
        errors.append("price must be a valid decimal")

    currency = record.get("currency")
    if not isinstance(currency, str) or not CURRENCY_PATTERN.fullmatch(currency):
        errors.append("currency must be a three-letter uppercase code")

    try:
        timestamp = datetime.fromisoformat(
            str(record["collected_at"]).replace("Z", "+00:00")
        )
        if timestamp.tzinfo is None:
            errors.append("collected_at must include a timezone")
    except (KeyError, TypeError, ValueError):
        errors.append("collected_at must be an ISO timestamp")

    return errors

This validator makes failures explicit. It does not assume that a missing value is zero, that a malformed URL can be repaired safely, or that a timestamp without a timezone is unambiguous.

Use different outcomes for different failure types:

  • Reject records that violate required types or formats.
  • Quarantine records that may be valid but need review.
  • Retry temporary transport failures.
  • Alert on dataset-wide anomalies.
  • Accept only records that meet the contract.

A failed record should include the error list. Multiple errors often point to one root cause, such as a layout change that moved several fields at once.

Detect silent extraction failures

The most expensive failures are often the ones that produce valid output with incorrect meaning. Add checks for these cases.

Empty or partial templates

JavaScript-rendered pages may return a shell before content appears. If the parser expects repeated records, require a minimum structural signal rather than accepting any non-empty HTML.

Repeated boilerplate

A selector can match the same navigation label or placeholder on every page. Track field uniqueness and compare values across records. A sudden collapse to one repeated value indicates a parser problem.

Wrong locale or unit

Dates, decimal separators, currencies, weights, and distances can vary by locale. Store the source locale when known, then normalize values with an explicit rule. Do not infer units from a number without documenting the assumption.

Broken pagination

Track page URLs and stable record identifiers. If the same page or identifier appears repeatedly, stop the batch and quarantine the affected results. A duplicate check is safer than silently appending repeated records.

Layout drift

A full-document hash can detect change, but it cannot tell you whether the change matters. Compare field-level extraction results and required markers. A layout-only change may leave the data intact, while a small template change may remove a critical field.

For permitted public pages that require JavaScript rendering, review the anti-bot handling guidance to understand the retrieval path. Keep the data contract independent from the retrieval method. A browser-rendered response still needs the same schema, semantic, and freshness checks.

Use freshness and completeness rules

Freshness is a data-quality property, not only a scheduling property. A job can run successfully while returning content that is too old for its use case.

Store at least:

  • collected_at, the time your worker received the content.
  • source_updated_at, when the source exposes a reliable update time.
  • source_url, after normalization.
  • schema_version, for the record contract.
  • parser_version, for the extraction logic.

Define freshness thresholds per dataset. A monitoring dataset may require frequent updates, while a reference dataset may tolerate older records. The threshold belongs in configuration, not hidden inside parser code.

Completeness also needs a baseline. Track record counts, required-field coverage, and unique identifier counts over time. Use rolling baselines where source volume changes naturally. Alert on material deviations instead of hard-coding one expected count for every run.

For recurring collection, see scheduling recurring scrapes with Python and cron. Scheduling and validation should remain separate so a successful run does not automatically imply that the resulting data is publishable.

Quarantine failures for replay

A quarantine path should preserve enough information to reproduce the decision. Store:

  • Raw response or a durable reference to it.
  • Normalized record, if normalization completed.
  • Validation errors and rule names.
  • Source URL and collection timestamp.
  • Parser and schema versions.
  • Batch or run identifier.
  • A content hash for deduplication and comparison.

Separate retry queues from quarantine queues. Retry a temporary network or service failure according to your retry policy. Do not repeatedly retry a deterministic schema failure. Send it to quarantine and investigate the rule or source change.

Build a replay command or job that can run quarantined inputs through a newer parser without fetching the source again. This reduces load on public sites and gives engineers a controlled way to test fixes.

Monitor validation as a pipeline health signal

Track validation metrics by source, parser version, schema version, and collection run:

  • Accepted record count.
  • Quarantined record count.
  • Failure rate by rule.
  • Missing-field rate.
  • Duplicate rate.
  • Freshness lag.
  • Record-count deviation.
  • Time spent in each pipeline stage.

Metrics need context. A high quarantine rate may indicate source drift, a parser deployment, a new locale, or a change in the data contract. Include representative failed records in an internal review view, with sensitive fields handled according to your retention policy.

Log enough detail to diagnose a failure without logging secrets. API keys, session tokens, and authorization headers should never enter raw logs or quarantine records.

Test validators with fixtures

Use saved fixtures for normal pages, empty responses, partial templates, locale variations, missing fields, and known layout changes. Unit-test individual validators, then run integration tests through the full normalization and validation pipeline.

Useful test cases include:

  • A valid record with all required fields.
  • A record with a missing title.
  • A negative numeric value.
  • A malformed URL.
  • A timestamp without timezone information.
  • A duplicate identifier in one batch.
  • A response containing an access notice instead of page content.
  • A valid source page with a changed layout.

Keep fixtures representative but scrub secrets and personal data. When a production failure occurs, add a sanitized fixture before changing the parser. This turns an incident into a regression test.

For examples of how scraped data supports different application workflows, review the AlterLab use cases.

Takeaway

Reliable scraped data comes from explicit contracts and observable validation, not from retrieval alone. Normalize at the boundary, check structure and meaning, measure freshness and completeness, and quarantine failures with enough context to replay them safely.

For retrieval pipelines that feed search and generation systems, see building a RAG pipeline with live web data.

Share

Was this article helpful?

Frequently Asked Questions

Data validation checks scraped content against transport, schema, semantic, freshness, and completeness rules before storage. Invalid records go to quarantine instead of entering downstream datasets.
Define a versioned data contract, normalize values, validate required fields and relationships, and monitor drift over time. Track acceptance, quarantine, duplicate, and freshness metrics.
Store the raw payload, normalized record, validation errors, and collection metadata in a quarantine path. Retry transport failures separately, then reprocess quarantined records after the parser or contract changes.