
Scraped Data Validation: A Practical Guide
Build reliable data pipelines with scraped data validation: schemas, type checks, freshness tests, anomaly detection, and quarantine workflows in Python.
AlterLab handles this automatically — scrape any URL with one API call. No infrastructure required.
Try it freeTL;DR
Scraped data validation checks whether fetched content is complete, correctly typed, semantically plausible, and fresh enough for downstream use. Validate at the retrieval, normalized-record, and dataset boundaries, then publish only records that pass the contract while sending failures to quarantine.
What is scraped data validation?
Scraped data validation is the process of comparing fetched content with explicit quality rules before storing or distributing it. A successful HTTP response does not prove that the page contains usable data. The response may contain an empty template, an access notice, stale content, or a layout change that breaks extraction.
A reliable validation pipeline separates five concerns:
- Transport validation checks whether the request returned usable content.
- Content validation checks whether the response contains the expected page or document.
- Schema validation checks fields, types, formats, and required values.
- Semantic validation checks whether values make sense together.
- Dataset validation checks duplicates, completeness, freshness, and drift across records.
Treat validation as a publish gate. Records that fail should remain available for inspection and replay rather than disappearing inside a parser.
After understanding the validation boundary, test the retrieval step with a permitted public page:
Try scraping this page with AlterLab
Validation does not change collection permissions. Collect only publicly accessible content that you are allowed to access, respect site policies and applicable directives, and do not treat rendering or anti-bot handling as a way around login walls, paywalls, or access controls.
Validate data at three boundaries
1. Validate the retrieved response
Start with checks that do not depend on the page parser:
- Confirm the response completed without a transport error.
- Check that the content type matches the expected document.
- Reject empty bodies and unexpectedly small responses.
- Record the final URL when redirects are relevant.
- Detect obvious access notices, challenge pages, or generic error documents.
- Record the collection timestamp and source URL.
A response can pass the transport layer and still fail content validation. For example, an HTML response may contain a generic error page where the pipeline expected a product listing, article, or documentation page.
Avoid using a single string check as the entire solution. Access notices can vary by locale, template, and rendering mode. Combine markers with structural checks, such as the presence of required headings, repeated records, or expected metadata.
2. Validate the normalized record
Raw HTML is difficult to validate consistently. Normalize values before applying business rules:
- Trim whitespace and normalize Unicode.
- Convert dates to timezone-aware timestamps.
- Convert numeric text to a consistent decimal representation.
- Normalize URLs before deduplication.
- Standardize empty values as
nullor a documented equivalent. - Preserve the original value when normalization may lose information.
A normalized record should have a stable contract. For a generic listing dataset, that contract might include a non-empty title, an HTTP or HTTPS URL, a non-negative price, a three-letter currency code, and a collection timestamp.
Do not silently coerce malformed values into valid-looking defaults. Turning an invalid price into zero makes the record harder to detect and can corrupt downstream calculations.
3. Validate the dataset
Record-level checks cannot detect every failure. Add checks across the collection:
- Compare the number of records with recent collection baselines.
- Detect duplicate canonical URLs or stable identifiers.
- Check that pagination does not repeat the same page.
- Measure freshness lag between collection time and source update time.
- Compare distributions for fields such as currency, category, or status.
- Detect sudden increases in null values.
- Compare field-level changes rather than relying only on full-page hashes.
Dataset checks should produce alerts and quarantine decisions, not just logs. A parser can return valid-looking records while losing half the source items. A count check is often the first signal that this happened.
For larger collection jobs, the batch scraping guide provides a related pattern for handling multiple URLs and collection runs.
Define a versioned data contract
A data contract describes what the pipeline promises to produce. Write it before implementing the parser.
A useful contract documents:
| Contract area | Example rule | Failure action |
|---|---|---|
| Required fields | title, url, and collected_at must exist | Quarantine |
| Types | price is a decimal, collected_at is an ISO timestamp | Quarantine |
| Formats | URLs use HTTP or HTTPS, currency uses uppercase ISO-style codes | Quarantine |
| Semantic range | Price cannot be negative | Quarantine |
| Relationships | min_value cannot exceed max_value | Quarantine |
| Freshness | Collection age stays below the dataset threshold | Hold or alert |
| Completeness | Record count stays within the expected range | Alert or quarantine |
Version the contract when fields, meanings, or accepted formats change. A version lets you distinguish a source change from an intentional schema update.
Keep raw and normalized data separate. The raw response supports debugging and replay. The normalized record supports validation and downstream use. Storing only the final record makes it difficult to determine whether a failure came from retrieval, parsing, normalization, or source drift.
Retrieve content before validating it
The Python SDK and API request below perform the same retrieval operation against a public example page. The validation layer runs after the response is returned.
The Python SDK is useful when retrieval and validation live in the same worker process:
from alterlab import AlterLabSync
BLOCK_MARKERS = ("access denied", "enable javascript to continue")
def validate_content(content: str) -> list[str]:
errors = []
if not isinstance(content, str) or not content.strip():
errors.append("empty content")
return errors
if len(content) > 5_000_000:
errors.append("unexpectedly large content")
first_chunk = content[:2000].lower()
if any(marker in first_chunk for marker in BLOCK_MARKERS):
errors.append("possible access notice or incomplete document")
return errors
client = AlterLabSync(api_key="YOUR_API_KEY")
result = client.scrape("https://example.com")
errors = validate_content(result["content"])
if errors:
raise ValueError("; ".join(errors))
print("content passed validation")The marker list is only an example. Maintain source-specific markers carefully because a legitimate page can mention terms such as “access denied” in its content. Combine content markers with structural and dataset checks.
The equivalent cURL request is:
curl -X POST https://api.alterlab.io/api/v1/scrape \
-H "X-API-Key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com"}'For production use, parse the API response according to the API docs, then pass the retrieved content into the same validation layer. Keep retrieval and validation separate so you can test the validator with saved fixtures without making a network request.
Validate normalized records with typed rules
Once the parser creates a normalized object, apply field-level and semantic rules. The following validator uses only Python's standard library and expects an internal record contract.
import re
from datetime import datetime
from decimal import Decimal, InvalidOperation
from urllib.parse import urlparse
CURRENCY_PATTERN = re.compile(r"^[A-Z]{3}$")
def validate_record(record: dict) -> list[str]:
errors = []
for field in ("title", "url", "price", "currency", "collected_at"):
value = record.get(field)
if value is None or value == "":
errors.append(f"missing required field: {field}")
title = record.get("title")
if not isinstance(title, str) or not title.strip():
errors.append("title must be a non-empty string")
parsed_url = urlparse(str(record.get("url", "")))
if parsed_url.scheme not in {"http", "https"} or not parsed_url.netloc:
errors.append("url must be an absolute HTTP or HTTPS URL")
try:
price = Decimal(str(record["price"]))
if price < 0:
errors.append("price cannot be negative")
except (KeyError, InvalidOperation, TypeError):
errors.append("price must be a valid decimal")
currency = record.get("currency")
if not isinstance(currency, str) or not CURRENCY_PATTERN.fullmatch(currency):
errors.append("currency must be a three-letter uppercase code")
try:
timestamp = datetime.fromisoformat(
str(record["collected_at"]).replace("Z", "+00:00")
)
if timestamp.tzinfo is None:
errors.append("collected_at must include a timezone")
except (KeyError, TypeError, ValueError):
errors.append("collected_at must be an ISO timestamp")
return errorsThis validator makes failures explicit. It does not assume that a missing value is zero, that a malformed URL can be repaired safely, or that a timestamp without a timezone is unambiguous.
Use different outcomes for different failure types:
- Reject records that violate required types or formats.
- Quarantine records that may be valid but need review.
- Retry temporary transport failures.
- Alert on dataset-wide anomalies.
- Accept only records that meet the contract.
A failed record should include the error list. Multiple errors often point to one root cause, such as a layout change that moved several fields at once.
Detect silent extraction failures
The most expensive failures are often the ones that produce valid output with incorrect meaning. Add checks for these cases.
Empty or partial templates
JavaScript-rendered pages may return a shell before content appears. If the parser expects repeated records, require a minimum structural signal rather than accepting any non-empty HTML.
Repeated boilerplate
A selector can match the same navigation label or placeholder on every page. Track field uniqueness and compare values across records. A sudden collapse to one repeated value indicates a parser problem.
Wrong locale or unit
Dates, decimal separators, currencies, weights, and distances can vary by locale. Store the source locale when known, then normalize values with an explicit rule. Do not infer units from a number without documenting the assumption.
Broken pagination
Track page URLs and stable record identifiers. If the same page or identifier appears repeatedly, stop the batch and quarantine the affected results. A duplicate check is safer than silently appending repeated records.
Layout drift
A full-document hash can detect change, but it cannot tell you whether the change matters. Compare field-level extraction results and required markers. A layout-only change may leave the data intact, while a small template change may remove a critical field.
For permitted public pages that require JavaScript rendering, review the anti-bot handling guidance to understand the retrieval path. Keep the data contract independent from the retrieval method. A browser-rendered response still needs the same schema, semantic, and freshness checks.
Use freshness and completeness rules
Freshness is a data-quality property, not only a scheduling property. A job can run successfully while returning content that is too old for its use case.
Store at least:
collected_at, the time your worker received the content.source_updated_at, when the source exposes a reliable update time.source_url, after normalization.schema_version, for the record contract.parser_version, for the extraction logic.
Define freshness thresholds per dataset. A monitoring dataset may require frequent updates, while a reference dataset may tolerate older records. The threshold belongs in configuration, not hidden inside parser code.
Completeness also needs a baseline. Track record counts, required-field coverage, and unique identifier counts over time. Use rolling baselines where source volume changes naturally. Alert on material deviations instead of hard-coding one expected count for every run.
For recurring collection, see scheduling recurring scrapes with Python and cron. Scheduling and validation should remain separate so a successful run does not automatically imply that the resulting data is publishable.
Quarantine failures for replay
A quarantine path should preserve enough information to reproduce the decision. Store:
- Raw response or a durable reference to it.
- Normalized record, if normalization completed.
- Validation errors and rule names.
- Source URL and collection timestamp.
- Parser and schema versions.
- Batch or run identifier.
- A content hash for deduplication and comparison.
Separate retry queues from quarantine queues. Retry a temporary network or service failure according to your retry policy. Do not repeatedly retry a deterministic schema failure. Send it to quarantine and investigate the rule or source change.
Build a replay command or job that can run quarantined inputs through a newer parser without fetching the source again. This reduces load on public sites and gives engineers a controlled way to test fixes.
Monitor validation as a pipeline health signal
Track validation metrics by source, parser version, schema version, and collection run:
- Accepted record count.
- Quarantined record count.
- Failure rate by rule.
- Missing-field rate.
- Duplicate rate.
- Freshness lag.
- Record-count deviation.
- Time spent in each pipeline stage.
Metrics need context. A high quarantine rate may indicate source drift, a parser deployment, a new locale, or a change in the data contract. Include representative failed records in an internal review view, with sensitive fields handled according to your retention policy.
Log enough detail to diagnose a failure without logging secrets. API keys, session tokens, and authorization headers should never enter raw logs or quarantine records.
Test validators with fixtures
Use saved fixtures for normal pages, empty responses, partial templates, locale variations, missing fields, and known layout changes. Unit-test individual validators, then run integration tests through the full normalization and validation pipeline.
Useful test cases include:
- A valid record with all required fields.
- A record with a missing title.
- A negative numeric value.
- A malformed URL.
- A timestamp without timezone information.
- A duplicate identifier in one batch.
- A response containing an access notice instead of page content.
- A valid source page with a changed layout.
Keep fixtures representative but scrub secrets and personal data. When a production failure occurs, add a sanitized fixture before changing the parser. This turns an incident into a regression test.
For examples of how scraped data supports different application workflows, review the AlterLab use cases.
Takeaway
Reliable scraped data comes from explicit contracts and observable validation, not from retrieval alone. Normalize at the boundary, check structure and meaning, measure freshness and completeness, and quarantine failures with enough context to replay them safely.
For retrieval pipelines that feed search and generation systems, see building a RAG pipeline with live web data.
Was this article helpful?
Frequently Asked Questions
Related Articles

SOCKS Proxy: A Practical Developer Guide
Learn how a SOCKS proxy routes TCP traffic, handles DNS, and fits into ethical scraping pipelines, with practical Python and cURL examples for developers.
Herald Blog Service

Akamai Bot Manager Sensor Checks: Technical Guide
Learn how Akamai Bot Manager sensor checks work, why requests fail, and how ethical scraping pipelines use browser execution, sessions, and tiered requests.
Herald Blog Service

Scheduling Recurring Scrapes with Python and Cron
Learn how to schedule recurring scrapes with Python and cron, choose safe intervals, prevent duplicate runs, handle failures, and build reliable data pipelines.
Herald Blog Service
Popular Posts
Recommended

Selenium Bot Detection: Why You Get Flagged and How to Fix It

Understanding Anti-Bot Detection: Fingerprinting and Rate Limits

Why Your Headless Browser Gets Detected (and How to Fix It)

Mastering Playwright Stealth for Agentic Web Workflows

Playwright Bot Detection: What Actually Works in 2026
Newsletter
Scraping insights and API tips. No spam.
Recommended Reading

Selenium Bot Detection: Why You Get Flagged and How to Fix It

Understanding Anti-Bot Detection: Fingerprinting and Rate Limits

Why Your Headless Browser Gets Detected (and How to Fix It)

Mastering Playwright Stealth for Agentic Web Workflows

Playwright Bot Detection: What Actually Works in 2026
Stay in the Loop
Get scraping insights, API tips, and platform updates. No spam — we only send when we have something worth reading.
Explore AlterLab
Web Scraping API Resources
Part of the Web Scraping API Documentation cluster
Complete API reference with 5-tier auto-escalation — Curl to challenge resolution.
Pillar pageConfigure Tier 4 browser rendering for SPAs and dynamic content.
Scrape pages behind login using session management.
Real success rates and cost data across all 5 tiers.
MCP Server, Python SDK, and Firecrawl-compatible API for AI agent workflows.