Scheduling Recurring Scrapes with Python and Cron

TL;DR

Recurring scraping means invoking a scrape request on a fixed schedule, saving each result as a versioned observation, and recording whether the run succeeded. Use cron or a managed scheduler for cadence, keep the scrape call stateless, and add idempotency, backoff, logging, and overlap controls around it. Collect only publicly accessible content and choose intervals that respect site policies and response load.

What recurring scraping does

A one-time scrape answers, “What does this page contain now?” A recurring scrape answers, “What changed since the previous observation?”

The basic pipeline is:

schedule → scrape request → raw result → normalization → storage → downstream processing

Related pipeline patterns include building a RAG pipeline with live web data and building a production web scraping pipeline for RAG in 2026.

The scraper should perform one clear operation: fetch a permitted public URL and return its content. The scheduler should decide when that operation runs. Keeping these responsibilities separate makes the system easier to test and operate.

A recurring job usually needs five pieces of information:

• Target: The public URL or set of URLs to collect.
• Cadence: An hourly, daily, weekly, or custom cron schedule.
• Output: The raw response, normalized fields, or both.
• Storage key: A deterministic identifier for the target and scheduled time.
• Run state: Started, completed, failed, skipped, or still running.

A schedule alone does not create a data pipeline. Without versioned storage and run state, you cannot reliably distinguish a new observation from a duplicate or determine whether a missing result represents unchanged content or a failed job.

Test one request before adding a schedule:

Choose where scheduling runs

There are two common execution models.

Use a managed scheduler

A platform-managed scheduler owns the cron expression and triggers the scrape service for you. This reduces infrastructure work and keeps schedules, targets, and job history in one place.

AlterLab supports cron-based recurring scrapes. When using a managed schedule, configure the target, cadence, and output destination through the supported scheduling interface. Use the API docs for the current request and configuration details.

For larger target collections, see the batch scraping guide.

Use your own cron or job runner

Infrastructure-managed scheduling works well when your team already operates cron, Kubernetes jobs, CI workers, or a queue. The scheduler calls the scrape API, then your application stores and processes the response.

This model gives you direct control over:

• Secret injection
• Retry policy
• Concurrency limits
• Storage format
• Logging and alerting
• Deployments and rollback

The scrape request remains the same regardless of the trigger. That makes it possible to run a job manually, from a test, or on a schedule without changing collection code.

Set a cadence based on content change frequency

The right interval depends on the source and the purpose of your pipeline.

Use a shorter interval when:

• The page changes several times during the day.
• Your application needs fresh availability or pricing data.
• You are tracking a time-sensitive public feed.
• A downstream process can handle frequent observations.

Use a longer interval when:

• The page changes only weekly or monthly.
• You are building a historical archive.
• Freshness is less important than low request volume.
• The source publishes updates on a predictable schedule.

Cron uses five fields:

minute hour day-of-month month day-of-week

Examples:

• 0 * * * * runs at the start of every hour.
• 30 2 * * * runs every day at 02:30.
• 0 0 * * 1 runs every Monday at midnight.

Choose one time zone and document it. UTC avoids daylight saving changes and makes timestamps consistent across workers. If a team operates in local time, record both the configured time zone and the actual UTC start time.

Avoid scheduling a large URL list at the same minute when the jobs can be distributed. For example, assigning different minute offsets across groups of targets reduces bursts and makes failures easier to isolate.

A recurring job should also respect the source's published policies and terms. Collect public content only. Do not use scheduling to bypass authentication, paywalls, robots restrictions, or other access controls. If a public page requires JavaScript rendering, choose a supported rendering mode that matches the page and your permitted use.

Run recurring scrapes with the Python SDK

The Python SDK can run the same scrape operation from a long-running worker. The example below waits one hour after each completed request.

import time
from datetime import datetime, timezone
from alterlab import AlterLabSync

URL = "https://example.com"
INTERVAL_SECONDS = 3600

client = AlterLabSync(api_key="YOUR_API_KEY")

while True:
    started_at = datetime.now(timezone.utc).isoformat()
    result = client.scrape(URL)
    print(started_at, len(result["content"]))
    time.sleep(INTERVAL_SECONDS)

This pattern is useful for a small worker or a local prototype. The interval begins after the scrape finishes, so a slow request shifts the next run. That behavior can be acceptable for a simple polling loop.

For fixed wall-clock times, use cron or another scheduler instead. A long-running loop also needs process supervision. If the process exits, the schedule stops until a supervisor restarts it.

Keep the API key outside source control. Inject it through the worker environment or a secret manager in production. The placeholder in the example is not a value to commit to a repository.

Trigger the same scrape with cURL and cron

A cron job can invoke the API directly. Save the request as an executable shell script:

#!/usr/bin/env bash
set -euo pipefail

curl -X POST https://api.alterlab.io/api/v1/scrape \
  -H "X-API-Key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'

The request performs the same operation as the Python example. The response is written to standard output, so the calling job can redirect it to a file, log stream, or downstream process.

Schedule the script at minute 15 of every hour:

15 * * * * /opt/scraper/run-scrape.sh >> /var/log/recurring-scrape.log 2>&1

Before enabling the schedule:

1. Run the script manually.
2. Confirm the API key is available to the cron environment.
3. Check that the output destination has enough storage.
4. Verify the server time zone.
5. Run the script twice and confirm how duplicates are handled.
6. Add a lock if the job must never overlap.

Cron does not know whether the previous process is still running. If a scrape can take longer than its interval, two processes may run at once. Use a host-level lock, a queue with a concurrency limit, or a scheduler setting that skips a new run while the previous run is active.

Store each scrape as a versioned observation

Do not overwrite the previous result unless you only need the latest state. Historical results make it possible to investigate changes, reprocess data, and recover from downstream failures.

At minimum, store:

• Target URL
• Scheduled time
• Actual start time
• Completion time
• Run status
• Raw response content
• Content hash
• Parser or extraction version
• Error details when the run fails

A useful storage key combines the target and scheduled time. For example, an hourly job can use a key based on the URL and the UTC hour. If the scheduler retries the same hour, the retry updates the existing observation instead of creating a second copy.

Hashing helps detect content changes, but hash the right representation. Raw HTML often contains changing timestamps, session markers, or advertisement slots. Normalize those volatile fields before comparing content when the business question concerns meaningful page changes.

Keep raw and normalized forms separate:

• Raw content supports audits and future parser changes.
• Normalized content supports comparisons and analytics.
• Extracted records support application queries.

A failed scrape should not replace the last successful observation with an empty value. Store the failure separately and preserve the previous successful result.

Add retries without creating request bursts

Transient failures happen in scheduled systems. A worker may lose network access, a response may time out, or a remote page may be temporarily unavailable.

Use bounded retries with increasing delays. A practical sequence can start with a short delay, then wait longer between attempts. Add jitter so multiple workers do not retry at the same second.

Separate failures into categories:

• Transport failure: The worker could not complete the request.
• Temporary service failure: A dependent service returned a transient error.
• Empty or changed content: The request completed, but the page content differs from expectations.
• Parsing failure: The response arrived, but your parser could not process it.
• Policy or access failure: The job should not keep retrying without human review.

Do not retry every failure indefinitely. An unavailable target, changed page structure, or disallowed access can create a permanent loop that consumes resources and obscures the original problem.

Choose one owner for retries. If cron retries the process and the worker also retries internally, a single scheduled run can produce more requests than expected. Document the total attempt limit across both layers.

Track these operational values:

• Consecutive failures per target
• Last successful run
• Last attempted run
• Next scheduled run
• Number of attempts in the current run
• Time spent processing the result

Alerts should focus on actionable states, such as repeated failures or a missed freshness window. A single transient failure can remain in job history without waking an operator.

Prevent duplicate and overlapping runs

Two duplicate problems appear in recurring pipelines.

Duplicate execution

A scheduler may trigger a retry after a timeout even though the original request eventually completed. Use a deterministic run key, such as target plus scheduled slot, and make storage writes idempotent.

If the API response does not contain an application-specific identifier, generate one from your own schedule metadata. Do not use the current wall-clock time alone, because retries would generate different keys for the same scheduled slot.

Overlapping execution

A new run can start before the previous run finishes. Select an explicit policy:

• Skip: Ignore a new slot when the previous run is active.
• Queue: Keep one pending run and execute it after the current run.
• Parallelize: Run concurrently when targets and downstream storage support it.
• Coalesce: Treat several missed slots as one current-state collection.

Skipping or coalescing works for current-state snapshots. Queueing preserves each scheduled observation. Parallel execution requires per-target rate limits and enough storage capacity.

Apply the policy per target, not just globally. A slow JavaScript-heavy page should not block unrelated lightweight pages.

Use monitoring when polling is not required

A fixed schedule is appropriate when you need regular snapshots, periodic extraction, or a time series. It may be unnecessary when the only requirement is to know whether a page changed.

For that use case, change monitoring can separate change detection from downstream retrieval. The monitor records page changes and can notify your system, while a follow-up scrape retrieves content only when the change matters.

For broader application patterns, see web scraping use cases.

Use scheduled scraping when:

• You need every observation, including unchanged ones.
• You need periodic structured extraction.
• You need a regular archive.
• Your downstream system expects a fixed cadence.

Use monitoring when:

• You only need notifications about changes.
• The page changes infrequently.
• Reprocessing unchanged content has little value.
• You want to reduce downstream work.

Both patterns still need clear retention, error handling, and permitted-use policies.

Production checklist for recurring scrape jobs

Before moving a recurring job into production, verify the following:

• The target is publicly accessible and permitted for automated collection.
• The cron expression has a documented time zone.
• The API key is injected securely.
• The job has a bounded concurrency limit.
• Overlap behavior is explicit.
• Retries have a maximum attempt count.
• Results include scheduled and actual timestamps.
• Duplicate writes are idempotent.
• Raw results and normalized records have separate retention policies.
• Logs exclude API keys and unnecessary sensitive content.
• Storage capacity matches the expected retention period.
• Spend limits are configured for the API key where available.
• Operators can see the last success, last failure, and next run.
• A page structure change produces an observable parsing failure instead of silent bad data.

Start with one target and a long interval. Confirm the complete path from scheduler to storage, then add more targets gradually. This exposes issues with permissions, time zones, secret handling, and output retention before they affect a larger pipeline.

FAQ

How do I schedule recurring web scraping with cron?

Create a script that sends one scrape request, then call it from a cron entry such as 15 * * * *. Store the response with a scheduled timestamp and protect the job from overlapping executions.

What is a reasonable interval for scheduled scraping?

Choose an interval based on the source's update frequency and your application's freshness requirement. Longer intervals reduce request load for stable pages, while faster-changing public content may need more frequent observations.

How do I stop duplicate scheduled scrape results?

Generate a deterministic key from the target and scheduled time, then make the storage write idempotent. Add a lock or queue so a retry cannot create a second active run for the same slot.

Takeaway

A reliable recurring scrape has three layers: a scheduler that controls cadence, a single scrape request that collects public content, and a storage pipeline that records versioned results. Add UTC timestamps, bounded retries, idempotent writes, overlap control, and clear monitoring before increasing the number of targets.