How to Monitor Website Changes and Get Alerted on Content Updates

TL;DR

Set up a recurring scrape with AlterLab, enable change detection, and attach a webhook to receive instant alerts when content updates. The process requires only a few API calls and works for any publicly accessible site.

Why Monitor Changes

Tracking updates on e‑commerce listings, documentation, or public data sources lets you trigger downstream pipelines, refresh caches, or notify stakeholders the moment information shifts. Doing this manually is error‑prone and unscalable. An automated approach gives you:

• Near‑real‑time visibility into data freshness
• Reduced need for manual checks
• Trigger‑based actions (e.g., re‑run ETL jobs, update dashboards)

Setting Up Change Detection with AlterLab

AlterLab’s monitoring mode stores a hash of each scrape result. On subsequent runs, it compares the new hash to the previous one and flags a change when they differ. You can also request a full diff if you need to see what changed.

To enable this:

1. Create a scrape job with the monitor flag set to true.
2. Provide a cron expression for how often to run the job.
3. Optionally specify a webhook URL to receive POST requests on change detection.

The API handles anti‑bot rotation, JavaScript rendering, and CAPTCHA solving automatically, so your monitoring logic stays simple.

Creating a Monitoring Job

Below are equivalent examples in Python (using the official SDK) and cURL. Both create a job that scrapes a target URL every hour and sends a webhook payload when the content changes.

import alterlab
from datetime import datetime

# Initialize client – replace with your actual key
client = alterlab.Client("YOUR_API_KEY")   # highlighted

# Define the monitoring job
job = {
    "url": "https://example.com/public-data",
    "schedule": "0 * * * *",               # every hour (cron)
    "monitor": True,                       # enable change detection
    "webhook": "https://yourdomain.com/alterlab-webhook",  # highlighted
    "formats": ["json"]                    # receive parsed JSON
}

# Create the job
response = client.jobs.create(job)         # highlighted
print(f"Job created: {response['id']}")

curl -X POST https://api.alterlab.io/v1/jobs \
  -H "X-API-Key: YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/public-data",
    "schedule": "0 * * * *",
    "monitor": true,
    "webhook": "https://yourdomain.com/alterlab-webhook",
    "formats": ["json"]
  }'                                       # highlighted

TryIt Block

Handling Notifications via Webhooks

When a change is detected, AlterLab POSTs a JSON payload to your webhook URL. The payload includes:

• job_id: the monitoring job that triggered the alert
• timestamp: when the change was found
• data: the latest scraped content (in the format you requested)
• change_type: either "hash" (simple change flag) or "diff" (line‑by‑line diff if you requested it)

Your webhook endpoint should verify the request (e.g., via a shared secret header) and then trigger whatever action you need—such as kicking off a downstream Airflow DAG, updating a database, or sending a Slack message.

Here’s a minimal Python Flask receiver:

from flask import Flask, request, abort
import hashlib
import hmac

app = Flask(__name__)
WEBHOOK_SECRET = b"your-shared-secret"

def verify_signature(payload, signature):
    """Validate AlterLab webhook signature."""
    mac = hmac.new(WEBHOOK_SECRET, payload, hashlib.sha256)
    return hmac.compare_digest(mac.hexdigest(), signature)

@app.route("/alterlab-webhook", methods=["POST"])
def alterlab_webhook():
    signature = request.headers.get("X-Alterlab-Signature", "")
    if not verify_signature(request.data, signature):
        abort(403)

    payload = request.get_json()
    # Example: log the change and trigger a refresh
    app.logger.info(f"Change detected in job {payload['job_id']} at {payload['timestamp']}")
    # TODO: call your pipeline or notification system here
    return "", 204

if __name__ == "__main__":
    app.run(port=5000)

Step‑by‑Step Process

Best Practices

• Idempotent webhooks: Design your endpoint to safely handle duplicate alerts (e.g., by checking timestamps).
• Secure secrets: Store your AlterLab API key and webhook secret in a vault or environment variables, never in source code.
• Rate limits: Monitor jobs consume API credits based on the tier needed for the target site. Review your usage in the dashboard to avoid unexpected costs.
• Diff granularity: If you only need to know that something changed, stick with hash comparison. If you need to know what changed, enable diff: true (available in the API) but expect larger payloads.
• Testing: Use the AlterLab playground to manually scrape a page and verify the output format before scheduling.

Conclusion

Automated change detection turns a manual, error‑prone task into a reliable, event‑driven workflow. By leveraging AlterLab’s built‑in scheduling, anti‑bot handling, and webhook system, you can focus on acting on the data rather than keeping the scraper running. Start with a simple hourly job, verify your webhook logic, then scale to dozens of monitored sources as your pipelines grow.

For more details, see the Python SDK or the quickstart guide. If you have questions, hit reply.