ETL is the foundational pattern for moving data between systems. In a scraping context: Extract means fetching raw HTML or API responses from target websites; Transform means parsing the raw responses, extracting fields, cleaning and normalising values, deduplicating, and applying business rules; Load means writing the processed records to the destination — a PostgreSQL database, a data warehouse like BigQuery or Snowflake, a data lake, or a streaming platform like Kafka.
Modern data stacks often use ELT instead: raw data is loaded first to a cloud warehouse, and transformations are applied using SQL or dbt after the fact. This approach preserves raw data for reprocessing and shifts compute to the warehouse tier, which is highly optimised for bulk transformations.
Scraping pipelines that implement ETL should treat each phase independently, with its own error handling, observability, and retry semantics. The extract phase is the most fragile (sites change HTML without warning); the transform phase is where most business logic lives; the load phase benefits from idempotent upserts.