An operation is idempotent if repeating it has the same observable effect as performing it once. HTTP GET and DELETE are idempotent by definition; HTTP POST (which creates a new resource each time) is not. Idempotency keys — unique identifiers sent with requests — allow servers to detect duplicate requests and return the cached result of the first execution instead of repeating the operation.
In scraping APIs, idempotency is important for reliable job submission under retries. If a client submits a scrape job and the network fails before the response arrives, the client cannot know whether the server received the request. An idempotency key lets the client safely retry: the server checks if a job with that key already exists and returns its status rather than creating a duplicate job.
Idempotency also applies to data writes in scraping pipelines: inserting a scraped record should use upsert semantics (insert if absent, ignore or update if present) to prevent duplicate rows when the same URL is scraped more than once.