How to Build a Web Scraper API Endpoint
If multiple services or team members need access to scraped data, a dedicated scraper API endpoint is more efficient than running scrapers in each service independently. A single API layer handles the scraping, caching, and data normalization.
Step-by-Step Guide
Set up a FastAPI project
Install FastAPI and uvicorn: `pip install fastapi uvicorn`. Create a main.py with a FastAPI app instance.
Create a scrape endpoint
Define a POST endpoint that accepts a URL and optional parameters, then calls the AlterLab API internally and returns parsed data.
Add response caching
Cache scraping results in Redis or a simple in-memory dict with a TTL to avoid re-scraping the same URL repeatedly within a short window.
Add error handling
Return structured error responses when the AlterLab API returns a non-200 status or when HTML parsing fails to find expected elements.
Code Example
from fastapi import FastAPI, HTTPException
import requests
app = FastAPI()
API_KEY = "YOUR_API_KEY"
@app.post("/scrape")
async def scrape(url: str, selector: str):
response = requests.post(
"https://alterlab.io/api/v1/scrape",
headers={"X-API-Key": API_KEY, "Content-Type": "application/json"},
json={"url": url, "render_js": True},
)
if response.status_code != 200:
raise HTTPException(status_code=502, detail="Scraping failed")
from bs4 import BeautifulSoup
html = response.json().get("html", "")
soup = BeautifulSoup(html, "html.parser")
elements = [el.get_text(strip=True) for el in soup.select(selector)]
return {"url": url, "selector": selector, "results": elements}Replace YOUR_API_KEY with your key from the . No credit card required.
Try this yourself with AlterLab
Run this tutorial on live websites with AlterLab's API. Free tier includes 5,000 requests — no credit card required.
Frequently Asked Questions
Should I cache scraping results?
Yes — caching is important for cost control and latency. Cache responses with a TTL that matches how frequently the data changes: minutes for live pricing, hours for product details, days for static content.
Responsible Use
AlterLab is designed for extracting publicly available data. Always review the terms of service for any website you access, respect robots.txt directives, and ensure your use case complies with applicable laws in your jurisdiction.
More tutorials
Browse all how-to guides for web scraping — from beginner extractions to advanced multi-page pipelines.
Your first scrape.
Sixty seconds.
$1 free credit — up to 5,000 scrapes. No credit card.
Just a POST request.
No credit card required · $1 free credit, up to 5,000 scrapes · Balance never expires