How to Build a Web Scraper API Endpoint
If multiple services or team members need access to scraped data, a dedicated scraper API endpoint is more efficient than running scrapers in each service independently. A single API layer handles the scraping, caching, and data normalization.
Step-by-Step Guide
Set up a FastAPI project
Install FastAPI and uvicorn: `pip install fastapi uvicorn`. Create a main.py with a FastAPI app instance.
Create a scrape endpoint
Define a POST endpoint that accepts a URL and optional parameters, then calls the AlterLab API internally and returns parsed data.
Add response caching
Cache scraping results in Redis or a simple in-memory dict with a TTL to avoid re-scraping the same URL repeatedly within a short window.
Add error handling
Return structured error responses when the AlterLab API returns a non-200 status or when HTML parsing fails to find expected elements.
Code Example
from fastapi import FastAPI, HTTPException
import requests
app = FastAPI()
API_KEY = "YOUR_API_KEY"
@app.post("/scrape")
async def scrape(url: str, selector: str):
response = requests.post(
"https://alterlab.io/api/v1/scrape",
headers={"X-API-Key": API_KEY, "Content-Type": "application/json"},
json={"url": url, "render_js": True},
)
if response.status_code != 200:
raise HTTPException(status_code=502, detail="Scraping failed")
from bs4 import BeautifulSoup
html = response.json().get("html", "")
soup = BeautifulSoup(html, "html.parser")
elements = [el.get_text(strip=True) for el in soup.select(selector)]
return {"url": url, "selector": selector, "results": elements}Replace YOUR_API_KEY with your key from the dashboard. No credit card required.
Ready to try it?
Run this tutorial on live websites with AlterLab's API. Start free — no credit card required.
Frequently Asked Questions
Responsible Use
AlterLab is designed for extracting publicly available data. Always review the terms of service for any website you access, respect robots.txt directives, and ensure your use case complies with applicable laws in your jurisdiction.
More tutorials
Browse all how-to guides for web scraping — from beginner extractions to advanced multi-page pipelines.
Your first scrape.
Sixty seconds.
$1 free balance. No credit card. No SDK.
Just a POST request.
No credit card required · Up to 5,000 free scrapes · Balance never expire