How to Scrape Data with Python
Python is the most popular language for web scraping — with excellent libraries for HTTP requests, HTML parsing, and data processing. This guide covers the complete setup from API key to clean extracted data.
Step-by-Step Guide
Install required libraries
Install the requests and beautifulsoup4 libraries: `pip install requests beautifulsoup4`. These handle HTTP requests and HTML parsing respectively.
Fetch a page with AlterLab
Use the requests library to POST to the AlterLab API with your target URL and API key. The response contains the rendered HTML.
Parse with BeautifulSoup
Pass the HTML to BeautifulSoup for parsing. Use CSS selectors (`soup.select()`) or tag finders (`soup.find()`) to extract the data you need.
Store the extracted data
Write your results to a CSV with the csv module, a JSON file, or a database. Use pandas for larger datasets that benefit from tabular operations.
Code Example
import requests
from bs4 import BeautifulSoup
import csv
API_KEY = "YOUR_API_KEY"
def scrape_page(url: str) -> list[dict]:
response = requests.post(
"https://alterlab.io/api/v1/scrape",
headers={"X-API-Key": API_KEY, "Content-Type": "application/json"},
json={"url": url, "render_js": True},
)
html = response.json().get("html", "")
soup = BeautifulSoup(html, "html.parser")
return [
{"title": el.get_text(strip=True)}
for el in soup.select("h2.article-title")
]
results = scrape_page("https://example.com/articles")
with open("output.csv", "w", newline="") as f:
writer = csv.DictWriter(f, fieldnames=["title"])
writer.writeheader()
writer.writerows(results)Replace YOUR_API_KEY with your key from the . No credit card required.
Try this yourself with AlterLab
Run this tutorial on live websites with AlterLab's API. Free tier includes 5,000 requests — no credit card required.
Frequently Asked Questions
Should I use requests or httpx for scraping with Python?
Both work well with AlterLab. Use requests for synchronous scripts, or httpx with asyncio for concurrent fetching of multiple URLs. The AlterLab API accepts the same JSON body from either library.
How do I parse tables with Python?
Use `pandas.read_html(html)` to automatically parse HTML tables into DataFrames. For more control, use `soup.find_all('table')` and iterate over rows and cells manually.
Responsible Use
AlterLab is designed for extracting publicly available data. Always review the terms of service for any website you access, respect robots.txt directives, and ensure your use case complies with applicable laws in your jurisdiction.
More tutorials
Browse all how-to guides for web scraping — from beginner extractions to advanced multi-page pipelines.
Your first scrape.
Sixty seconds.
$1 free credit — up to 5,000 scrapes. No credit card.
Just a POST request.
No credit card required · $1 free credit, up to 5,000 scrapes · Balance never expires