general

Form Submission

Form submission scraping involves programmatically filling and submitting HTML forms to navigate sites that gate content behind search fields, logins, or multi-step workflows.

Many websites expose their most valuable data only after a user completes an HTML form — a search box, a date-range picker, a login screen, or a multi-step wizard. Scrapers that need this content must replicate the form submission lifecycle: locate the form element, extract hidden fields (CSRF tokens, session IDs), populate the visible fields with the desired query values, and submit via HTTP POST or browser-level click.

HTTP-based form submission requires reading the form's `action` URL, collecting all `<input>` and `<select>` values (including hidden ones), and issuing a POST request with the correct Content-Type. JavaScript-rendered forms that validate input client-side before submitting require a headless browser to execute the validation logic.

CSRF tokens are a common obstacle: the token is generated server-side and embedded in the form as a hidden field, then verified on POST. Scrapers must fetch the form page, extract the token, and include it in the POST body — and must do so within the token's validity window.

What is Form Submission?

Form submission scraping involves programmatically filling and submitting HTML forms to navigate sites that gate content behind search fields, logins, or multi-step workflows.

How does AlterLab handle Form Submission?

CSRF tokens are a common obstacle: the token is generated server-side and embedded in the form as a hidden field, then verified on POST. Scrapers must fetch the form page, extract the token, and include it in the POST body — and must do so within the token's validity window.

Examples

# Python: submit a form with a CSRF token
from bs4 import BeautifulSoup
import requests

session = requests.Session()
page = session.get("https://example.com/search")
soup = BeautifulSoup(page.text, "html.parser")
csrf = soup.find("input", {"name": "csrf_token"})["value"]
results = session.post("https://example.com/search",
    data={"csrf_token": csrf, "q": "web scraping"})

Related Terms

Session Management
The handling of cookies, authentication tokens, and browser state across multiple requests to maintain a continuous browsing session.
Cookie Handling
Cookie handling in web scraping refers to managing browser cookies to maintain sessions, pass anti-bot checks, and access authenticated content.
JavaScript Rendering
The process of executing a page's JavaScript in a browser engine before extracting content, required for React, Vue, and Angular SPAs.
Headless Browser
A web browser that runs without a graphical user interface, executing JavaScript and rendering pages exactly as a real browser would.
Dynamic Content
Page content generated client-side by JavaScript after the initial HTML is served, requiring a browser engine to render.

Extract Form Submission data from any website

AlterLab returns clean, structured data from any public URL — no scraper infrastructure needed. Start free, no credit card required.

View API docs
Back to Glossary

Your first scrape.
Sixty seconds.

$1 free credit — up to 5,000 scrapes. No credit card.Just a POST request.

terminal
curl -X POST https://api.alterlab.io/v1/scrape \
-H "X-API-Key: YOUR_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com", "formats": ["markdown"]}'

No credit card required · $1 free credit, up to 5,000 scrapes · Balance never expires