general

Form Submission

Form submission scraping involves programmatically filling and submitting HTML forms to navigate sites that gate content behind search fields, logins, or multi-step workflows.

Many websites expose their most valuable data only after a user completes an HTML form — a search box, a date-range picker, a login screen, or a multi-step wizard. Scrapers that need this content must replicate the form submission lifecycle: locate the form element, extract hidden fields (CSRF tokens, session IDs), populate the visible fields with the desired query values, and submit via HTTP POST or browser-level click.

HTTP-based form submission requires reading the form's `action` URL, collecting all `<input>` and `<select>` values (including hidden ones), and issuing a POST request with the correct Content-Type. JavaScript-rendered forms that validate input client-side before submitting require a headless browser to execute the validation logic.

CSRF tokens are a common obstacle: the token is generated server-side and embedded in the form as a hidden field, then verified on POST. Scrapers must fetch the form page, extract the token, and include it in the POST body — and must do so within the token's validity window.

Examples

# Python: submit a form with a CSRF token
from bs4 import BeautifulSoup
import requests

session = requests.Session()
page = session.get("https://example.com/search")
soup = BeautifulSoup(page.text, "html.parser")
csrf = soup.find("input", {"name": "csrf_token"})["value"]
results = session.post("https://example.com/search",
    data={"csrf_token": csrf, "q": "web scraping"})

Related Terms

Session Management
The handling of cookies, authentication tokens, and browser state across multiple requests to maintain a continuous browsing session.
Cookie Handling
Cookie handling in web scraping refers to managing browser cookies to maintain sessions, pass anti-bot checks, and access authenticated content.
JavaScript Rendering
The process of executing a page's JavaScript in a browser engine before extracting content, required for React, Vue, and Angular SPAs.
Headless Browser
A web browser that runs without a graphical user interface, executing JavaScript and rendering pages exactly as a real browser would.
Dynamic Content
Page content generated client-side by JavaScript after the initial HTML is served, requiring a browser engine to render.

Extract Form Submission data from any website

AlterLab returns clean, structured data from any public URL — no scraper infrastructure needed. Start free, no credit card required.

View API docs
Back to Glossary

Your first scrape.
Sixty seconds.

$1 free balance. No credit card. No SDK.Just a POST request.

terminal
curl -X POST https://api.alterlab.io/v1/scrape \
-H "X-API-Key: YOUR_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com", "formats": ["markdown"]}'

No credit card required · Up to 5,000 free scrapes · Balance never expires

    Form Submission — Web Scraping Glossary | AlterLab