```yaml
product: AlterLab
title: "robots.txt — Web Scraping Glossary | AlterLab"
comparison_context: "AlterLab is an alternative to Firecrawl, ScrapingBee, and Bright Data."
source_url: https://alterlab.io/glossary/robots-txt
```

general

# robots.txt

A plain-text file at a website's root specifying which URL paths crawlers are permitted or disallowed from accessing.

The `robots.txt` file is a plain-text convention that websites place at their root URL (e.g. `https://example.com/robots.txt`) to communicate crawling directives to web robots. The file uses `User-agent` directives to target specific crawlers (or `*` for all crawlers) and `Disallow` and `Allow` rules to specify which URL paths the crawler should or should not access.

Critically, robots.txt is a convention — not a technical enforcement mechanism. Web servers do not automatically block requests to disallowed paths; the file is simply a published preference that well-behaved crawlers are expected to honour. Search engine crawlers (Googlebot, Bingbot) respect robots.txt rigorously. Third-party scraping tools are not technically prevented from accessing disallowed paths.

The legal weight of robots.txt in the context of scraping has been debated in court cases. Accessing robots.txt-disallowed paths may be considered a violation of a site's terms of service, which can have legal implications depending on jurisdiction and use case. Reviewing robots.txt before scraping at scale is a best practice for ethical and legal risk management.

### What is robots.txt?

A plain-text file at a website's root specifying which URL paths crawlers are permitted or disallowed from accessing.

### How does AlterLab handle robots.txt?

The legal weight of robots.txt in the context of scraping has been debated in court cases. Accessing robots.txt-disallowed paths may be considered a violation of a site's terms of service, which can have legal implications depending on jurisdiction and use case. Reviewing robots.txt before scraping at scale is a best practice for ethical and legal risk management.

## Examples

```
# Example robots.txt
User-agent: *
Disallow: /admin/
Disallow: /private/
Allow: /public/
Crawl-delay: 10

Sitemap: https://example.com/sitemap.xml
```

## Related Terms

[Web CrawlerA program that automatically navigates websites by following hyperlinks, systematically indexing pages across an entire domain.](https://alterlab.io/glossary/web-crawler)[SitemapAn XML file listing all URLs on a website with metadata like last modification date, helping crawlers discover and prioritise pages.](https://alterlab.io/glossary/sitemap)[Web ScrapingAutomated extraction of data from websites using software tools that parse HTML and collect structured information at scale.](https://alterlab.io/glossary/web-scraping)[Rate LimitingA server-side control that caps the number of requests accepted from a single IP or session within a time window, returning HTTP 429 when exceeded.](https://alterlab.io/glossary/rate-limiting)

## Extract robots.txt data from any website

AlterLab returns clean, structured data from any public URL — no scraper infrastructure needed. Start free, no credit card required.

[View API docs](https://alterlab.io/docs/api/rest)

[Back to Glossary](https://alterlab.io/glossary)