```yaml
product: AlterLab
title: "Noindex — Web Scraping Glossary | AlterLab"
comparison_context: "AlterLab is an alternative to Firecrawl, ScrapingBee, and Bright Data."
source_url: https://alterlab.io/glossary/noindex
```

general

# Noindex

The `noindex` directive instructs search engine crawlers not to include a page in search index results, often used for private, duplicate, or low-value pages.

A page can be excluded from search engine indexes via two mechanisms: a `<meta name='robots' content='noindex'>` tag in the HTML `<head>`, or an `X-Robots-Tag: noindex` HTTP response header. Both signal to compliant crawlers (Googlebot, Bingbot) that the page should be fetched but not indexed — it will not appear in search results.

For web scrapers, `noindex` pages are interesting because they often contain content the site operator wants to keep private from search engines but still accessible to logged-in users — admin panels, cart pages, staging previews, or thin content variants. Scrapers are not bound by the noindex directive (it is a crawler convention, not a technical restriction), but responsible scrapers respect it in the same spirit as robots.txt.

From an SEO standpoint, noindex pages should not be included in a site's canonical content inventory. When auditing a site's crawlability, distinguishing between indexed and noindexed pages helps identify technical SEO issues like canonicalisation conflicts.

### What is Noindex?

The `noindex` directive instructs search engine crawlers not to include a page in search index results, often used for private, duplicate, or low-value pages.

### How does AlterLab handle Noindex?

From an SEO standpoint, noindex pages should not be included in a site's canonical content inventory. When auditing a site's crawlability, distinguishing between indexed and noindexed pages helps identify technical SEO issues like canonicalisation conflicts.

## Examples

```
# Detect noindex directive when crawling
from bs4 import BeautifulSoup

soup = BeautifulSoup(html, "html.parser")
robots_meta = soup.find("meta", attrs={"name": "robots"})
if robots_meta and "noindex" in robots_meta.get("content", "").lower():
    print("Page is noindex — excluded from search engine index")
```

## Related Terms

[robots.txtA plain-text file at a website's root specifying which URL paths crawlers are permitted or disallowed from accessing.](https://alterlab.io/glossary/robots-txt)[Canonical URLA canonical URL is the preferred URL for a piece of content when multiple URLs serve the same or similar content, indicated by a `<link rel='canonical'>` tag.](https://alterlab.io/glossary/canonical-url)[SitemapAn XML file listing all URLs on a website with metadata like last modification date, helping crawlers discover and prioritise pages.](https://alterlab.io/glossary/sitemap)[Crawl BudgetCrawl budget is the number of pages a search engine crawler (or a custom crawler) will fetch from a site within a given timeframe, influenced by server capacity and site size.](https://alterlab.io/glossary/crawl-budget)[SERPSearch Engine Results Page — the page returned by a search engine for a query, containing organic results, ads, and rich features.](https://alterlab.io/glossary/serp)

## Extract Noindex data from any website

AlterLab returns clean, structured data from any public URL — no scraper infrastructure needed. Start free, no credit card required.

[View API docs](https://alterlab.io/docs/api/rest)

[Back to Glossary](https://alterlab.io/glossary)