Question 1

How do I test if a URL is blocked by robots.txt?

Accepted Answer

Paste your robots.txt content into the input area, enter the URL path you want to test, select a user-agent, and click Test. The tool instantly shows whether the URL is allowed or disallowed with the matching rule highlighted.

Question 2

What user-agents can I test against?

Accepted Answer

You can test against common bots including Googlebot, Bingbot, GPTBot, Anthropic-AI, and more. You can also enter a custom user-agent string for testing proprietary crawlers.

Question 3

Does this tool validate robots.txt syntax?

Accepted Answer

Yes. The validator checks for common syntax errors like missing colons, invalid directives, improperly formatted wildcards, and unreachable rules that are overridden by earlier patterns.

Question 4

Is my robots.txt data sent to a server?

Accepted Answer

No. All parsing and testing happens directly in your browser. Your robots.txt content never leaves your device.

Question 5

How does robots.txt matching work?

Accepted Answer

Robots.txt uses path-prefix matching. A Disallow rule blocks any URL starting with that path. Wildcards (*) match any sequence of characters, and $ anchors the match to the end of the URL. More specific rules take precedence over broader ones.

robots.txt Tester

More Free Tools

Need to process data at scale?