Create account
SEO and auditingChecked through WebDiag API

Robots.txt Tester

Enter a URL and user-agent to see whether the path is allowed for a search crawler, which rule matched, and whether Sitemap directives are declared.

  • Single URL
  • Allow/Disallow
  • Sitemap directives
Online tool

Try the tool

Enter a URL: WebDiag will run a safe network check and show the result.

URL and user-agent

Check result

After the check, robots.txt availability, the URL rule, and Sitemap directives will appear here.

Network check through backend

For HTTP/SEO tools, the URL is sent to the WebDiag API, where SSRF protection, DNS/IP policy, and redirect limits are applied.

Reviewed: 2026-07-17
01

How to use it

  1. 1

    Paste the page URL you need to test for crawling.

  2. 2

    Keep WebDiagBot or enter another user-agent, for example Googlebot.

  3. 3

    Review robots.txt status, the matching rule, and the Sitemap directives list.

02

What the tool can do

  • Safe /robots.txt fetching with SSRF protection and response size limits.
  • Checks a specific URL path against Allow/Disallow rules for the selected user-agent.
  • Shows the matched rule, Disallow rule count, and discovered Sitemap directives.
03

Common use cases

  • Check whether an important landing page is blocked from search crawlers.
  • Find which robots.txt rule affects a catalog, filter, blog, or service page.
  • Verify Sitemap declarations after a release, migration, or CMS change.
04

How it works inside

WebDiag builds the /robots.txt URL from the tested address origin and analyzes it with the existing robots parser.

Allow wins over Disallow at equal specificity, and a more specific rule overrides a broader one.

05

Questions and answers

If robots.txt is unavailable, is the site blocked from indexing?

No. An unavailable robots.txt usually means no explicit crawl restrictions were found. Production sites should publish robots.txt and Sitemap directives explicitly.

Can I test Googlebot or Yandex?

Yes, you can enter another user-agent. The tool applies the most specific matching rule group.