How to use it
- 1
Enter the user-agent and Allow/Disallow rules.
- 2
Add Sitemap URL and Crawl-delay only when actually needed.
- 3
Copy the file and verify it with Robots.txt Tester after publishing.
What the tool can do
- Normalizes relative paths to /path.
- Checks that Sitemap is an HTTP(S) URL.
- Runs fully in the browser without sending rules to the backend.
Common use cases
- Prepare a basic robots.txt for a new site.
- Quickly build a safe template for staging or utility sections.
- Add a Sitemap directive without syntax mistakes.
How it works inside
The generator is intentionally deterministic: output depends only on entered fields.
This is not a crawler and not a robots policy adviser.
Questions and answers
Can I Disallow everything?
Technically yes, but in production it usually blocks crawling. Use such rules only deliberately.
Why is there no automatic URL list?
Because that requires crawler or project-structure knowledge. This generator accepts only explicit rules.