Free SEO Tool
Free Robots.txt Generator
Build a valid robots.txt — allow/disallow rules, crawl-delay and your sitemap line — with one-click presets. Free, no signup, runs in your browser.
Save as robots.txt in your site root. Runs in your browser — enable JavaScript to use it.
How to use the robots.txt generator
- Pick a preset or set a user-agent. Use
*for all crawlers, or name a specific bot. - Add disallow/allow paths. The file updates live; watch for the "blocks your whole site" warning.
- Add your sitemap, then download the file and upload it to your site root.
What robots.txt does and why it matters
robots.txt is how you guide search-engine crawlers around your site — keeping them out of admin areas, duplicate paths or staging sections, and pointing them to your sitemap. A correct robots.txt protects crawl budget and prevents accidental blocking; a wrong one (like a stray Disallow: /) can quietly remove your whole site from search. Remember that disallow controls crawling, not indexing — use a noindex meta tag to keep a page out of results.
Frequently asked questions
What is a robots.txt file?
robots.txt is a plain-text file at the root of your site that tells search-engine crawlers which paths they may or may not crawl. It is the first file most bots request before crawling your site.
Where do I put the robots.txt file?
In your site root, reachable at https://yourdomain.com/robots.txt. It must be at the root — a robots.txt in a subfolder is ignored.
Does robots.txt stop a page from being indexed?
No. Disallow stops crawling, not indexing — a blocked URL can still appear in results without a snippet. To keep a page out of the index, allow crawling and use a noindex meta tag instead.
What does "Disallow: /" do?
It blocks crawlers from your entire site. This tool warns you when a rule would block everything, so you do not deploy it by accident.
Should I add my sitemap to robots.txt?
Yes. Adding a Sitemap line helps crawlers discover your XML sitemap. You can generate one with our XML Sitemap Generator and paste the URL here.
Explore more free SEO tools, build an XML sitemap, or run a full site audit.
Catch crawl issues before they cost you
Start free, every tool included, no credit card.