Easily generate custom and optimized robots.txt files for your website.
What robots.txt actually controls, how to generate and deploy one correctly, common mistakes to avoid, and answers to the questions we get asked most.
Use the form above to choose your default crawl policy, restricted paths, and sitemap, then copy or download the generated file and upload it to your site's root directory.
1. Set your rules. Decide whether bots are allowed by
default, check the boxes for backend folders you want blocked, and
add your sitemap URL so crawlers can find your pages faster.
2. Review the output. The preview box updates live
and is read-only by design, so the syntax highlighting and the
built-in rule tester always reflect exactly what your settings
produce — use the form controls (or the custom-path and custom-bot
fields) to add anything the checkboxes don't cover, rather than
editing the preview directly.
3. Deploy it. Download the file, keep the name
exactly robots.txt, and upload it to your site's root —
for example public_html/robots.txt — so it's reachable
at https://yourdomain.com/robots.txt.
Here's an example combining a general rule, an AI-crawler block, and a sitemap — the shape the generator above produces:
Each User-agent line starts a new group; the rules that
follow apply only to that bot until the next User-agent
line. A bot with no group of its own falls back to the
User-agent: * rules.
Blocking the entire site by accident. A stray
Disallow: / under User-agent: * tells every
search engine not to crawl any page on your domain — this is the
single most common way a site accidentally disappears from search
results, so it should only ever be live on a staging or private site,
never production.
Using it to hide sensitive data. robots.txt is a
public file — anyone can view it at /robots.txt, which
means listing a private folder there actually advertises its location
rather than hiding it. Keep sensitive paths off the public internet
entirely (authentication, server config) rather than relying on a
crawling directive to hide them.
Leaving out the sitemap. A Sitemap: line
gives crawlers a direct, organized list of your pages instead of
relying on them to discover everything through links — it's a small
addition that meaningfully speeds up how quickly new or updated pages
get found.
Tells search engines and other bots which paths to request and which to skip.
Block AI training bots like GPTBot or CCBot independently of your normal search-engine rules.
Check whether a specific URL is allowed or blocked for a specific bot before you deploy the file.
noindex meta tag to that page's HTML instead.
https://yourdomain.com/robots.txt. Placing it
inside a subfolder, like /blog/robots.txt, means
crawlers will never find it and its rules are simply ignored.
User-agent group, separate from
User-agent: *, so you can give an AI crawler its
own Disallow: / group while leaving Googlebot and
Bingbot fully allowed. The AI crawler controls section above
generates exactly this kind of per-bot rule.