Our free Robots.txt Generator helps you create a robots.txt file without manually writing the directives. The tool provides options for setting default crawler rules, configuring crawl delays, specifying your sitemap, managing AI crawlers, and restricting directories.
Robots.txt File Generator
AI crawlers and bots
Choose a rule for each crawler. "Use default" inherits the main rule above.
Choose rules for selected AI crawlers
Uncheck a bot to omit its individual rule. It will then be governed by any applicable broader rule and the crawler's own interpretation of robots.txt.
Restricted directories
Enter paths relative to your website root. Each path must start and end with a slash, for example /private/.
Generated robots.txt
Review the rules before publishing. You can edit the generated text directly.
Important: robots.txt controls crawling, not indexing or privacy. For private pages, use authentication or server-side access controls. Test the final file and upload it to the root of your domain (for example, https://www.example.com/robots.txt).
What is Robots.txt File?
The robots.txt file is a plain-text file that provides instructions to web crawlers about which parts of a website they are allowed or asked not to crawl. Search engines such as Google and Bing use this file to understand a website owner’s crawling preferences. Website owners can also use it to specify rules for certain AI crawlers. The robots.txt file should be placed in the root directory of your website. For example, if your website is https://www.example.com/, the file should be accessible at https://www.example.com/robots.txt. On many hosting platforms, the root directory is named public_html, although the exact directory name may vary.
Search engines and other compliant crawlers look for the robots.txt file at the root of the relevant domain. However, robots.txt is not a security mechanism. It does not prevent visitors from accessing public URLs, and crawlers may ignore its instructions. Use authentication or server-side access restrictions to protect private content.
How to Use the Robots.txt Generator Tool?
Our free tool runs directly in your browser, so your settings are processed on your device without requiring to store any data on server-side. Here are the available options in the tool and how to use them:
1. Default Rule
The “Default Rule” dropdown determines whether the general group of crawlers should be allowed or disallowed from crawling your website. You can choose between the following options:
- Allow all: Allows crawling by default, subject to any additional restrictions you configure.
- Disallow all: Instructs all crawlers matched by the wildcard rule not to crawl your website.
For most public websites that rely on organic search traffic, allowing crawlers is the appropriate starting point. You can then add specific restrictions for directories or individual bots. For example, selecting “Allow all” generates the following directive:
User-agent: *
Disallow:
Selecting “Disallow all” generates:
User-agent: *
Disallow: /
The wildcard symbol (*) represents all user agents that match the general rule.
2. Crawl Delay
The “Crawl delay” option lets you specify a delay of 1, 2, 5, or 10 seconds between requests made by crawlers that support this directive. You can also select “No crawl delay” to omit the directive. For example, selecting a 5-second delay generates:
User-agent: *
Disallow:
Crawl-delay: 5
Most shared hosting companies require you to set crawl-daly to stop automated bots. However, there is no universal crawl-delay value that every shared hosting provider requires. Choose a delay based on your website’s capacity and actual server load rather than assuming that a particular value is necessary for all websites.
Note: Not all bots support this directive, and its behavior may vary between crawlers. For example, Google does not support the Crawl-delay directive in robots.txt. If you need to manage Googlebot’s crawling activity, check the available crawl management options in Google Search Console. Bing and some other crawlers may recognize crawl-delay, but support and behavior can change. Check the relevant crawler’s documentation before relying on this setting.
3. Sitemap URL
The “Sitemap URL” field allows you to specify the location of your XML sitemap. A sitemap helps search engines discover URLs on your website. If your sitemap is located at https://www.example.com/sitemap.xml, enter the complete URL in the “Sitemap URL” field.
The tool adds a Sitemap directive to your generated file like below:
Sitemap: https://www.example.com/sitemap.xml

If you leave the field blank, no Sitemap directive is added. You can also include multiple Sitemap directives by editing the generated file after creating it. If your website uses an SEO plugin that generates a sitemap index, use the sitemap index URL rather than an individual sitemap unless you have a specific reason to list individual sitemaps.
Note: Adding a sitemap to robots.txt does not replace submitting it through Google Search Console or Bing Webmaster Tools.
4. AI Crawler Management
One of the key features of the Robots.txt Generator is its ability to manage rules for popular AI-related crawlers. AI companies use different crawlers for different purposes. Some collect information that may be used for model training, while others help search engines and AI assistants retrieve web content to answer user queries. The tool includes the following crawlers:
| Crawler | General Purpose |
|---|---|
| GPTBot | OpenAI crawler associated with model training |
| OAI-SearchBot | OpenAI search crawler |
| ChatGPT-User | Fetches web pages in response to user requests |
| ClaudeBot | Anthropic crawler used for model-related purposes |
| Claude-SearchBot | Anthropic search crawler |
| Claude-User | Fetches pages in response to user requests |
| Google-Extended | Controls certain Google AI training and grounding uses |
| Applebot-Extended | Controls certain uses of web content for Apple AI models |
| PerplexityBot | Perplexity crawler |
| Perplexity-User | Fetches pages in response to user requests |
| Bytespider | Crawler associated with ByteDance |
| CCBot | Common Crawl web crawler |
| cohere-ai | Cohere-related crawler |
| Meta-ExternalAgent | Meta crawler associated with AI-related uses |
| Amazonbot | Amazon web crawler |
| YouBot | You.com search crawler |
How to Configure Individual AI Crawlers?
Each crawler has a checkbox and a corresponding rule dropdown. The checkbox determines whether the tool includes an individual rule for that crawler. The dropdown lets you select one of three options:
- Use default: Does not generate a separate rule for that crawler.
- Allow: Explicitly permits crawling under that crawler’s user-agent group.
- Block: Generates a directive instructing that crawler not to crawl any URL.
For example, if you want to allow general search engine crawling but block GPTBot, select “Allow” all under “Default rule” and choose “Block” for “GPTBot”. The resulting directives will include:
User-agent: *
Disallow:
User-agent: GPTBot
Disallow: /
This instructs general crawlers to crawl the site while telling GPTBot not to crawl any URLs.
Understanding the AI Crawler Presets
The tool also provides presets to simplify configuration.
- Allow Listed AI Crawlers: Sets the selected AI crawlers to “Allow”. This is useful if you want to explicitly allow the listed bots.
- Block Common AI Training Crawlers, Allow Search and Visitor Bots: Blocks crawlers categorized in the tool as training-related while allowing selected search and visitor bots.
- Block All Listed AI Crawlers: Sets the listed AI crawlers to “Block”.
- Custom Settings: Lets you configure the rules individually.
Note that AI training crawlers, search crawlers, and user-request crawlers are not interchangeable. For example, blocking GPTBot does not automatically block OAI-SearchBot or ChatGPT-User. Similarly, blocking Google-Extended does not block Google Search crawling. If you want your content to appear in search results or be retrieved by AI-powered search services, consider the effect of each rule before blocking a crawler.
5. Restricted Directories
The “Restricted directories” section allows you to specify website paths that crawlers should not access. For example, if you want to restrict the /private/ directory, enter /private/. The tool automatically adds a leading slash if necessary and appends a trailing slash. You can add multiple directory paths using the “Add directory” button. Use “Remove” to delete a directory from the list.
For a WordPress website, you can restrict /wp-admin/. This instructs compatible crawlers not to crawl URLs under that path when the applicable rules permit the directive to take effect. However, WordPress websites require care when restricting directories. Avoid blocking resources that search engines need to render your pages, such as relevant CSS and JavaScript files.
6. Generating Your Robots.txt File
After configuring the default rule, crawl delay, sitemap, AI crawlers, and restricted directories, you have the following options:
- Generate robots.txt: The tool creates the corresponding directives and displays them in the “Generated robots.txt” text box. Review the output carefully before publishing it. The generated file is editable, so you can make additional changes directly in the text box.
- Download robots.txt: This will save the generated content as a plain-text file named
robots.txt. After downloading the file, upload it to the root directory of your website using your hosting control panel’s File Manager or an FTP/SFTP client. - Copy to Clipboard: If you prefer to create the file manually, click “Copy to Clipboard”. The tool copies the generated robots.txt content so you can paste it into a text editor, such as Notepad or another plain-text editor. Save the file using the filename
robots.txtand UTF-8 encoding. Ensure that your editor does not add an unintended.txtextension to an existing filename or save the file as a rich-text document. You can then upload it to your website’s root directory. - Reset: The “Reset” button restores the tool’s default settings. It resets the default crawler rule, crawl delay, sitemap URL, AI crawler selections, individual crawler rules, and restricted directories. It also clears the generated output. Use this option when you want to start over with a new configuration.
Examples of Robots.txt Files
The following examples illustrate how the tool’s settings translate into robots.txt directives.
Example 1: Allow All Crawlers
Selection:
- Default Rule: Allow all
- No additional crawler restrictions
- No crawl delay
- No sitemap
Output:
User-agent: *
Disallow:
This allows crawling by default.
Example 2: Block All Crawlers
Selection:
- Default Rule: Disallow all
Output:
User-agent: *
Disallow: /
This instructs all crawlers matched by the wildcard rule not to crawl any URLs on the website. Use this setting cautiously on a live website, as it can prevent search engines from crawling your content.
Example 3: Block GPTBot but Allow General Crawling
Selection:
- Default Rule: Allow all
- GPTBot: Block
Output:
User-agent: *
Disallow:
User-agent: GPTBot
Disallow: /
General crawlers are allowed, while GPTBot is explicitly instructed not to crawl the site.
Example 4: Add a Sitemap and Restrict a Directory
Selection:
- Default Rule: Allow all
- Sitemap:
https://www.example.com/sitemap.xml - Restricted Directory:
/private/ - No crawl delay
Output:
User-agent: *
Disallow: /private/
Sitemap: https://www.example.com/sitemap.xml
This allows crawling by default but restricts the specified directory, and provides the sitemap location.
Example 5: Block AI Training Crawlers While Allowing Selected Search Crawlers
Selection:
- Default Rule: Allow all
- AI preset: Block common AI training crawlers, allow search and visitor bots
The output contains a general allow rule, followed by individual rules blocking the selected training-related crawlers and allowing the selected search and visitor crawlers. The exact output depends on which crawler checkboxes are enabled.
How to Upload the robots.txt File to Your Server?
Once you have generated and downloaded your file, follow these steps to publish it.
- Sign in to your hosting control panel.
- Open File Manager or connect to your server using FTP/SFTP.
- Locate your website’s document root. On many shared hosting platforms, this directory is named
public_html. - Upload the file named
robots.txtinto that directory. - If a robots.txt file already exists, back it up before replacing it.
- Open the file in your browser to confirm that the uploaded version is accessible.

For example, if your website is https://www.example.com/, the file should be available at:
https://www.example.com/robots.txt
If your website uses multiple subdomains, remember that each subdomain can have its own robots.txt file.
How to Verify Your robots.txt File?
After uploading the file, open your browser and visit the following URL:
https://www.example.com/robots.txt
Replace the example domain with your own domain. You should see the plain-text directives generated by the tool.
Make sure to check the following:
- The URL returns the correct file rather than a 404 error.
- The file contains the directives you intended to publish.
- The Sitemap directive points to the correct XML sitemap, if provided.
- The file does not accidentally block important website content.
- Specific AI crawler rules are present and configured as intended.
You can also use the Google Search Console to check how Google’s crawler interprets your robots.txt rules.

Remember that a correctly published robots.txt file provides crawling instructions, not guaranteed indexing behavior or access protection. Review it whenever your website structure, search strategy, or AI crawler policy changes.





